MULTI-PROCESS OPTIMIZATION VIA HORIZONTAL MESSAGE QUEUE PARTITIONING

Matthias Boehm, Dirk Habich, Wolfgang Lehner

2010

Abstract

Message-oriented integration platforms execute integration processes—in the sense of workflow-based process specifications of integration tasks—in order to exchange data between heterogeneous systems and applications. The overall optimization objective is throughput maximization, i.e., maximizing the number of processed messages per time period. Here, moderate latency time of single messages is acceptable. The efficiency of the central integration platform is crucial for enterprise data management because both the data consistency between operational systems and the up-to-dateness of analytical query results depend on it. With the aim of integration process throughput maximization, we propose the concept of multi-process optimization (MPO). In this approach, messages are collected during a waiting period and executed in batches to optimize sequences of process instances of a single process plan. We introduce a horizontal—and thus, valuebased—partitioning approach for message batch creation and show how to compute the optimal waiting time with regard to throughput maximization. This approach significantly reduces the total processing time of a message sequence and hence, it maximizes the throughput while accepting moderate latency time.

References

  1. Agrawal, S., Narasayya, V. R., and Yang, B. (2004). Integrating vertical and horizontal partitioning into automated physical database design. In SIGMOD.
  2. Avnur, R. and Hellerstein, J. M. (2000). Eddies: Continuously adaptive query processing. In SIGMOD.
  3. Biornstad, B., Pautasso, C., and Alonso, G. (2006). Control the flow: How to safely compose streaming services into business processes. In SCC.
  4. Boehm, M., Habich, D., Preissler, S., Lehner, W., and Wloka, U. (2009). Cost-based vectorization of instance-based integration processes. In ADBIS.
  5. Boehm, M., Wloka, U., Habich, D., and Lehner, W. (2008). Workload-based optimization of integration processes. In CIKM.
  6. Candea, G., Polyzotis, N., and Vingralek, R. (2009). A scalable, predictable join operator for highly concurrent data warehouses. PVLDB, 2(1).
  7. Cecchet, E., Candea, G., and Ailamaki, A. (2008). Middleware-based database replication: the gaps between theory and practice. In SIGMOD.
  8. Ceri, S., Negri, M., and Pelagatti, G. (1982). Horizontal data partitioning in database design. In SIGMOD.
  9. Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., and Gruber, R. (2006). Bigtable: A distributed storage system for structured data. In OSDI.
  10. Chaudhuri, S. and Shim, K. (1994). Including group-by in query optimization. In VLDB.
  11. Eadon, G., Chong, E. I., Shankar, S., Raghavan, A., Srinivasan, J., and Das, S. (2008). Supporting table partitioning by reference in oracle. In SIGMOD.
  12. Harizopoulos, S., Shkapenyuk, V., and Ailamaki, A. (2005). Qpipe: A simultaneously pipelined relational query engine. In SIGMOD.
  13. Ivanova, M., Kersten, M. L., Nes, N. J., and Goncalves, R. (2009). An architecture for recycling intermediates in a column-store. In SIGMOD.
  14. Ives, Z. G., Halevy, A. Y., and Weld, D. S. (2004). Adapting to source properties in processing data integration queries. In SIGMOD.
  15. Johnson, R., Hardavellas, N., Pandis, I., Mancheril, N., Harizopoulos, S., Sabirli, K., Ailamaki, A., and Falsafi, B. (2007). To share or not to share? In VLDB.
  16. Johnson, T., Muthukrishnan, S. M., Shkapenyuk, V., and Spatscheck, O. (2008). Query-aware partitioning for monitoring massive network data streams. In SIGMOD.
  17. Kementsietsidis, A., Neven, F., de Craen, D. V., and Vansummeren, S. (2008). Scalable multi-query optimization for exploratory queries over federated scientific databases. In VLDB.
  18. Lee, R., Zhou, M., and Liao, H. (2007). Request window: an approach to improve throughput of rdbmsbased data integration system by utilizing data sharing across concurrent distributed queries. In VLDB.
  19. Li, H. and Zhan, D. (2005). Workflow timed critical path optimization. Nature and Science, 3(2).
  20. Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., and Maier, D. (2008). Out-of-order processing: a new architecture for high-performance stream systems. PVLDB, 1(1).
  21. Little, J. D. C. (1961). A proof for the queueing formula: l = lw. Operations Research, 9.
  22. Qiao, L., Raman, V., Reiss, F., Haas, P. J., and Lohman, G. M. (2008). Main-memory scan sharing for multicore cpus. PVLDB, 1(1).
  23. Roy, P., Seshadri, S., Sudarshan, S., and Bhobe, S. (2000). Efficient and extensible algorithms for multi query optimization. In SIGMOD.
  24. Scheuermann, P. and Ouksel, A. M. (1982). Multidimensional b-trees for associative searching in database systems. Inf. Syst., 7(2).
  25. Silberstein, A., Cooper, B. F., Srivastava, U., Vee, E., Yerneni, R., and Ramakrishnan, R. (2008). Efficient bulk insertion into a distributed ordered table. In SIGMOD.
  26. Srivastava, U., Munagala, K., Widom, J., and Motwani, R. (2006). Query optimization over web services. In VLDB.
  27. Unterbrunner, P., Giannikis, G., Alonso, G., Fauser, D., and Kossmann, D. (2009). Predictable performance for unpredictable workloads. PVLDB, 2(1).
  28. Vrhovnik, M., Schwarz, H., Suhre, O., Mitschang, B., Markl, V., Maier, A., and Kraft, T. (2007). An approach to optimize data processing in business processes. In VLDB.
  29. Xiao, Z., Chang, H., and Yi, Y. (2006). Optimal allocation of workflow resources with cost constraint. In CSCWD.
Download


Paper Citation


in Harvard Style

Boehm M., Habich D. and Lehner W. (2010). MULTI-PROCESS OPTIMIZATION VIA HORIZONTAL MESSAGE QUEUE PARTITIONING . In Proceedings of the 12th International Conference on Enterprise Information Systems - Volume 1: ICEIS, ISBN 978-989-8425-04-1, pages 5-14. DOI: 10.5220/0002862600050014


in Bibtex Style

@conference{iceis10,
author={Matthias Boehm and Dirk Habich and Wolfgang Lehner},
title={MULTI-PROCESS OPTIMIZATION VIA HORIZONTAL MESSAGE QUEUE PARTITIONING},
booktitle={Proceedings of the 12th International Conference on Enterprise Information Systems - Volume 1: ICEIS,},
year={2010},
pages={5-14},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0002862600050014},
isbn={978-989-8425-04-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Conference on Enterprise Information Systems - Volume 1: ICEIS,
TI - MULTI-PROCESS OPTIMIZATION VIA HORIZONTAL MESSAGE QUEUE PARTITIONING
SN - 978-989-8425-04-1
AU - Boehm M.
AU - Habich D.
AU - Lehner W.
PY - 2010
SP - 5
EP - 14
DO - 10.5220/0002862600050014