TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains

Andy C. Bavier, Marco Yuen, Jessica Blaine, Rick McGeer, Alvin Au Young, Yvonne Coady, Chris Matthews, Chris Pearson, Alex Snoeren, Joe Mambretti

2011

Abstract

In this position paper, we consider architectures of distributed interconnected clouds across geographically distributed, independently-administered storage and computation clusters. We consider two problems: federation of access across heterogeneous administrative domains and computation jobs run over the wide area and heterogeneous data sets. We argue that a single, flexible architecture, analogous to the TCP/IP stack for networking, is sufficient to support these jobs, and outline its major elements. As with the networking stack, many elements are in place today to build an initial version of this architecture over existing facilities. With the sponsorship of the US National Science Foundation GENI project and the cooperation of the EU FIRE project, we are building an initial implementation, the TransCloud. We describe our initial results.

References

  1. Bavier, A. et al., 2004. “Operating system support for planetary-scale network services”, Proceedings NSDI, 2004
  2. Bavier, A. et al., 2004. “Operating system support for planetary-scale network services”, Proceedings NSDI, 2004
  3. Barham, P. et al., 2003. “Xen and the art of virtualization”, Proceedings SOSP, 2003
  4. Barham, P. et al., 2003. “Xen and the art of virtualization”, Proceedings SOSP, 2003
  5. Peterson, L. et al. 2007 “Slice-based facility architecture”, http://www.cs.princeton.edu/llp/arch_abridged.pdf, 2007
  6. Peterson, L. et al. 2007 “Slice-based facility architecture”, http://www.cs.princeton.edu/llp/arch_abridged.pdf, 2007
  7. Brett, P. et al., 2004. “A Shared Global Event Propagation System to Enable Next Generation Distributed Services”, Proceedings WORLDS, 2004
  8. Brett, P. et al., 2004. “A Shared Global Event Propagation System to Enable Next Generation Distributed Services”, Proceedings WORLDS, 2004
  9. Dean, J. and Ghernawat, S. 2004. “MapReduce: Simplified Data Processing on Large Clusters”, Proceedings of OSDI 2004, December, 2004.”
  10. Dean, J. and Ghernawat, S. 2004. “MapReduce: Simplified Data Processing on Large Clusters”, Proceedings of OSDI 2004, December, 2004.”
  11. Borthakur, D., 2009. “The Hadoop Distributed File System: Architecture and Design”, http://hadoop. apache.org/common/docs/r0.18.0/hdfs_design.pdf
  12. Borthakur, D., 2009. “The Hadoop Distributed File System: Architecture and Design”, http://hadoop. apache.org/common/docs/r0.18.0/hdfs_design.pdf
  13. Cooper, B. F. et al. 2010. “PNUTS: Yahoo!'s Hosted Data Serving Platform”, http://research.yahoo.com/files/ pnuts.pdf
  14. Cooper, B. F. et al. 2010. “PNUTS: Yahoo!'s Hosted Data Serving Platform”, http://research.yahoo.com/files/ pnuts.pdf
  15. E. Evans. 2010. “Cassandra by Example”, http://www.rackspacecloud.com/blog/2010/05/12/cass andra-by-example/
  16. E. Evans. 2010. “Cassandra by Example”, http://www.rackspacecloud.com/blog/2010/05/12/cass andra-by-example/
  17. Paul Brown, 2001 Object Relational Database Development - A Plumber's Guide, 2001, PrenticeHall, Upper Saddle River, NJ 07458
  18. Paul Brown, 2001 Object Relational Database Development - A Plumber's Guide, 2001, PrenticeHall, Upper Saddle River, NJ 07458
  19. Cooperative Atmospheric Sensing Apparatus (CAA), 2010. University of Massachusetts, http:// www.casa.umass.edu/
  20. Cooperative Atmospheric Sensing Apparatus (CAA), 2010. University of Massachusetts, http:// www.casa.umass.edu/
  21. Gu, Y., Lu, L., Grossman, R., and Yoo, Y., 2010. “Processing Massived Sized Graphs using Sector/Sphere”, Proceedings 3rd Workshop on ManyTask Computing on Grids and Supercomputers, colocated with SC10, New Orleans, LA, Nov. 15, 2010.
  22. Gu, Y., Lu, L., Grossman, R., and Yoo, Y., 2010. “Processing Massived Sized Graphs using Sector/Sphere”, Proceedings 3rd Workshop on ManyTask Computing on Grids and Supercomputers, colocated with SC10, New Orleans, LA, Nov. 15, 2010.
  23. Gu, Y. and Grossman, R., 2009. “Lessons Learned From a Year's Worth of Benchmarks of Large Data Clouds,” Proceedings 2nd Workshop on Many-Task Computing on Grids and Supercomputers, co-located with SC09, Portland, Oregon -- November 16th, 2009.
  24. Gu, Y. and Grossman, R., 2009. “Lessons Learned From a Year's Worth of Benchmarks of Large Data Clouds,” Proceedings 2nd Workshop on Many-Task Computing on Grids and Supercomputers, co-located with SC09, Portland, Oregon -- November 16th, 2009.
  25. Gu, Y. and Grossman, R. 2009 “Sector and Sphere: The Design and Implementation of a High Performance Data Cloud”, Theme Issue of the Philosophical Transactions of the Royal Society A: Crossing Boundaries: Computational Science, E-Science and Global E-Infrastructure, 28 June 2009 vol. 367 no. 1897 2429-2445.
  26. Gu, Y. and Grossman, R. 2009 “Sector and Sphere: The Design and Implementation of a High Performance Data Cloud”, Theme Issue of the Philosophical Transactions of the Royal Society A: Crossing Boundaries: Computational Science, E-Science and Global E-Infrastructure, 28 June 2009 vol. 367 no. 1897 2429-2445.
  27. Gu, Y. and Grossman, R. 2008. “Exploring Data Parallelism and Locality in Wide Area Networks”, Proceedings of the Workshop on Many-task Computing on Grids and Supercomputers (MTAGS), co-located with SC08, Austin, TX. Nov. 2008.
  28. Gu, Y. and Grossman, R. 2008. “Exploring Data Parallelism and Locality in Wide Area Networks”, Proceedings of the Workshop on Many-task Computing on Grids and Supercomputers (MTAGS), co-located with SC08, Austin, TX. Nov. 2008.
  29. Gu, Y. and Grossman, R., 2008. “Data Mining Using High Performance Data Clouds: Experimental Studies Using Sector and Sphere”, Proceedings SIGKDD 2008, Las Vegas, NV, Aug. 2008.
  30. Gu, Y. and Grossman, R., 2008. “Data Mining Using High Performance Data Clouds: Experimental Studies Using Sector and Sphere”, Proceedings SIGKDD 2008, Las Vegas, NV, Aug. 2008.
  31. Van Renesse, R., Birman, K., and Vogels, W., 2003. “Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining”, ACM Transactions on Computer Systems, May, 2003
  32. Van Renesse, R., Birman, K., and Vogels, W., 2003. “Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining”, ACM Transactions on Computer Systems, May, 2003
  33. Cappos, J., Dadgar, A., Rasley, J., Samuel, J., IBeschastnikh, I., Barsan, C., Krishnamurthy, A. and Anderson, T. 2010. "Retaining Sandbox Containment Despite Bugs in Privileged Memory-Safe Code." Conference on Computer and Communications Security (CCS 7810), Chicago, IL, 2010.
  34. Cappos, J., Dadgar, A., Rasley, J., Samuel, J., IBeschastnikh, I., Barsan, C., Krishnamurthy, A. and Anderson, T. 2010. "Retaining Sandbox Containment Despite Bugs in Privileged Memory-Safe Code." Conference on Computer and Communications Security (CCS 7810), Chicago, IL, 2010.
  35. Cappos, J., Beschastnikh, I., Krishnamurthy, A. and Anderson. T. 2009. “Seattle: A Platform for Educational Cloud Computing." SIGCSE 7809.
  36. Cappos, J., Beschastnikh, I., Krishnamurthy, A. and Anderson. T. 2009. “Seattle: A Platform for Educational Cloud Computing." SIGCSE 7809.
  37. Yee, B., Sehr, D., Dardyk, G., Chen, B., Muth, R., Ormandy, T., Okasaka, T. Narula, N., and Fullagar, N. 2009. “Native Client: A Sandbox for Portable, Untrusted x86 Native Code”, IEEE Symposium on Security and Privacy (Oakland'09), 2009.
  38. Yee, B., Sehr, D., Dardyk, G., Chen, B., Muth, R., Ormandy, T., Okasaka, T. Narula, N., and Fullagar, N. 2009. “Native Client: A Sandbox for Portable, Untrusted x86 Native Code”, IEEE Symposium on Security and Privacy (Oakland'09), 2009.
  39. Matthews, C., Cappos, J., Coady, Y., Hartman, J., Jacky, J., and McGeer, R. 2010. “NanoXen: Better Systems Through Rigorous Containment and Active Modeling”, OSDI 2010 (Poster).
  40. Matthews, C., Cappos, J., Coady, Y., Hartman, J., Jacky, J., and McGeer, R. 2010. “NanoXen: Better Systems Through Rigorous Containment and Active Modeling”, OSDI 2010 (Poster).
  41. Yalagandula, P., Sharma, P., Banerjee, S., Lee, S-J. and Sujoy Basu, S., 2006. “S3: A Scalable Sensing Service for Monitoring Large Networked Systems”, Proceedings of ACM INM 2006(in conjunction with Sigcomm 2006), Pisa, Italy, September 2006.
  42. Yalagandula, P., Sharma, P., Banerjee, S., Lee, S-J. and Sujoy Basu, S., 2006. “S3: A Scalable Sensing Service for Monitoring Large Networked Systems”, Proceedings of ACM INM 2006(in conjunction with Sigcomm 2006), Pisa, Italy, September 2006.
  43. Madden, S., Franklin, M. J., Hellerstein, J. M. and Hong, W. 2005. “TinyDB: An Acqusitional Query Processing System for Sensor Networks”. ACM TODS, 2005.
  44. Madden, S., Franklin, M. J., Hellerstein, J. M. and Hong, W. 2005. “TinyDB: An Acqusitional Query Processing System for Sensor Networks”. ACM TODS, 2005.
  45. Oppenheimer, D. et al., 2004. “Distributed Resource Discovery on PlanetLab with SWORD”, Proceedings WORLDS, 2004.
  46. Oppenheimer, D. et al., 2004. “Distributed Resource Discovery on PlanetLab with SWORD”, Proceedings WORLDS, 2004.
  47. Massie, M. et al. “The Ganglia Distributed Monitoring System: Design, Implementation And Experience”, Parallel Computing, 2003.
  48. Massie, M. et al. “The Ganglia Distributed Monitoring System: Design, Implementation And Experience”, Parallel Computing, 2003.
  49. Apache Project. 2010. Pig, http://pig.apache.org
  50. Apache Project. 2010. Pig, http://pig.apache.org
  51. Bavier, A. et al. 2006. “Increasing TCP Throughput with an Enhanced Internet Control Plane, Proceedings MILCOMM 2006
  52. Bavier, A. et al. 2006. “Increasing TCP Throughput with an Enhanced Internet Control Plane, Proceedings MILCOMM 2006
  53. Brassil J. et al, The CHART System: A HighPerformance, Fair Transport Architecture Based on Explicit-Rate Signaling, ACM SIGOPS Review, February, 2009
  54. Brassil J. et al, The CHART System: A HighPerformance, Fair Transport Architecture Based on Explicit-Rate Signaling, ACM SIGOPS Review, February, 2009
  55. R. McGeer, B. L. Mark, J. Brassil, P. Sharma, P. Yalagandula, S. Schwab, and S. Zhang, 2009. "The Case for Service Overlays," Proc. 18th IEEE Int. Conf. on Computer Communications and Networks (ICCCN'09), San Francisco, CA, Aug. 2009.
  56. R. McGeer, B. L. Mark, J. Brassil, P. Sharma, P. Yalagandula, S. Schwab, and S. Zhang, 2009. "The Case for Service Overlays," Proc. 18th IEEE Int. Conf. on Computer Communications and Networks (ICCCN'09), San Francisco, CA, Aug. 2009.
  57. Lee, J., Sharma, P., Tourrilhes, J., McGeer, R., Brassil, J., and Bavier, A., 2010. “Network Integrated Transparent TCP Accelerator”, Proceedings AINA 2010, May 2010
  58. Lee, J., Sharma, P., Tourrilhes, J., McGeer, R., Brassil, J., and Bavier, A., 2010. “Network Integrated Transparent TCP Accelerator”, Proceedings AINA 2010, May 2010
  59. Nurmi, D. et al. 2009. “The Eucalyptus Open-Source Cloud-Computing System”, CCGRID 7809, 2009
  60. Nurmi, D. et al. 2009. “The Eucalyptus Open-Source Cloud-Computing System”, CCGRID 7809, 2009
  61. Kosuch, M. et al. 2009. "Tashi: Location-aware Cluster Management", First Workshop on Automated Control for Datacenters and Clouds (ACDC'09), June 2009
  62. Kosuch, M. et al. 2009. "Tashi: Location-aware Cluster Management", First Workshop on Automated Control for Datacenters and Clouds (ACDC'09), June 2009
  63. Keahey, K., Freeman, T. 2008. "Contextualization: Providing One-Click Virtual Clusters", 2008 Fourth IEEE International Conference on eScience, pp.301- 308. doi:10.1109/eScience.2008.82
  64. Keahey, K., Freeman, T. 2008. "Contextualization: Providing One-Click Virtual Clusters", 2008 Fourth IEEE International Conference on eScience, pp.301- 308. doi:10.1109/eScience.2008.82
Download


Paper Citation


in Harvard Style

C. Bavier A., Yuen M., Blaine J., McGeer R., Au Young A., Coady Y., Matthews C., Pearson C., Snoeren A. and Mambretti J. (2011). TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains . In Proceedings of the 1st International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-8425-52-2, pages 120-126. DOI: 10.5220/0003450301200126


in Harvard Style

C. Bavier A., Yuen M., Blaine J., McGeer R., Au Young A., Coady Y., Matthews C., Pearson C., Snoeren A. and Mambretti J. (2011). TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains . In Proceedings of the 1st International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-8425-52-2, pages 120-126. DOI: 10.5220/0003450301200126


in Bibtex Style

@conference{closer11,
author={Andy C. Bavier and Marco Yuen and Jessica Blaine and Rick McGeer and Alvin Au Young and Yvonne Coady and Chris Matthews and Chris Pearson and Alex Snoeren and Joe Mambretti},
title={TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains},
booktitle={Proceedings of the 1st International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2011},
pages={120-126},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003450301200126},
isbn={978-989-8425-52-2},
}


in Bibtex Style

@conference{closer11,
author={Andy C. Bavier and Marco Yuen and Jessica Blaine and Rick McGeer and Alvin Au Young and Yvonne Coady and Chris Matthews and Chris Pearson and Alex Snoeren and Joe Mambretti},
title={TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains},
booktitle={Proceedings of the 1st International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2011},
pages={120-126},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003450301200126},
isbn={978-989-8425-52-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains
SN - 978-989-8425-52-2
AU - C. Bavier A.
AU - Yuen M.
AU - Blaine J.
AU - McGeer R.
AU - Au Young A.
AU - Coady Y.
AU - Matthews C.
AU - Pearson C.
AU - Snoeren A.
AU - Mambretti J.
PY - 2011
SP - 120
EP - 126
DO - 10.5220/0003450301200126


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - TRANSCLOUD - Design Considerations for a High-performance Cloud Architecture Across Multiple Administrative Domains
SN - 978-989-8425-52-2
AU - C. Bavier A.
AU - Yuen M.
AU - Blaine J.
AU - McGeer R.
AU - Au Young A.
AU - Coady Y.
AU - Matthews C.
AU - Pearson C.
AU - Snoeren A.
AU - Mambretti J.
PY - 2011
SP - 120
EP - 126
DO - 10.5220/0003450301200126