"We amplified our Chef infrastructure and streamlined configuration scripts to help scalability. We developed an approximate timing architecture which benchmarked and facilitated the increased peak number of nodes and the nodes per unit time. Chef's Chef enabled us to easily and reliably configure our systems to meet our demanding scale requirements." Jason Stowe, CEO
Cycle Computing helps customers of all sizes gain fast and efficient access to the kind of supercomputing muscle usually reserved for universities and large research organizations. Cycle's founder, CEO and HPC expert Jason Stowe wanted to test the envelope of cloud computing and take its technology to the next level - spinning up a 10,000- core cluster in the cloud. Stowe teamed up with a prominent life science company who needed supercomputing power to research how proteins bind to each other, and utilized Amazon Web Services to begin provisioning 10,000 cores in an HPC cluster on Amazon's EC2 compute cloud.
Cycles's Technical Operations department deployed using a CentOS Linux cluster; Cycle built its 10,000 cores using 1,250 instances with eight cores each. Scaling up roughly 2,000 cores at a time, Cycle enlisted its own CycleCloud software, the Condor scheduling system, and Chef's Chef cloud infrastructure automation engine. Using Chef, the total time to provision the entire cluster was 45 minutes, and cost approximately $1,000 per hour for a 10,000-core cluster.
Cycle Computing has delivered its proven, secure and flexible high performance computing (HPC) and data software solution since 2005. The company’s technology helps clients maximize existing infrastructure and speed computations on desktops and servers on-demand in the cloud. With its CycleServer HPC management software and CycleCloud fully supported and secured HPC clusters, Cycle clients experience faster time-to-market, decreased operating costs, and unprecedented service and support. Cycle uses Amazon EC2 resources to power its unique high-performance cloud computing platform, providing highly scalable, reliable and secure infrastructure for Cycle's customers.
"Amazon's nearly limitless scale means no job is too big, while our CycleServer and CycleCloud solutions offer seamless management and robust security," said Jason Stowe, CEO Cycle Computing. "However, it would be nearly impossible for us to manage infrastructure as the scale our customers require without automation."
Cycle deployed Chef to automate the provisioning and configuration of thousands of Amazon EC2 servers, ensuring maximum infrastructure consistency and reducing deployment times from months to minutes. With Chef, the Cycle team can deploy infrastructure as code, easily repeating configurations across hundreds or thousands of Amazon EC2 instances in minutes and delivering compute power to customers on-demand.
"Chef enables us to deploy infrastructure at unprecedented scale with maximum consistency and minimal management," Stowe continued. "We use a few simple lines of code to configure thousands of servers simultaneously, making us highly responsive to customer needs and giving us the ability to pursue ever-larger projects."
When Schrodinger, a worldwide leader in computational chemistry research, needed to virtually screen 21 million molecule conformations against one possible cancer target, it turned to Cycle because the project was too intense for its available physical infrastructure. To meet Schrodinger's needs, Cycle Computing used Chef to provision and configure a 50,000 core utility supercomputer in Amazon EC2 in less than two hours.
Using Chef to automate server configuration, Cycle scaled out Schrodinger's cloud supercomputer across seven different geographic regions, maintaining full availability throughout. Then, Cycle deployed its CycleServer software on top of the supercomputer to intelligently provision compute tasks to each region based on real-time bandwidth measurements, ensuring no region was over or under utilized.
"The Schrodinger project was our largest to date by far, but is nowhere close to the limits we'll be able to achieve with Amazon and Chef," Stowe added. "By automating what would take months to do by hand, we've eliminated management time and human resources as barriers to scale, which opens up nearly infinite growth potential."