YellowDog spins up AWS cluster with 3.2m vCPUs

YellowDog Platform

Cloud workload management company YellowDog said that it spun up a huge distributed supercomputer on Amazon Web Services.

On behalf of biotech firm OMass Therapeutics, the company spun up 3.2m vCPUs (virtual CPUs) to analyze and screen 337 million compounds in seven hours.

The company used a mixture of Amazon EC2 (AWS’ primary elastic compute platform) and Amazon EC2 Spot Fleets (where AWS sells spare EC2 capacity at a lower price), as well as Amazon Simple Storage Service (Amazon S3) for data transfer and access.

YellowDog said that it launched 46,733 Spot Instances, utilizing 24 Amazon EC2 Fleets and eight different instance types. This culminated with 3.2 million vCPUs provisioned, with a more than 95 percent utilization.

The compute was spread out across data centers in North America and Europe.

“The run had one million vCPUs working within seven minutes. The two million mark was achieved within 11 minutes,” Colin Bridger, YellowDog’s principal HPC GTM specialist, said in a blog post.

“At these milestones, over 95 percent of vCPUs were processing data at full utilization. Within 33 minutes, the 3.2 million vCPUs were processing the workload, evenly split amongst the Amazon Elastic Compute Cloud (Amazon EC2) fleets.”

The result allowed OMass Therapeutics to analyze compounds in seven hours that would have taken two months on the company’s undisclosed on-premises supercomputer.

“The professional and timely manner of this collaboration has enabled OMass to rapidly screen a novel target and help the company in its ambition to build a pipeline of small molecule therapeutics,” said Dr. Giles Brown, VP and head of medicinal chemistry at OMass.

“The scale and speed of the platform is something OMass would not be able to replicate.”