Virtual Clusters for Community Computation

Deploy HTCondor, WorkQueue and Spark clusters/ Integrate diverse HPC resources/ Enable collaborative science/ Open to beta testers early 2018

VC3 automates deployment of cluster frameworks to access diverse computing resources for collaborative science teams

Virtual Clusters 3 Divider
Virtual Clusters 3 Divider

About VC3

VC3 is built from a number of software tools, technologies, and integrative experiences.

APF = AutoPyFactory = Automatic Python Factory. APF is a scalable, flexible, easily deployed generic provisioning factory. It monitors idle workloads, calculates the appropriate number of jobs/containers to submit/launch (or kill), and then does so.

Leveraging Condor-G from HTCondor, APF can manage submissions to a wide array of resources. It is plugin-based to allow flexible and robust combinations of monitored workloads, resource targets, and submission types. A unique 'algorithm by configuration' mechanism allows the deployer to program APF's submission behavior without coding.

A cluster framework from the Cooperative Computing Lab for massively scalable scientific applications supported by VC3. Tasks are executed by a standard worker process that can run on any available machine. Each worker calls home to the master process, arranges for data transfer, and executes the tasks. The system handles a wide variety of failures, allowing for dynamically scalable and robust applications.

VC3 builds on lessons from the the CI Connect project which resulted in OSG Connect, ATLAS Connect, UChicago CI Connect, Duke CI Connect, and CMS Connect virtual cluster systems.

VC3 leverages advanced identity management and authorization capabilities of the Globus Platform. Globus federates identities from over 400 identity providers including the DOE national laboratories, InCommon, Google, XSEDE and Internet2. Researchers can use their home institution credentials to access VC3 services.

VC3 extends the foundational high-throughput computing principles established by the HTCondor team and implemented at scale by the Open Science Grid Consortium.

VC3 utilized the technical advising and consulting services of the Science Gateways Community Institute. The Extended Developer Support from SGCI provided guidance on the overall design of the portal architecture and its integration with VC3 core platform components.