With the Fall "capability demonstrator" at the end of the month, the VC3 team had a face-to-face developer meeting in Chicago on September 5 and 6 2017. The main goals where to stand up virtual clusters, create virtual clusters from the website, and finalize details for the "capability demonstrator."
We were able to create a virtual cluster and submit sample workloads to UChicago's Midway campus cluster and MWT2's CoreOS Cluster and request the creation of a virtual cluster from the website. We also finalized where and which payloads we would submit during the "capability demonstrator," mainly UChicago's Midway campus cluster, MWT2's CoreOS Cluster, NERSC's Cori, and Syracuse's OrangeGrid.
With the first release of VC3 and the Fall "capability demonstrator" on the horizon, the VC3 team had a face-to-face developer meeting in Chicago from July 17 to July 19 2017. The main goals where to finalize the software architecture of the VC3 static infrastructure, commission the VC3 static infrastructure on our OpenStack private cloud, integrate the website into static infrastructure, manually create a virtual cluster (VC), and sketch out a road map forward to the first release.
For the VC3 static infrastructure, we were able to finalize and implement the software architecture, and commission the VC3 static infrastructure including a VC3 master, VC3 inforservice, VC3 factory, and static HTCondor central manager using the our OpenStack private cloud. We were able to use the static infrastructure to submit jobs to MWT2 resources and the UChicago campus cluster. This provides us the foundation for create a VC.
We were able to nearly complete the connection between the VC3 website and the static infrastructure. As for the manual creation of a VC, we had all the individual components running. We ran out of time to be able to have information flow from one end of the VC3 chain (website or CLI interface) to submitting pilot jobs.
Overall, we considered this meeting a success. We were able to accomplish nearly all the goals we had set out and were able to see jobs going to sites we will be using for our Fall "capability demonstrator".
The VC3 Command Line Interface for Administrators
This example shows how the vc3-client interacts with the VC3 information service to create new users, projects and resources, and using these spin up an HTCondor virtual cluster. These commands are meant to be executed by VC3 administrators.
Create new users, projects, and resource profiles
$ vc3-client user-create --firstname Lincoln --lastname Bryant --email firstname.lastname@example.org --institution UChicago lincolnb $ vc3-client project-create --owner rwg --members rwg,jhover ATLAS $ vc3-client project-adduser ATLAS jcaballero $ vc3-client resource-create --owner lincolnb --accesstype batch --accessmethod ssh --accessflavor htcondor --accesshost condor.grid.uchicago.edu --accessport 22 uchicago-grid $ vc3-client resource-create --owner lincolnb --accesstype batch --accessmethod ssh --accessflavor slurm --accesshost cori.nersc.gov --accessport 22 nersc-cori
Watch the service logs when creating new users, projects, and resource profiles.
A seminar describing the VC3 architecture was given to the SDSS/RACF computing group at Brookhaven National Laboratory by John Hover, the lead architect of the VC3 core. Slides are available here.
Our goal for the fall is to create a "capability demonstrator". As we're just getting started we'll build from existing software and applications and mold them to higher level abstractions. The examples below demonstrate prototyping features of a VC3 pilot and factory service.
Deploy a global file system on the fly
$ vc3-pilot --require cvmfs
Deploy the MAKER application on the fly
$ vc3-pilot --require maker-ecoli-example-01
Send VC3 pilots to a remote cluster via ssh "glide in"
Start VC3 pilots on a resource requiring 2-factor authentication
Today and tomorrow we have the Next Generation Networks for Science (NGNS) Principal Investigators' (PI) Meeting at Argonne National Laboratory. More information can be found here. Doug Thain will be giving the VC3 presentation tomorrow.
Our official "kickoff" meeting to get the project formally started. The main theme will be to revisit our vision and proposal, to identify and carve out the “VC3 space” given changes in the community since the proposal, and to prepare for the NGNS PI workshop. Expect the meeting will be small and informal with lots of white-boarding. Success here would be to come out of the meeting with a year 1 program of work and a Fall "capability demonstrator".