Job Composer App¶
(formerly called “My Jobs”)
This Open OnDemand application provides a web-based utility for creating and managing batch jobs from template directories. This application is built with the Ruby on Rails web application framework.
The Job Composer App attempts to model a simple but common workflow that typical users of an HPC center use. When users create new batch jobs they will follow the given workflow:
- Copy a directory of a previous job, either one of their previous jobs or a job from a group member
- Make minor modifications to the input files
- Submit this new job
The Job Composer app replicates this through “templates” which are just directories with common job input files. The user will be able to create their own template from a job directory they have access to on the file system or they can use one provided to them from a system-installed location. This system-installed location is located at:
Once a user has chosen a template that will meet their needs, they can create a job from it. This essentially just copies the template directory and its contents to a new job directory under the user’s home directory:
The user can then modify these input files and then submit the job.
The Job Composer app also provides integrated support for launching the Files App in the currently chosen job or template, giving the user a web-interface for modifying the files in the directory. It also provides support for launching the Shell App in the job directory for direct access to the terminal when needed.
This app is deployed on the OnDemand Server under the following path on the local file system:
and can be accessed with the following browser request:
Launches the Job Composer App web-based utility.
A template is a directory with input files for a job as well as a
manifest.yml file. The manifest contains metadata about the corresponding
# ./manifest.yml --- name: Example Template host: cluster1 script: my_job.sh notes: Notes about the template that describes its function to the user
In the event that a job is created from a template that has no manifest or has missing metadata, the Job Composer app will assign the following default values:
- The template directory name
- The first cluster with a valid Resource Manager Server in the OOD cluster config
- The first
*.shappearing in the template directory
- The path to this manifest if it existed
How it Works¶
Requirements needed for the Job Composer App to work on your local HPC network:
- OnDemand Server
- Resource Manager Server (e.g., Torque/PBS Batch Server)
- Shared File System
Fig. 20 details how the Job Composer App works on a local HPC
system. The user’s PUN running on the OnDemand Server launches the Ruby on
Rails Job Composer app through Passenger as the user. The Job Composer app reads from a
database all the previous recorded jobs. It then queries the Resource Manager
Server using either a library call or fork’ing a binary (e.g.,
parsing the output for the status of all the remaining jobs, followed by
generating a table of information that is displayed to the user.
For the currently selected job it will access the Shared File System to display information about the contents of the job directory (e.g., input and output files).
For job submission the Job Composer app will communicate the job submission and its
arguments to the Resource Manager Server again through either a library call or
fork’ing a binary (e.g.,
qsub) and parsing the output for the job id.