v1.7 Release Notes

Highlights in 1.7:

Special thanks

Special thanks to these contributors (alpha by first name):

If you contributed to the 1.7 OnDemand release and you are not on the core team and your name is not listed here please let us know and we will add you.

Upgrading from v1.6

Warning

In v1.7 support for CentOS/RedHat 6 has been dropped

Before upgrading to v1.7, please consider the following:

  • If upgrading from OnDemand 1.3.x or older you must upgrade to 1.6.x before upgrading to 1.7. See directions for upgrading from 1.4 to 1.5 (v1.5 Release Notes) and from 1.5 to 1.6 (v1.6 Release Notes).
  • As always please update the development or test instances of OnDemand installed at your center first to test and verify before you modify the production instance.
  • All running PUNs must be restarted after the upgrade due to upgrading to new versions of Passenger and Ruby. It’s recommended to schedule an short outage of your OnDemand instance to perform this upgrade.
  1. Update OnDemand release RPM

    sudo yum install -y https://yum.osc.edu/ondemand/1.7/ondemand-release-web-1.7-1.noarch.rpm
    
  2. Update OnDemand

    sudo yum clean all
    sudo yum update ondemand
    
  3. Update Apache configuration and restart Apache.

    sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal --force
    sudo systemctl try-restart httpd24-httpd.service httpd24-htcacheclean.service
    

    Warning

    Due to changes with ood-portal-generator it is necessary to run update_ood_portal with --force flag after upgrading to OnDemand 1.7. The RPM upgrade will generate /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf.new. If there are custom manual changes made to /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf it will be necessary to merge those changes in after running update_ood_portal --force.

  4. Force all PUNs to restart

    sudo /opt/ood/nginx_stage/sbin/nginx_stage nginx_clean -f
    
  5. Optionally remove old dependencies from prior versions of OOD if they are not used by other applications.

    sudo yum remove rh-ruby24\* rh-nodejs6\* rh-git29\*
    

Upgrading from v1.5

Follow the directions for upgrading to v1.6 first (v1.6 Release Notes). After the upgrade is complete and working, follow the directions above.

Details

Support running OnDemand with SELinux

Beta support for running OnDemand with SELinux has been added. Support is enabled by installing the ondemand-selinux package. For details see SELinux

Warning

If the ondemand-selinux package was installed from the OnDemand 1.6 release there are several changes that must be addressed manually. Several SELinux booleans were enabled by the ondemand-selinux package and are no longer needed by OnDemand. The following SELinux booleans can be disabled if not used outside OnDemand:

sudo setsebool -P httpd_execmem=off
sudo setsebool -P httpd_unified=off
sudo setsebool -P httpd_enable_homedirs=off
sudo setsebool -P httpd_read_user_content=off

Update project dependencies

Table 11 Project Dependencies
Dependency Old Version New Version
Passenger 5.3.7 6.0.4
NGINX 1.14.0 1.17.3
Ruby 2.4 2.5
NodeJS 6 10
Rails 4.2 5.2
noVNC 1.0.0 1.1.0

Drop EL6 and add EL8 support

OnDemand has dropped support for CentOS/RedHat 6 and added support for CentOS/RedHat 8.

Added the Linux Host Adapter

Support for running jobs directly on Linux hosts has been added. This means interactive jobs can now run on a login node instead of through a batch scheduler.

See the documentation for the Configure LinuxHost Adapter (beta). for more details.

Show account balance warnings

The dashboard can now show account balance warnings if the users’ account balance is some threshold.

See these docs for more details on how to enable this feature.

Job Composer and Active Jobs match Dashboard Branding

The Job Composer and Active Jobs apps’ top navigation bar now share the same branding scheme configured for the dashboard.

There’s no additional configuration required for this other than what’s documented here.

Special thanks to @zooley (Shawn Rice) for implementing this feature!

Maintenance mode

OOD can be put into ‘maintenance mode’ where it serves a static page to either all users or a subset (staff on a VPN for example). This static page informs users that maintenance is underway while administrators perform disruptive tasks.

See these docs on how to configure and enable this feature.

Titles for Files menu entries

The shortcuts in the files dropdown menu can now show an optional title along with the directory path. The original documentation on this feature has been updated to reflect the new options.

Documentation for Keycloak as identity broker with CILogon and Keycloak with Duo 2FA

Documentation for Keycloak as identity broker with CILogon has been added in the documentation entitled ‘Configure Keycloak with CILogon’.

Documentation for Keycloak with Duo two factor authentication has been updated in the documentation entitled ‘Two Factor Auth using Duo with Keycloak’.

Add Grafana support for ActiveJobs app

OnDemand’s ActiveJobs app can display graphs for jobs that are pulled from Grafana. Details on how to configure Grafana support are in these docs.

Improvements to ood-portal-generator

The ood-portal-generator app has been rewritten in Ruby and extensive testing added. Checksums generated for ood-portal.conf now exclude comments.

Add an ssh wrapper to the shell app

An ssh wrapper script can now be used instead of just the ssh command in the shell app. See how to configure this here.

Special thanks to @baverhey (Bart Verheyde) for implementing this feature!

Regenerate ood-portal.conf with every apache restart

Systemd will now regenerate the ood-portal.conf before every apache restart. This means administrators can now edit the ood_portal.yml configuration and then restart httpd directly without having to do the additional step of running the ood-portal-generator.

Apache httpd will read the new configuration, because a new ood-portal.conf will have been written just before restarting.

However you will have to follow the upgrade instructions (above), and then this will be available. Specifically updating the ood portal through sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal --force.

This feature relies on checksums generated by ood-portal-generator. If the checksums differ (i.e., ood-portal.conf has been edited by hand, outside of the ood-portal-generator program) this will fail and you’ll have to force an update through update_ood_portal --force.

Support sanitizing job names

Administrators can now set the OOD_JOB_NAME_ILLEGAL_CHARS environment variable to prevent characters from being used in job names. For example if you do not want to use / in job names (as is the case with some Grid Engine versions) you would set OOD_JOB_NAME_ILLEGAL_CHARS: '/' in the pun_custom_env attribute of nginx_stage.yml.

nginx_clean support for a specific user

The nginx_stage nginx_clean command now supports a -u or --user option so it may kill a specific users’ PUN. For example nginx_stage nginx_clean -u johndoe would only kill johndoe’s PUN and disregard all the others.

Dashboard now alerts on malformed cluster configs

Prior to 1.7 the dashboard would not start if there was a cluster cluster definition file (the files in /etc/ood/config/clusters.d/) that had invalid yaml.

The dashboard now handles this gracefully and shows an error message to the user stating that this file is unusable and should indicate the line of the file that is problematic.

Job composer now shows suggested scripts

When changing the job script in the job composer the user is now presented with a dropdown of ‘Suggested Files’ first along with ‘Other valid files’.

Files in folder will be suggested if they match any of these criteria:

  1. Have one of these extensions: “.sh”, “.job”, “.slurm”, “.batch”, “.qsub”, “.sbatch”, “.srun”, “.bsub”
  2. The file starts with a shebang line (#!)
  3. Has a resource manager’s directive (#PBS, #SBATCH, #BSUB or #$) in the first 1000 characters.

Other valid files only have to meet a size requirement of less than OOD_MAX_SCRIPT_SIZE_KB which defaults to 65 (meaning 65 kb).

Reduced installation size by 30 percent

We reduced the size of the main OnDemand RPM by nearly 40%.

The 1.6 ondemand-1.6.22-1.el7.x86_64.rpm was 162M and in 1.7 this has been reduced by 63M to a total of 99M, split between two RPMs: ondemand-gems-1.7.10-1.7.10-2.el7.x86_64.rpm with 68M, and ondemand-1.7.10-2.el7.x86_64.rpm with 31M. This is in part due to switching to a monorepo at https://github.com/osc/ondemand and installing all the gems into a central shared location, installed using the new ondemand-gems RPM.

The resulting sizes of the two main install directories, /opt/ood and /var/www/ood/apps/sys, were in total reduced by 181M:

$ cat /opt/ood/VERSION
v1.6.22
$ sudo du -h -s  -x /var/www/ood/apps/sys /opt/ood
592M /var/www/ood/apps/sys
17M  /opt/ood

vs

$ cat /opt/ood/VERSION
v1.7.10-2
$ sudo du -h -s  -x /var/www/ood/apps/sys /opt/ood
172M /var/www/ood/apps/sys
256M /opt/ood

New shared OnDemand gem sets

The new ondemand-gems RPMs allow other Passenger apps to be developed using the same gems that the core OnDemand apps use. If you do scl enable ondemand and then gem list you can see all the gems available.

Because the ondemand-gems RPMs will include the version in the RPM name, it is possible to install multiple versions of these RPMs on the same system. For example, “ondemand-gems-1.7.10” is the name and “1.7.10-2.el7.x86_64.rpm” is the version of ondemand-gems-1.7.10-1.7.10-2.el7.x86_64.rpm. That way you can upgrade to newer versions of OnDemand but still have the ability to provide apps access to older gems by installing the older gem sets.

Having multiple versions of ondemand-gems RPMs ensure that any custom apps relying on gems provided by OnDemand will not break as OnDemand is upgraded as previous installs of ondemand-gems will remain installed when OnDemand is upgraded.