Creating a Status App

Overview of App

We will make a copy of a status app that displays the running Passenger processes on the OnDemand host. We will use this as a starting point to create a new status app that displays quota information in a table.

The app we will be copying is: https://github.com/OSC/ood-example-ps. Running this app looks like:

../../_images/app-dev-tutorial-ps-to-quota-1.png

Fig. 7 What app looks like after cloning and launching.

After this tutorial the resulting app will be:

../../_images/app-dev-tutorial-ps-to-quota-2.png

Fig. 8 What app looks like after modifying in this tutorial.

This assumes you have followed the directions to Enabling App Development on the Dashboard.

  1. The app uses the custom branded Bootstrap 3 that Job Composer and Active Jobs apps use.

  2. The navbar contains a link back to the dashboard.

  3. On a request, the app runs a shell command, parses the output, and displays the result in a table.

  4. It is built in Ruby using the Sinatra framework, a lightweight web framework similar to Python’s Flask and Node.js’s Express

Benefits

This serves as a good starting point for any status app to build for OnDemand, because

  1. the app has the branding matching other OnDemand apps

  2. all status apps will do something similar on a request to the app:

    1. get raw data from a shell command or http request

    2. parse the raw data into an intermediate object representation

    3. use that intermediate object representation to display the data formatted as a table or graph

  3. the app can be deployed without requiring a build step because gem dependencies (specified in Gemfile and Gemfile.lock) are pure ruby and match those that are provided by the ondemand-gems rpm

  4. most of the app can be modified without requiring a restart due to proper use of Sinatra reloader extension

  5. app has a built in scaffold for unit testing using minitest

OnDemand System Gems

This app is able to run in OnDemand 1.8+ without installing the gems specified in the Gemfile.

All pre-installed Ruby gems used by OnDemand are available to make it easier to develop simple apps. These include gems used by this example app:

  • sinatra

  • sinatra-contrib

  • erubi

On the OnDemand web host, you can execute the command source scl_source enable ondemand and then gem list to see all available gems. These gems are provided by a separate ondemand-gems rpm that is installed when you do yum install ondemand. The name of the RPM includes the OnDemand release version, such as ondemand-gems-1.7.12-1.7.12-1.el7.x86_64.rpm. This ensures that if you do yum update this gem will not be removed - so apps can depend on the presence of these gems.

Files and Their Purpose

Table 9 Main files

File

Description

config.ru

entry point of the Passenger Ruby app

app.rb

Sinatra app config and routes; this in a separate file from config.ru so that code reloading will work

command.rb

class that defines an AppProcess struct, executes ps, and parses the output of the ps command producing an array of structs

test/test_command.rb

a unit test of the parsing code

views/index.html

the main section of the html page template using an implementation of ERB called erubi which auto-escapes output of ERB tags by default (for security)

views/layout.html

the rendered HTML from views/index.html is inserted into this layout, where css and javascript files are included

Table 10 Other files

File

Description

Gemfile, Gemfile.lock

defines gem dependencies for the app (see Bundler’s Rationale)

tmp/

tmp directory is kept so its easier to touch tmp/restart.txt when you want to force Passenger to restart an app

public/

serve up static assets like Bootstrap css; in OnDemand, NGINX auto-serves all files under public/ directly, without going through the Passenger process, which makes this much faster; as a result, each static file is in a directory with an explicit version number, so if these files ever change we change the version, which is one cache busting strategy

Rakefile

this provides a default rake task for running the automated tests under test/, so you can run the tests by running the command rake

test/minitest_helper.rb

contains setup code common between all tests

vendor/bundle

This directory is added if you execute bin/bundle install --path vendor/bundle to store app specific gems. This is necessary if you want to add gems or specify specific gem versions used by the app that deviate from those provided by system gemset, or if you are using OnDemand 1.7 or earlier.

Clone and Setup

  1. Login to Open OnDemand, click “Develop” dropdown menu and click the “My Sandbox Apps (Development)” option.

  2. Click “New App” and “Clone Existing App”.

  3. Fill out the form:

    1. Directory name: quota

    2. Git remote: https://github.com/OSC/ood-example-ps

    3. Check “Create new Git Project from this?”

    4. Click Submit

  4. Launch the app by clicking the large blue Launch button. In a new browser window/tab you will see the output of a ps command filtered using grep.

  5. Switch browser tab/windows back to the dashboard Details view of the app and click the Files button on the right to open the app’s directory in the File Explorer.

Edit to Run and Parse Quota

The app runs and parses this command:

ps aux | grep '[A]pp'

We will change it to run and parse this command:

quota -spw

Update test/test_command.rb

Run the command to get example data. Copy and paste the output into the test, and update the assertions to expect an array of “quotas” instead of “processes” with appropriate attributes.

Diff:

  def test_command_output_parsing
    output = <<-EOF
-
-efranz    30328  0.1  0.1 462148 28128 ?        Sl   20:28   0:00 Passenger RackApp: /users/PZS0562/efranz/ondemand/dev/quota
-
+Disk quotas for user efranz (uid 10851):
+     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
+10.11.200.32:/PZS0562/  99616M    500G    500G       0    933k   1000k   1000k       0
EOF
-    processes = Command.new.parse(output)
+    quotas = Command.new.parse(output)

-    assert_equal 1, processes.count
+    assert_equal 1, quotas.count, "number of structs parsed should equal 1"

-    p = processes.first
+    q = quotas.first

-    assert_equal "efranz", p.user
-    assert_equal "462148", p.vsz
-    assert_equal "28128", p.rss
-    assert_equal "0:00", p.time
-    assert_equal "Passenger RackApp: /users/PZS0562/efranz/ondemand/dev/quota", p.command
+    assert_equal "10.11.200.32:/PZS0562/", q.filesystem, "expected filesystem value not correct"
+    assert_equal "99616M", q.blocks, "expected blocks value not correct"
+    assert_equal "500G", q.blocks_limit, "expected blocks_limit value not correct"
+    assert_equal "933k", q.files, "expected files value not correct"
+    assert_equal "0", q.files_grace, "expected files_grace value not correct"
  end

Resulting test method:

class TestCommand < Minitest::Test

  def test_command_output_parsing
    output = <<-EOF
Disk quotas for user efranz (uid 10851):
    Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
10.11.200.32:/PZS0562/  99616M    500G    500G       0    933k   1000k   1000k       0
EOF
    quotas = Command.new.parse(output)

    assert_equal 1, quotas.count, "number of structs parsed should equal 1"

    q = quotas.first

    assert_equal "10.11.200.32:/PZS0562/", q.filesystem, "expected filesystem value not correct"
    assert_equal "99616M", q.blocks, "expected blocks value not correct"
    assert_equal "500G", q.blocks_limit, "expected blocks_limit value not correct"
    assert_equal "933k", q.files, "expected files value not correct"
    assert_equal "0", q.files_grace, "expected files_grace value not correct"
  end
end

Update command.rb

Run the test by running the rake command and you will see it fail:

$ rake
Run options: --seed 58990

# Running:

F

Finished in 0.000943s, 1060.4569 runs/s, 1060.4569 assertions/s.

  1) Failure:
TestCommand#test_command_output_parsing [/users/PZS0562/efranz/ondemand/dev/quota/test/test_command.rb:14]:
number of structs parsed should equal 1.
Expected: 1
  Actual: 3

1 runs, 1 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => default => test
(See full trace by running task with --trace)

Warning

To run commands like rake through the shell you need to make sure you are on a host that has the correct version of Ruby installed. For OnDemand that likely means using Software Collections with the same packages used to install OnDemand.

With SCL, running rake with ondemand SCL package looks like:

scl enable ondemand -- rake

You can avoid this by loading the SCL packages in your .bashrc or .bash_profile file. For example, in my .bash_profile I have:

if [[ ${HOSTNAME%%.*} == webtest04*  ]]
then
  scl enable ondemand -- bash
fi

This means when I login to the host webtest04.osc.edu the SCL packages will be enabled in a new bash session. If you did the same you would replace webtest04 with the hostname of the node you are developing on.

To get the unit test to pass we need to:

  1. Change the command we are using.

  2. Fix the command output parsing.

  3. Fix the struct definition.

class Command
  def to_s
-    "ps aux | grep '[A]pp'"
+    "quota -spw"
  end

-  AppProcess = Struct.new(:user, :pid, :pct_cpu, :pct_mem, :vsz, :rss, :tty, :stat, :start, :time, :command)
+  Quota = Struct.new(:filesystem, :blocks, :blocks_quota, :blocks_limit, :blocks_grace, :files, :files_quota, :files_limit, :fil

  # Parse a string output from the `ps aux` command and return an array of
  # AppProcess objects, one per process
  def parse(output)
    lines = output.strip.split("\n")
-    lines.map do |line|
-      AppProcess.new(*(line.split(" ", 11)))
+    lines.drop(2).map do |line|
+      Quota.new(*(line.split))
    end
  end

After the changes part of the command.rb will look like this:

class Command
  def to_s
    "quota -spw"
  end

  Quota = Struct.new(:filesystem, :blocks, :blocks_quota, :blocks_limit, :blocks_grace, :files, :files_quota, :files_limit, :files_grace)

  # Parse a string output from the `ps aux` command and return an array of
  # AppProcess objects, one per process
  def parse(output)
    lines = output.strip.split("\n")
    lines.drop(2).map do |line|
      Quota.new(*(line.split))
    end
  end

Now when we run the test they pass:

$ rake
Run options: --seed 60317

# Running:

.

Finished in 0.000966s, 1035.1494 runs/s, 6210.8963 assertions/s.

1 runs, 6 assertions, 0 failures, 0 errors, 0 skips

Update app.rb and view/index.html

Update app.rb:

helpers do
  def title
-    "Passenger App Processes"
+    "Quota"
  end
end

# Define a route at the root '/' of the app.
get '/' do
  @command = Command.new
-  @processes, @error = @command.exec
+  @quotas, @error = @command.exec

  # Render the view
  erb :index
end

In views/index.erb, replace the table with this:

<table class="table table-bordered">
  <tr>
    <th>Filesystem</th>
    <th>Blocks</th>
    <th>Blocks Quota</th>
    <th>Blocks Limit</th>
    <th>Blocks Grace</th>
    <th>Files</th>
    <th>Files Quota</th>
    <th>Files Limit</th>
    <th>Files Grace</th>
  </tr>
  <% @quotas.each do |quota| %>
  <tr>
    <td><%= quota.filesystem %></td>
    <td><%= quota.blocks %></td>
    <td><%= quota.blocks_quota %></td>
    <td><%= quota.blocks_limit %></td>
    <td><%= quota.blocks_grace %></td>
    <td><%= quota.files %></td>
    <td><%= quota.files_quota %></td>
    <td><%= quota.files_limit %></td>
    <td><%= quota.files_grace %></td>
  </tr>
  <% end %>
</table>

These changes should not require an app restart. Go to the launched app and reload the page to see the changes.

Brand App

The app is looking good, but the details page still shows the app title “Passenger App Processes”. To change this and the icon, edit the manifest.yml:

-name: Passenger App Processes
-description: Display your running Passenger app processes in a table
+name: Quota
+description: Display quotas
+icon: fa://hdd-o

Publish App

Publishing an app requires two steps:

  1. Updating the manifest.yml to specify the category and optionally subcategory, which indicates where in the dashboard menu the app appears.

  2. Having an administrator checkout a copy of the production version to a directory under /var/www/ood/apps/sys.

Steps:

  1. Add category to manifest so the app appears in the Files menu:

    name: Quota
    description: Display quotas
    icon: fa://hdd-o
    +category: Files
    +subcategory: Utilities
    
  2. Version these changes. Click Shell button on app details view, and then commit the changes:

    git add .
    git commit -m "update manifest for production"
    
    # if there is an external remote associated with this, push to that
    git push origin master
    
  3. As the admin, sudo copy or git clone this repo to production

    # as sudo on OnDemand host:
    cd /var/www/ood/apps/sys
    git clone /users/PZS0562/efranz/ondemand/dev/quota
    
  4. Reload the dashboard.

../../_images/app-dev-tutorial-ps-to-quota-published.png

Fig. 9 Every user can now launch the Quota from the Files menu.

Warning

Accessing this new app for the first time will cause your NGINX server to restart, killing all websocket connections, which means resetting your active web-based OnDemand Shell sessions.