Creating a Status App¶

Overview of App¶

We will make a copy of a status app that displays the running Passenger processes on the OnDemand host. We will use this as a starting point to create a new status app that displays quota information in a table.

The app we will be copying is: https://github.com/OSC/ood-example-ps. Running this app looks like:

../../_images/app-dev-tutorial-ps-to-quota-1.png

Fig. 7 What app looks like after cloning and launching.¶

After this tutorial the resulting app will be:

../../_images/app-dev-tutorial-ps-to-quota-2.png

Fig. 8 What app looks like after modifying in this tutorial.¶

This assumes you have followed the directions to Enabling App Development on the Dashboard.

The app uses the custom branded Bootstrap 3 that Job Composer and Active Jobs apps use.
The navbar contains a link back to the dashboard.
On a request, the app runs a shell command, parses the output, and displays the result in a table.
It is built in Ruby using the Sinatra framework, a lightweight web framework similar to Python’s Flask and Node.js’s Express

Benefits¶

This serves as a good starting point for any status app to build for OnDemand, because

the app has the branding matching other OnDemand apps
all status apps will do something similar on a request to the app:
1. get raw data from a shell command or http request
2. parse the raw data into an intermediate object representation
3. use that intermediate object representation to display the data formatted as a table or graph
the app can be deployed without requiring a build step because gem dependencies (specified in Gemfile and Gemfile.lock) are pure ruby and match those that are provided by the ondemand-gems rpm
most of the app can be modified without requiring a restart due to proper use of Sinatra reloader extension
app has a built in scaffold for unit testing using minitest

OnDemand System Gems¶

This app is able to run in OnDemand 1.8+ without installing the gems specified in the Gemfile.

All pre-installed Ruby gems used by OnDemand are available to make it easier to develop simple apps. These include gems used by this example app:

sinatra
sinatra-contrib
erubi

On the OnDemand web host, you can execute the command source scl_source enable ondemand and then gem list to see all available gems. These gems are provided by a separate ondemand-gems rpm that is installed when you do yum install ondemand. The name of the RPM includes the OnDemand release version, such as ondemand-gems-1.7.12-1.7.12-1.el7.x86_64.rpm. This ensures that if you do yum update this gem will not be removed - so apps can depend on the presence of these gems.

Files and Their Purpose¶

Table 8 Main files¶
File	Description
`config.ru`	entry point of the Passenger Ruby app
`app.rb`	Sinatra app config and routes; this in a separate file from `config.ru` so that code reloading will work
`command.rb`	class that defines an AppProcess struct, executes `ps`, and parses the output of the ps command producing an array of structs
`test/test_command.rb`	a unit test of the parsing code
`views/index.html`	the main section of the html page template using an implementation of ERB called erubi which auto-escapes output of ERB tags by default (for security)
`views/layout.html`	the rendered HTML from `views/index.html` is inserted into this layout, where css and javascript files are included

Table 9 Other files¶
File	Description
`Gemfile`, `Gemfile.lock`	defines gem dependencies for the app (see Bundler’s Rationale)
`tmp/`	tmp directory is kept so its easier to `touch tmp/restart.txt` when you want to force Passenger to restart an app
`public/`	serve up static assets like Bootstrap css; in OnDemand, NGINX auto-serves all files under public/ directly, without going through the Passenger process, which makes this much faster; as a result, each static file is in a directory with an explicit version number, so if these files ever change we change the version, which is one cache busting strategy
`Rakefile`	this provides a default `rake` task for running the automated tests under `test/`, so you can run the tests by running the command `rake`
`test/minitest_helper.rb`	contains setup code common between all tests
`vendor/bundle`	This directory is added if you execute `bin/bundle install --path vendor/bundle` to store app specific gems. This is necessary if you want to add gems or specify specific gem versions used by the app that deviate from those provided by system gemset, or if you are using OnDemand 1.7 or earlier.

Clone and Setup¶

Login to Open OnDemand, click “Develop” dropdown menu and click the “My Sandbox Apps (Development)” option.
Click “New App” and “Clone Existing App”.
Fill out the form:
1. Directory name: quota
2. Git remote: https://github.com/OSC/ood-example-ps
3. Check “Create new Git Project from this?”
4. Click Submit
Launch the app by clicking the large blue Launch button. In a new browser window/tab you will see the output of a ps command filtered using grep.
Switch browser tab/windows back to the dashboard Details view of the app and click the Files button on the right to open the app’s directory in the File Explorer.

Edit to Run and Parse Quota¶

The app runs and parses this command:

ps aux | grep '[A]pp'

We will change it to run and parse this command:

quota -spw

Update `test/test_command.rb`¶

Run the command to get example data. Copy and paste the output into the test, and update the assertions to expect an array of “quotas” instead of “processes” with appropriate attributes.

Diff:

  def test_command_output_parsing
    output = <<-EOF
-
-efranz    30328  0.1  0.1 462148 28128 ?        Sl   20:28   0:00 Passenger RackApp: /users/PZS0562/efranz/ondemand/dev/quota
-
+Disk quotas for user efranz (uid 10851):
+     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
+10.11.200.32:/PZS0562/  99616M    500G    500G       0    933k   1000k   1000k       0
EOF
-    processes = Command.new.parse(output)
+    quotas = Command.new.parse(output)

-    assert_equal 1, processes.count
+    assert_equal 1, quotas.count, "number of structs parsed should equal 1"

-    p = processes.first
+    q = quotas.first

-    assert_equal "efranz", p.user
-    assert_equal "462148", p.vsz
-    assert_equal "28128", p.rss
-    assert_equal "0:00", p.time
-    assert_equal "Passenger RackApp: /users/PZS0562/efranz/ondemand/dev/quota", p.command
+    assert_equal "10.11.200.32:/PZS0562/", q.filesystem, "expected filesystem value not correct"
+    assert_equal "99616M", q.blocks, "expected blocks value not correct"
+    assert_equal "500G", q.blocks_limit, "expected blocks_limit value not correct"
+    assert_equal "933k", q.files, "expected files value not correct"
+    assert_equal "0", q.files_grace, "expected files_grace value not correct"
  end

Resulting test method:

class TestCommand < Minitest::Test

  def test_command_output_parsing
    output = <<-EOF
Disk quotas for user efranz (uid 10851):
    Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
10.11.200.32:/PZS0562/  99616M    500G    500G       0    933k   1000k   1000k       0
EOF
    quotas = Command.new.parse(output)

    assert_equal 1, quotas.count, "number of structs parsed should equal 1"

    q = quotas.first

    assert_equal "10.11.200.32:/PZS0562/", q.filesystem, "expected filesystem value not correct"
    assert_equal "99616M", q.blocks, "expected blocks value not correct"
    assert_equal "500G", q.blocks_limit, "expected blocks_limit value not correct"
    assert_equal "933k", q.files, "expected files value not correct"
    assert_equal "0", q.files_grace, "expected files_grace value not correct"
  end
end

Update `command.rb`¶

Run the test by running the rake command and you will see it fail:

$ rake
Run options: --seed 58990

# Running:

F

Finished in 0.000943s, 1060.4569 runs/s, 1060.4569 assertions/s.

  1) Failure:
TestCommand#test_command_output_parsing [/users/PZS0562/efranz/ondemand/dev/quota/test/test_command.rb:14]:
number of structs parsed should equal 1.
Expected: 1
  Actual: 3

1 runs, 1 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1)

Tasks: TOP => default => test
(See full trace by running task with --trace)

Warning

To run commands like rake through the shell you need to make sure you are on a host that has the correct version of Ruby installed. For OnDemand that likely means using Software Collections with the same packages used to install OnDemand.

With SCL, running rake with ondemand SCL package looks like:

scl enable ondemand -- rake

You can avoid this by loading the SCL packages in your .bashrc or .bash_profile file. For example, in my .bash_profile I have:

if [[ ${HOSTNAME%%.*} == webtest04*  ]]
then
  scl enable ondemand -- bash
fi

This means when I login to the host webtest04.osc.edu the SCL packages will be enabled in a new bash session. If you did the same you would replace webtest04 with the hostname of the node you are developing on.

To get the unit test to pass we need to:

Change the command we are using.
Fix the command output parsing.
Fix the struct definition.

class Command
  def to_s
-    "ps aux | grep '[A]pp'"
+    "quota -spw"
  end

-  AppProcess = Struct.new(:user, :pid, :pct_cpu, :pct_mem, :vsz, :rss, :tty, :stat, :start, :time, :command)
+  Quota = Struct.new(:filesystem, :blocks, :blocks_quota, :blocks_limit, :blocks_grace, :files, :files_quota, :files_limit, :fil

  # Parse a string output from the `ps aux` command and return an array of
  # AppProcess objects, one per process
  def parse(output)
    lines = output.strip.split("\n")
-    lines.map do |line|
-      AppProcess.new(*(line.split(" ", 11)))
+    lines.drop(2).map do |line|
+      Quota.new(*(line.split))
    end
  end

After the changes part of the command.rb will look like this:

class Command
  def to_s
    "quota -spw"
  end

  Quota = Struct.new(:filesystem, :blocks, :blocks_quota, :blocks_limit, :blocks_grace, :files, :files_quota, :files_limit, :files_grace)

  # Parse a string output from the `ps aux` command and return an array of
  # AppProcess objects, one per process
  def parse(output)
    lines = output.strip.split("\n")
    lines.drop(2).map do |line|
      Quota.new(*(line.split))
    end
  end

Now when we run the test they pass:

$ rake
Run options: --seed 60317

# Running:

.

Finished in 0.000966s, 1035.1494 runs/s, 6210.8963 assertions/s.

1 runs, 6 assertions, 0 failures, 0 errors, 0 skips

Update `app.rb` and `view/index.html`¶

Update app.rb:

helpers do
  def title
-    "Passenger App Processes"
+    "Quota"
  end
end

# Define a route at the root '/' of the app.
get '/' do
  @command = Command.new
-  @processes, @error = @command.exec
+  @quotas, @error = @command.exec

  # Render the view
  erb :index
end

In views/index.erb, replace the table with this:

<table class="table table-bordered">
  <tr>
    <th>Filesystem</th>
    <th>Blocks</th>
    <th>Blocks Quota</th>
    <th>Blocks Limit</th>
    <th>Blocks Grace</th>
    <th>Files</th>
    <th>Files Quota</th>
    <th>Files Limit</th>
    <th>Files Grace</th>
  </tr>
  <% @quotas.each do |quota| %>
  <tr>
    <td><%= quota.filesystem %></td>
    <td><%= quota.blocks %></td>
    <td><%= quota.blocks_quota %></td>
    <td><%= quota.blocks_limit %></td>
    <td><%= quota.blocks_grace %></td>
    <td><%= quota.files %></td>
    <td><%= quota.files_quota %></td>
    <td><%= quota.files_limit %></td>
    <td><%= quota.files_grace %></td>
  </tr>
  <% end %>
</table>

These changes should not require an app restart. Go to the launched app and reload the page to see the changes.

Brand App¶

The app is looking good, but the details page still shows the app title “Passenger App Processes”. To change this and the icon, edit the manifest.yml:

-name: Passenger App Processes
-description: Display your running Passenger app processes in a table
+name: Quota
+description: Display quotas
+icon: fa://hdd-o

The icon follows format of fa://{FONTAWESOMENAME} where you replace {FONTAWESOMENAME} with an icon from https://fontawesome.com/icons/. In this case we are using fa-hdd-o which we write in the manifest as fa://hdd-o. You can see details on this icon at https://fontawesome.com/icons/hdd?style=regular

Publish App¶

Publishing an app requires two steps:

Updating the manifest.yml to specify the category and optionally subcategory, which indicates where in the dashboard menu the app appears.
Having an administrator checkout a copy of the production version to a directory under /var/www/ood/apps/sys.

Steps:

Add category to manifest so the app appears in the Files menu:

name: Quota
description: Display quotas
icon: fa://hdd-o
+category: Files
+subcategory: Utilities

Version these changes. Click Shell button on app details view, and then commit the changes:

git add .
git commit -m "update manifest for production"

# if there is an external remote associated with this, push to that
git push origin master

As the admin, sudo copy or git clone this repo to production

# as sudo on OnDemand host:
cd /var/www/ood/apps/sys
git clone /users/PZS0562/efranz/ondemand/dev/quota

Reload the dashboard.

../../_images/app-dev-tutorial-ps-to-quota-published.png

Fig. 9 Every user can now launch the Quota from the Files menu.¶

Warning

Accessing this new app for the first time will cause your NGINX server to restart, killing all websocket connections, which means resetting your active web-based OnDemand Shell sessions.