Prometheus Monitoring

OnDemand provides the ondemand_exporter that allows for monitoring some metrics specific to OnDemand.

General metrics provided:

  • Number of active PUNs and type of apps running

  • Types of web connections for OnDemand

  • Aggregate PUN resource consumption

  • Passenger app resource consumption aggregated by app

Dependencies:

  • OnDemand >= 1.8

  • RPM Installs - ondemand-passenger >= 6.0.4-6

Install via RPM

For yum/dnf based systems the ondemand_exporter can be installed via RPM.

sudo yum install ondemand_exporter

The RPM will install the following files that should work out of the box:

  • RHEL/Rocky/AlmaLinux 8 & 9 only: /etc/httpd/conf.d/ondemand_exporter.conf

  • /etc/sudoers.d/ondemand_exporter

Ensure that the new Apache configuration is loaded by restarting Apache

sudo systemctl restart httpd

Start the service

sudo systemctl start ondemand_exporter

Install from Source

Check for the ondemand_exporter Latest Release version number. Replace VERSION with latest release version

Warning

If Passenger was not installed using ondemand-passenger RPM then it is required that before starting the ondemand_exporter daemon you must specify the path to passenger-status via the --path.passenger-status flag.

VERSION="0.10.0"
ARCHIVE="ondemand_exporter-${VERSION}.linux-amd64"
wget -O /tmp/${ARCHIVE}.tar.gz https://github.com/OSC/ondemand_exporter/releases/download/v${VERSION}/${ARCHIVE}.tar.gz
tar xf /tmp/${ARCHIVE}.tar.gz -C /tmp
sudo install -o root -g root -m 0755 /tmp/${ARCHIVE}/ondemand_exporter /usr/bin/ondemand_exporter
sudo install -o root -g root -m 0440 /tmp/${ARCHIVE}/files/sudo /etc/sudoers.d/ondemand_exporter
sudo install -o root -g root -m 0644 /tmp/${ARCHIVE}/files/ondemand_exporter.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo install -o root -g root -m 0440 /tmp/${ARCHIVE}/files/apache.conf /etc/httpd/conf.d/ondemand_exporter.conf
sudo systemctl restart httpd

(Optional) If Passenger was not installed via ondemand-passenger RPM. Adjust the path to passenger-status as needed

sudo mkdir /etc/systemd/system/ondemand_exporter.service.d
sudo cat > /etc/systemd/system/ondemand_exporter.service.d/passenger-status.conf <<'EOF'
[Service]
Environment="PASSENGER_STATUS=/usr/sbin/passenger-status"
EOF
sudo systemctl daemon-reload

Start the service

sudo systemctl start ondemand_exporter

Test Prometheus Exporter

By default the exporter listens on port 9301 and can be tested using curl.

curl http://localhost:9301/metrics

Prometheus Configuration

The following is an example of how to configure the Prometheus scrape if the OnDemand host is web.example.com and the OnDemand ServerName is ondemand.example.com.

- job_name: ondemand
  metrics_path: /metrics
  scrape_timeout: 20s
  scrape_interval: 2m
  static_configs:
  - targets:
    - web.example.com:9301
    labels:
      environment: production
      service: ondemand.example.com

Grafana Dashboard

An example Grafana Dashboard is available.

Process Exporter

If you’re site is using the Process Exporter with Prometheus the following is an example configuration that can be used to collect metrics similar to the ondemand_exporter about running processes for OnDemand.

process_names:
- name: ood-pun
  comm:
  - nginx
  - Passenger
  - Passenger NodeA
  - PassengerAgent
  - ruby
- name: "{{.Comm}}:{{.Username}}"
  cmdline:
  - ".+"

The above example only makes sense on a host that is only running OnDemand and not other services that might also be using NGINX, Passenger or Ruby.