Making ServiceNow expose metrics

3 minutes read
monitoring • servicenow • prometheus

image alt text

A lot of companies use ServiceNow. I’m regularly positively suprised by what is possible inside this multi-purpose application. In generic sense, ServiceNow provides an interface for business and IT users to request items like laptops, but also more IT related ‘items’ like membership of AD groups. If I expose a service through a catalog item in ServiceNow, I like to have my eyes on what’s happening with that service.

Monitoring with Prometheus

My go-to tools for monitoring are Prometheus and Grafana. If I want to have ServiceNow metrics in Prometheus, ServiceNow should expose those metrics or I need to write a custom exporter that pulls data from an exposed API endpoint. Both are viable options, but a service exposing its own metrics is the simplest one.

Scripted REST API to the rescue. It enables the developer to write custom api endpoints with JavaScript. This made me very happy, as this means freedom. If I can create a custom endpoint, I can make something that Prometheus can work with.

Getting Started

Go to Scripted Web Services > Scripted REST APIs (or go to a Scoped Application), and create a new Scripted REST API. I called mine metrics, as this is commonly the endpoint name in Prometheus exporters. Continue to create a new Resource, I called mine all, as this will be the script for the ‘metrics/’ endpoint. I took Incidents as an example, and chose to group the amount of incidents by their assignment group.

So what do we need for that? Something Prometheus can work with, like this:

# HELP metric_name Description of the metric
# TYPE metric_name metric_type
metric_name{label_name_1="Group",label_name_2="Closed",} 3.0

For my proof of concept, I looked at the examples and came up with this:

function createGaugeHeader(name, description, type) {
  var tmpl1 = "# HELP " + name + " " + description;
  var tmpl2 = "# TYPE " + type + "\n";
  return [tmpl1, tmpl2].join("\n");
}

function createGauge(name, value, tag) {
  var tmpl3 = name + tag + " " + (value).toFixed(1) + "\n";
  return tmpl3;
}

var groups = {};

function getAssignmentGroupById(id) {
  if (id in groups) {
    return groups[id];
  }
  var gr = new GlideRecord('sys_user_group');
  gr.get(id);
  var group = gr.getValue('name');
  groups[id] = group;
  return groups[id];
}

(function process( /*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
  var writer = response.getStreamWriter(),
    hdrs = {},
    table = 'incident',
    record_limit = 100,
    gr = new GlideRecord(table);


  hdrs['Content-Type'] = 'text/plain';
  response.setStatus(200);
  response.setHeaders(hdrs);

  gr.setLimit(record_limit);
  gr.query();


  var counts = {};

  while (gr.next()) {
    var group = getAssignmentGroupById(gr.assignment_group);
    if (group in counts) {
      counts[group] = counts[group] + 1;
    } else {
      counts[group] = 1;
    }
  }
  var name = "incidents_total";
  var description = "Total amount of incidents per assignment group.";
  var type = "gauge";

  var gaugeHeader = createGaugeHeader(name, description, type);

  writer.writeString(gaugeHeader);
  for (var key in counts) {
    var val = counts[key];
    var tag = '{assignment_group="' + key + '",}';
    var gaugeContent = createGauge(name, val, tag);
    writer.writeString(gaugeContent);
  }

})(request, response);

This results in:

# HELP incidents_total Total amount of incidents per assignment group.
# TYPE gauge
incidents_total{assignment_group="Network",} 6.0
incidents_total{assignment_group="null",} 24.0
incidents_total{assignment_group="Service Desk",} 12.0
incidents_total{assignment_group="Database",} 3.0
incidents_total{assignment_group="Hardware",} 9.0
incidents_total{assignment_group="Software",} 9.0
incidents_total{assignment_group="CAB Approval",} 1.0
incidents_total{assignment_group="Openspace",} 1.0

And thát is something Prometheus can work with.

Query used as example in the image: sum(incidents_total) by (assignment_group)