Making ServiceNow expose metrics

3 minutes read
monitoring • servicenow • prometheus

image alt text

ServiceNow is widely used by many companies. I am consistently impressed by the capabilities of this versatile application. Broadly speaking, ServiceNow offers an interface that allows both business and IT users to request various items, ranging from laptops to more IT-specific needs like membership in Active Directory groups. When I integrate a service through a catalog item in ServiceNow, I prefer to keep a close watch on the activity related to that service.

Monitoring with Prometheus

My preferred monitoring tools are Prometheus and Grafana. To incorporate ServiceNow metrics into Prometheus, either ServiceNow must directly expose these metrics, or I need to develop a custom exporter that retrieves data from an available API endpoint. Both approaches are feasible, but having a service self-expose its metrics is generally simpler.

Scripted REST API to the rescue. It enables the developer to write custom api endpoints with JavaScript. This made me very happy, as this means freedom. If I can create a custom endpoint, I can make something that Prometheus can work with.

Getting Started

Go to Scripted Web Services > Scripted REST APIs (or go to a Scoped Application), and create a new Scripted REST API. I called mine metrics, as this is commonly the endpoint name in Prometheus exporters. Continue to create a new Resource, I called mine all, as this will be the script for the ‘metrics/’ endpoint. I took Incidents as an example, and chose to group the amount of incidents by their assignment group.

So what do we need for that? Something Prometheus can work with, like this:

# HELP metric_name Description of the metric
# TYPE metric_name metric_type
metric_name{label_name_1="Group",label_name_2="Closed",} 3.0

For my proof of concept, I looked at the examples and came up with this:

function createGaugeHeader(name, description, type) {
  var tmpl1 = "# HELP " + name + " " + description;
  var tmpl2 = "# TYPE " + type + "\n";
  return [tmpl1, tmpl2].join("\n");
}

function createGauge(name, value, tag) {
  var tmpl3 = name + tag + " " + (value).toFixed(1) + "\n";
  return tmpl3;
}

var groups = {};

function getAssignmentGroupById(id) {
  if (id in groups) {
    return groups[id];
  }
  var gr = new GlideRecord('sys_user_group');
  gr.get(id);
  var group = gr.getValue('name');
  groups[id] = group;
  return groups[id];
}

(function process( /*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
  var writer = response.getStreamWriter(),
    hdrs = {},
    table = 'incident',
    record_limit = 100,
    gr = new GlideRecord(table);


  hdrs['Content-Type'] = 'text/plain';
  response.setStatus(200);
  response.setHeaders(hdrs);

  gr.setLimit(record_limit);
  gr.query();


  var counts = {};

  while (gr.next()) {
    var group = getAssignmentGroupById(gr.assignment_group);
    if (group in counts) {
      counts[group] = counts[group] + 1;
    } else {
      counts[group] = 1;
    }
  }
  var name = "incidents_total";
  var description = "Total amount of incidents per assignment group.";
  var type = "gauge";

  var gaugeHeader = createGaugeHeader(name, description, type);

  writer.writeString(gaugeHeader);
  for (var key in counts) {
    var val = counts[key];
    var tag = '{assignment_group="' + key + '",}';
    var gaugeContent = createGauge(name, val, tag);
    writer.writeString(gaugeContent);
  }

})(request, response);

This results in:

# HELP incidents_total Total amount of incidents per assignment group.
# TYPE gauge
incidents_total{assignment_group="Network",} 6.0
incidents_total{assignment_group="null",} 24.0
incidents_total{assignment_group="Service Desk",} 12.0
incidents_total{assignment_group="Database",} 3.0
incidents_total{assignment_group="Hardware",} 9.0
incidents_total{assignment_group="Software",} 9.0
incidents_total{assignment_group="CAB Approval",} 1.0
incidents_total{assignment_group="Openspace",} 1.0

And thát is something Prometheus can work with.

Query used as example in the image: sum(incidents_total) by (assignment_group)