Sourcegraph monitoring generator

The monitoring generator manages converting monitoring definitions into integrations with Sourcegraph's monitoring ecosystem.

Its purpose is to help enable a cohesive observability experience for site administrators, codify Sourcegraph’s monitoring pillars, and make it easy for developers to add monitoring for their Sourcegraph services by generating integrations with Sourcegraph’s monitoring ecosystem for free.

Reference

Features

Documentation generation

The generator automatically creates documentation from monitoring definitions that customers and engineers can reference. These include:

Links to generated documentation can be provided in our other generated integrations—for example, Slack alerts will provide a link to the appropriate alert solutions entry, and Grafana panels will link to the appropriate dashboards reference entry.

Grafana integration

The generator automatically generates and ships dashboards from monitoring definitions within the Sourcegraph Grafana distribution.

It also takes care of the following:

  • Graphs within rows are sized appropriately
  • Alerts visualization through the ObservableAlertDefinition API:
    • Overview graphs for alerts (both Sourcegraph-wide and per-service)
    • Threshold lines for alerts of all levels are rendered in graphs
  • Formatting of units, labels, and more (using either the defaults, or the ObservablePanelOption API)
  • Providing links to generated documentation
  • Configuring annotation layers for alert events and version changes
  • Maintaining a uniform look and feel across all dashboards

Links to generated documentation can be provided in our other generated integrations—for example, Slack alerts will provide a link to the appropriate service’s dashboard.

Prometheus integration

The generator automatically generates and ships Prometheus recording rules and alerts within the Sourcegraph Prometheus distribution. This include the following, all with appropriate and consistent labels:

Generated Prometheus recording rules are leveraged by the Grafana integration.

Alertmanager integration

The generator’s Prometheus integration is a critical part of the Sourcegraph’s alerting capabilities, which handles alert routing by level and formatting of alert messages to include links to documentation and dashboards. Learn more about using Sourcegraph alerting in the alerting documentation. This is possible due to the labels generated by the Prometheus integration.

At Sourcegraph, extended routing based on team ownership (as defined by ObservableOwner) is also used to route customer support requests and on-call events through OpsGenie.