Sourcegraph Prometheus

We ship a custom Prometheus image as part of a standard Sourcegraph distribution. It currently bundles Alertmanager as well as integrations to the Sourcegraph web application. Learn more about it in our monitoring architecture.

Adding recording rules, alerts, etc. to this image is handled by the monitoring generator.

The image is defined in docker-images/prometheus.

Metrics

See the metrics and dashboards documentation.

To learn more about developing metrics, see the observability developer guides.

Prom-wrapper

The entrypoint of the image is a sidecar program called the prom-wrapper. Learn more about it here.

The source code for this program is currently kept in docker-images/prometheus/cmd/prom-wrapper.

To learn more about developing our observability stack, see the local Sourcegraph monitoring development guide.

Upgrading Prometheus or Alertmanager

To upgrade Prometheus or Alertmanager:

  1. Make the appropriate version and sum changes to the sourcegraph/prometheus Dockerfile
  2. Ensure no image update steps are required by checking upstream Dockerfiles where required as noted in the sourcegraph/prometheus Dockerfile where appropriate
  3. Upgrade the Alertmanager and Prometheus Go client dependencies where appropriate
    1. For the Alertmanager dependency, the fork needs to be upgraded to the appropriate version first: sourcegraph/alertmanager
  4. Ensure the image still builds: ./docker-images/prometheus/build.sh
  5. Run the monitoring stack locally and verify that:
    1. If upgrading Prometheus: all Prometheus rules are evaluated successfully (localhost:9090/rules)
    2. If upgrading Alertmanager: Alertmanager starts up correctly (localhost:9090/alertmanager/#/status), and observability.alerts can be configured via the Sourcegraph web application