Set up local Sourcegraph monitoring development
This guide documents how to spin up and develop Sourcegraph's monitoring stack locally. Sourcegraph employees should also refer to the handbook's monitoring section for Sourcegraph-specific documentation. The developing observability page contains relevant documentation as well, including background about the components listed here.
Running monitoring components
With all services
The monitoring stack is not included in
sg start (or
sg start enterprise) scripts.
It needs to be started separately with
sg start monitoring.
Learn more about these in the general development getting started guide.
Without all services
For convenience, there are a number of ways to spin up Sourcegraph's monitoring services without having to start up every other service as well.
You can follow the instructions below for spinning up individual monitoring components, or use one of the following:
sg start monitoring: Spin up just monitoring components
sg start monitoring-alerts: Spin up frontend components as well as some monitoring components to test out the alerting integration.
Running just Grafana is a convenient way to validate dashboards.
When doing so, you may wish to connect Grafana to a remote Prometheus instance that you have administrator access to (such as Sourcegraph's instances), to show more real data than is available on your dev server.
For Kubernetes deployments, you can accomplish this by creating a
sg.config.overwrite.yaml file that replaces your local Prometheus instance with a
kubectl command that port-forwards traffic from the Prometheus service on the Kubernetes cluster that you're currently connected to:
# sg.config.overwrite.yaml commands: prometheus: # install can just be set up gcloud credentials for a cluster # e.g. https://handbook.sourcegraph.com/departments/product-engineering/engineering/process/deployments/instances install: gcloud container clusters get-credentials ... # make remote prometheus accessible to local grafana cmd: kubectl port-forward svc/prometheus 9090:30090 monitoring-generator: env: # don't reload your production Prometheus! PROMETHEUS_DIR: ''
Then, you can start up the local dev monitoring stack by using:
sg start monitoring
Grafana dashboards will be available at
Note that instead of
kubectl, you can replace the command in the
sg.config.overwrite.yaml above to use whichever port-forwarding mechanism you wish to use to connect to a remote Prometheus instance (as long as Prometheus is available on port
The dev targets for Grafana are defined in the following files:
Running just Prometheus is a convenient way to validate the generated recording and alert rules. You can start up a standalone Prometheus using:
sg run prometheus
The loaded generated recording and alert rules are available at
The bundled Alertmanager is available at
Some configuration options are available:
true, disables the prom-wrapper's integration with the Sourcegraph frontend.
true, disables the bundled Alertmanager entirely. This includes the behaviour of
Note that without services to scrape, running a standalone Prometheus will not provide any metrics—if you need to test metrics as well, you should start all services instead. The dev targets for Prometheus are defined in the following files:
The Sourcegraph Prometheus service features an integration with the Sourcegraph frontend that requires a frontend instance to be running to develop or test these features. Note that the Prometheus service will still run without additional configuration even if no frontend is accessible.
One way to do this is to start up Prometheus alongside all Sourcegraph services. You can alternatively spin up just the frontend separately:
sg run enterprise-frontend # or: sg run frontend
This should be sufficient to access the frontend API and the admin console (
/site-admin), which is where most of the integration is.
The docsite is used to serve generated monitoring documentation, such as the alert solutions reference. You can spin it up by running:
sg run docsite
Learn more about docsite development in the product documentation implementation guide.
Using the monitoring generator
The dev startup scripts used in this guide all mount relevant configuration directories into each monitoring service. This means that you can:
- Update your monitoring definitions
- Run the generator to regenerate and reload monitoring services
- Validate the result of your changes immediately (for example, by checking Prometheus rules in
/rulesor Grafana dashboards in
To run the generator and trigger a reload on changes:
sg run monitoring-generator
Make sure to provide the following parameters as well, where relevant:
GRAFANA_DIR='', if you are not running Grafana
PROMETHEUS_DIR='', if you are not running Prometheus
SRC_LOG_LEVEL=dbugto enable potentially helpful output for debugging issues