Set up local Sourcegraph monitoring development
This guide documents how to spin up and develop Sourcegraph's monitoring stack locally.Sourcegraph employees should also refer to the handbook's monitoring section for Sourcegraph-specific documentation.The developing observability page contains relevant documentation as well, including background about the components listed here.
Running monitoring components
With all services
The monitoring stack is included in the
./enterprise/dev/start.sh scripts.Learn more about these in the general development getting started guide.
Without all services
For convenience, there are a number of ways to spin up Sourcegraph's monitoring services without having to start up every other service as well.
Running just Grafana is a convenient way to validate dashboards.When doing so, you may wish to connect Grafana to a remote Prometheus instance that you have administrator access to (such as Sourcegraph's instances), to show more real data than is available on your dev server.For Kubernetes deployments, you can do this by getting
kubectl connected to a Sourcegraph cluster and then port-forwarding Prometheus via:
kubectl port-forward svc/prometheus 9090:30090
Then, you can start up a standalone Grafana using:
Dashboards will be available at
Note that instead of
kubectl, you can use whichever port-forwarding mechanism you wish to connect to a remote Prometheus instance as well, as long as Prometheus is available on port
9090 locally.The dev targets for Grafana are defined in the following files:
Running just Prometheus is a convenient way to validate the generated recording and alert rules.You can start up a standalone Prometheus using:
The loaded generated recording and alert rules are available at
http://localhost:9090/rules.The bundled Alertmanager is available at
Some configuration options are available:
true, disables the prom-wrapper's integration with the Sourcegraph frontend.
true, disables the bundled Alertmanager entirely.This includes the behaviour of
Note that without services to scrape, running a standalone Prometheus will not provide any metrics - if you need to test metrics as well, you should start all services instead.The dev targets for Prometheus are defined in the following files:
The Sourcegraph Prometheus service features an integration with the Sourcegraph frontend that requires a frontend instance to be running to develop or test these features.Note that the Prometheus service will still run without additional configuration even if no frontend is accessible.
One way to do this is to start up Prometheus alongside all Sourcegraph services.You can alternatively spin up just the frontend separately:
./dev/start.sh --only frontend
This should be sufficient to access the frontend API and the admin console (
/site-admin), which is where most of the integration is.
The docsite is used to serve generated monitoring documentation, such as the alert solutions reference.You can spin it up by running:
Learn more about docsite development in the product documentation implementation guide.
Using the monitoring generator
The dev startup scripts used in this guide all mount relevant configuration directories into each monitoring service.This means that you can:
- Update your monitoring definitions
- Run the generator to regenerate and reload monitoring services
- Validate the result of your changes immediately (for example, by checking Prometheus rules in
/rulesor Grafana dashboards in
To run the generator and trigger a reload:
RELOAD=true go generate ./monitoring
Make sure to provide the following parameters as well, where relevant:
GRAFANA_DIR='', if you are not running Grafana
PROMETHEUS_DIR='', if you are not running Prometheus
SRC_LOG_LEVEL=dbugto enable potentially helpful output for debugging issues