Alerts

Alerts can be configured to notify site admins when there is something wrong or noteworthy on the Sourcegraph instance.

Alerts are configured in Grafana. (Prometheus Alertmanager may also be used, but this documentation prefers Grafana.)

Setting up alerting

Sourcegraph 3.17+

Visit your site configuration (e.g. https://sourcegraph.example.com/site-admin/configuration) to configure alerts using the observability.alerts field. As always, you can use Ctrl+Space at any time to get hints about allowed fields as well as relevant documentation inside the configuration editor.

Configured alerts will automatically create notifiers in Grafana and attach them to relevant alerts.

Examples

Slack

"observability.alerts": [
  {
    "id": "my-unique-alert",
    "level": "critical",
    "notifier": {
      "type": "slack",
      // Slack incoming webhook URL.
      "url": "https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxx",
      // Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer
      // to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the
      // overflow button on each user’s Slack profile).
      "mentionUsers": "U0XXXXX,U0XXXXXX"
    }
  }
]

PagerDuty

"observability.alerts": [
  {
    "id": "my-unique-alert",
    "level": "critical",
    "notifier": {
      "type": "pagerduty",
      // Integration key for PagerDuty.
      "integrationKey": "XXXXXXXX"
    }
  }
]

Webhook

"observability.alerts": [
  {
    "id": "my-unique-alert",
    "level": "critical",
    "notifier": {
      "type": "webhook",
      // Webhook URL.
      "url": "https://my.webhook.url"
    }
  }
]

Before 3.17: Configure alert channels in Grafana

Before configuring specific alerts in Grafana, you must set up alert channels. Each channel corresponds to an external service to which Grafana will push alerts.

1. Access Grafana directly as a Grafana admin:
2. Follow these instructions to access Grafana directly (instead of going through the Sourcegraph Site Admin Monitoring page as usual).
3. Navigate to the Grafana /login URL (e.g., http://localhost:3070/-/debug/grafana/login or append /login to your Grafana direct access URL if different). If you are doing this for the first time, user the username and password admin and admin.
4. In the left sidebar, click the bell icon 🔔 and select Notification channels.
5. Click New channel and then specify the settings of your channel. The Type field selects the type of external service (e.g., PagerDuty, Slack, Email). Some service types will require additional configuration in the service itself. Here are some examples:
6. Slack
   1. Go to https://api.slack.com/apps to create a new Slack App.
   2. Click Create an App, give the app a name, and click Create App.
   3. Click Incoming Webhooks and toggle on Activate Incoming Webhooks.
   4. Click Add New Webhook to Workspace.
   5. Pick the channel to which this Slack App will post.
   6. Back on the Grafana New Notification Channel page, copy the webhook URL to the Url field.
7. PagerDuty
   1. Go to https://app.pagerduty.com/developer/apps.
   2. Click Create New App. Give the app a name and decription, and set the category to "Application Performance Management". For "Would you like to publish the app for all PagerDuty users?", select "No". Click Save.
   3. On the Configure App page, under Functionality > Events Integration, click Add.
   4. On the Event Integration page, under Events Integration Test > Create a Test Service, enter a name and click Create.
   5. Copy the Integration Key. Click Save and then Save again on the Configure App page.
   6. Back in the Grafana New Notification Channel page, paste the Integration Key into the Integration Key field.
8. After you have specified the settings on the Grafana New Notification Channel page, click Send Test to send a test notification. You should receive a notification from Grafana via your specified channel. If this worked, click Save.

Before 3.17: Set up an individual alert

After adding the appropriate notification channels, configure individual alerts to notify those channels.

1. Navigate to the dashboard with the panel and metric for which you'd like to configure an alert.
   1. Make sure the dashboard is not read-only (the default Sourcegraph-provided dashboards are read-only, because they are provisioned from disk). If the dashboard is read-only, go to the dashboard settings (the gear icon in the upper right) and click Save As.. to create a writeable copy.
2. The panel title has a small dropdown next to it. Click the dropdown icon and select Edit.
3. In the left sidebar, choose the bell icon 🔔 for Alert.
4. Fill out the fields for the alert rule and select a notification channel.
5. Verify your rule by clicking Test Rule or viewing State History.
6. Return to the dashboard page by clicking the left arrow in the upper left. Save the dashboard by clicking the save icon in the upper right.

Understanding alerts

See alert solutions for possible solutions when alerts are firing.