Troubleshooting guide

This is a guide for investigating errors and performance issues, with the goal of resolving theissue or generating a high-quality issue report.

How to use this guide:

1. Scan through Specific scenarios to see if any of these applies to you. Ifany do, follow the instructions in the subsection.
2. Scan through General scenarios to find which scenario(s) applies to you. If any do,follow the instructions to update your instance or collect information for the issue report.
3. If you cannot resolve the issue on your own, file an issue on the Sourcegraph issuetracker with the collectedinformation. Enterprise customers may alternatively file a support ticket or email[email protected].

Specific scenarios

Scenario: search and code pages take a long time to load

If this is the case, this could indicate high gitserver load. To confirm, take the following steps:

1. Open Grafana.
2. If using Sourcegraph 3.14: Simply check if either of these alerts are firing:
   • gitserver: 50+ concurrent command executions (abnormally high load)
   • gitserver: echo command execution duration exceeding 1s
3. If using an older version of Sourcegraph:
   • Go to the Sourcegraph Internal > Gitserver rev2 dashboard.
   • Examine the "Echo Duration Seconds" dashboard (tracks the src_gitserver_echo_duration_seconds metric) and "Commands running concurrently" dashboard (tracks the src_gitserver_exec_running metric). If either of these is high (> 1s echo duration or 100s simultaneous execs), then thisindicates gitserver is under heavy load and likely the bottleneck.
4. Confirm your gitserver is not under-provisioned, by e.g. comparing its allocated resources with what the resource estimator shows.

Solution: set USE_ENHANCED_LANGUAGE_DETECTION=false in the Sourcegraph runtimeenvironment.

Scenario: no cloning, syncing, updating or deleting is happening

Observed state: Sourcegraph instance does not react to any updates to code hosts and no cloning is happening.The cause of this state could be repo-updater queries that are too large for the limits of the running Postgres DB.One symptom is seeing a line like the one below in the repo-updater logs:

t=2020-05-28T18:41:02+0000 lvl=eror msg=Syncer error="syncer.sync.store.upsert-repos: delete: driver: bad connection

or seeing the same error in the "Code host status panel" (Clicking the cloud icon).

The fix is to increase the memory on Postgres DB which will increase certain Postgres-internal limits and will allowthe queries from repo-updater to go through.

Another cause could be that the repo-updater is in a crash loop for some reason. If there are large numbers of reposto be updated it could be from Out of memory errors. A fix here is to increase the memory for repo-updater instead.

General scenarios

This section contains a list of scenarios, each of which contains instructions that includeactions that are appropriate to the scenario. Note that more than one scenario may applyto a given issue.

Scenario: the issue is NOT performance-related and there is a consistent reproduction.

Record the following information in the issue report:

1. Reproduction steps
2. A screenshot of the error page or error message
3. The output of the browser developer console
4. Log output while reproducing the issue
5. Sourcegraph configuration
6. When was the most recent update or deployment change applied?

Scenario: the issue is NOT performance-related, but it is hard to reproduce.

Without a consistent reproduction, the issue will be harder to diagnose, so we recommend trying tofind a repro if possible. If that isn't possible, file an issue with the following information:

1. Steps the user took before encountering the issue, including as much detail as possible.
2. Bad example: "User encountered a 502 error when trying to search for something."
3. Good example: "User encountered a 502 error on the search results page when trying to conduct aglobal search for the following query. On refresh, the search worked with no error. The desiredresult appears in our main repository, which is rather large (takes about a minute to fullyclone). The issue doesn't reproduce consistently, but we saw two other reports like this around2pm PT yesterday, during peak usage hours."
4. Examine the error rates for any anomalies.
5. If you know the approximate time the issue occurred or if there is a spike in error rate around acertain time, copy the logs around that time.
6. Note any pattern in the issue reports. E.g., did users encountering the issue all visit the samerepository or belong to the same organization. Do site admins encounter the issue or onlynon-admin users?
7. Sourcegraph configuration
8. When was the most recent update or deployment change applied?

Scenario: the issue is performance-related and there is a consistent reproduction

1. Include the reproduction steps in the error report, along with relevant context (e.g., repositorysize).
   1. Bad example: "User did a search and it timed out."
   2. Good example: "User issued the following search query in the following repository. Therepository is one of our larger repositories (takes about 1 minute to fully clone and the sizeof the .git directory is 5GB). The results page took about 60 seconds to load and when itfinally did, the results was an error message that said 'timeout'".
2. Open the browser developer network panel and identify slowrequests.
3. Use Jaeger to drill down into slow requests and understand whichcomponents of the request are slow. Remember that many Sourcegraph API requests identify theJaeger trace ID in the x-trace HTTP response header, which makes it easy to look up the tracecorresponding to a particular request.
   1. If Jaeger is unavailable or unreliable, you can collect trace data from the Go net/traceendpoint.
4. Copy the Sourcegraph configuration to the error report.

Scenario: the issue is performance-related and there is NOT a consistent reproduction

Without a consistent reproduction, the issue will be harder to diagnose, so we recommend trying tofind a repro if possible. If that isn't possible, try the following:

1. Examine resource usage, usage stats, and errorrates over time in Grafana and Prometheus.
2. Are there spikes in latencies or error rate over time?
3. Are there spikes in usage or traffic over time that correlate with when the issue is reported.
4. Are there spikes in memory usage, CPU, or disk usage over time?
5. If you know the approximate time the issue occurred or if there is a suspicious spike in metricsaround a certain time, check the logs around that time.
6. If the issue is ongoing or if you know the time during which the issue occurred, searchJaeger for long-running request traces in the appropriate time window.
7. If Jaeger is unavailable, you can alternatively use the Go net/trace endpoint. (You will haveto scan the traces for each service to look for slow traces.)
8. If tracing points to a specific service as the source of high latency, examine thelogs and net/trace info for that service.

Scenario: multiple actions are slow or Sourcegraph as a whole feels sluggish

If Sourcegraph feels sluggish overall, the likely culprit is resource allocation.

1. Examine memory, CPU, and disk usage metrics.
2. If the metrics indicate high resource consumption, adjust the resource allocation higher.
3. If metrics are unavailable or inaccessible, here is a rough correspondence between end-userslowness and the services that are usually the culprit:
   1. Global search (i.e., no repository scope is specified) results page takes a long time to load.
   2. Increase indexed-search memory limit or CPU limit. The number of indexed-searchshards can also be increased if using Sourcegraph on Kubernetes.
   3. Search results show up quickly, but code snippets take awhile to populate. File contents takeawhile to load.
   4. Increase gitserver memory usage. Gitserver memory may be the bottleneck, especially if thereare many repositories or repositories are large.
   5. Increase number of gitserver shards. This can help if memory is the bottleneck. It can alsohelp if there are too many repositories per shard. Gitserver "shells out" to git for everyrepository data request, so a high volume of user traffic that generates many simultaneousrequests for many repositories can lead to a spike in Linux process exec latency.
   6. Increase memory and CPU limit of syntect-server. This helps if syntax highlighting is thebottleneck.
   7. Multiple UI pages take awhile to load.
   8. Increase frontend CPU and memory limit.
   9. Searches show intermittent HTTP 502 errors or timeouts, possibly concurrent with frontendcontainer restarts.
   10. Increase frontend memory and CPU. This may indicate the frontend is running out of memorywhen loading search results. This can be a problem when dealing with large monorepos.
4. If it is unclear which service is underallocated, examine Jaeger toidentify long-running traces and see which services take up the most time.
   1. Alternatively, you can use the Go net/trace endpoint to pull tracedata.

Scenario: Prometheus scraping metrics outside Sourcegraph Kubernetes namespace

If you are seeing cAdvisor metrics from a namespace outside of the one Sourcegraph is currently deployed into.

1. Uncomment our namespaced Prometheus cAdvisor configuration
2. Apply this configuration and restart Prometheus

Note: This is unneeded if you are using the 'namespaced' overlay

Scenario: search timeouts

If your users are experiencing search timeouts or search performance issues, please perform the following steps:

1. Try appending variations of index:only, timeout:60s and count:999999 to the search query to see if it is still slow.
2. Access Grafana directly.
3. Select the + icon on the left-hand side, then choose Import.
4. Paste this JSON into the input and click Load.
5. Once the dashboard appears, include screenshots of the entire dashboard in the issue report.
6. Include the logs of zoekt-webserver container in the indexed-search pods. If you are using single Docker container enable debug logs first.

Scenario: zoekt-webserver is in a CrashloopBackOff and err cannot allocate memory

Sourcegraph uses a mmap to store its indices. The default operating system limits on mmap counts may to be too low, which may result in out of memory exceptions.If you are seeing this error on large scale deployments with a lot repos to be indexed, please use the following steps to verify the source of the errors:

1. Ensure pods are not actually running out of allocated memory and being OOMKilled by running (if they are, then you should give them more memory and the below will not help you):

    kubectl top pod indexed-search-<pod_number>
   
    kubectl describe nodes 
   

2. On the host operating system execute sudo sysctl -n vm.max_map_count.

    $ sysctl -n vm.max_map_count
    65530
   

3. Calculate the number of repos in your deployment divided by the number of indexed-search replicas. For example:

   • 250,000 repositories / 2 indexed-search repliacas = 125,000 repos to index per replica.

4. If the vm.max_map_count is lower than the result of the above calculation. Adjust the vm.max_map_count by executing:

    sudo sysctl -w vm.max_map_count=262144
   

5. Verify the change.

    $ sudo sysctl -n vm.max_map_count
    262144
   

6. Ensure the change will persist a system reboot by updating the vm.max_map_count setting in /etc/sysctl.conf.

Scenario: zoekt-webserver is restarting due to watchdog

zoekt-webserver has a built in watchdog which ensures it can respond to search requests. If the watchdog fails, it will panic causing the process to exit with a non-zero exit code. This is like a Kubernetes health check, but works across all our deployment environments.

By default the watchdog runs every 30s. If the watchdog fails 3 consecutive times (with a 30s sleep in-between) it will trigger the panic. This is usually indicative a server which is consistently overloaded. It is recommended to increase the CPU quota assigned to it or horizontally scale to more replicas.

From Sourcegraph 3.22 you can configure the watchdog via environment variables:

• ZOEKT_WATCHDOG_TICK :: Duration of how often it runs. (default 30s)
• ZOEKT_WATCHDOG_ERRORS :: Consecutive error count before exit. (default 3)

If either is 0 the watchdog is disabled.

You can further diagnose an overloaded zoekt-webserver via watchdog logs or metrics. See log messages mentioning watchdog, or view the following metrics in grafana:

• zoekt_webserver_watchdog_errors :: The current error count for zoektwatchdog.
• zoekt_webserver_watchdog_total :: The total number of requests done byzoekt watchdog.
• zoekt_webserver_watchdog_errors_total :: The total number of errors fromzoekt watchdog.

Actions

This section contains various actions that can be taken to collect information or update Sourcegraphin order to resolve an error or performance issue. You should typically not read this sectiondirectly, but start with the General scenarios section to determine which actions areappropriate.

Check browser console

Open the browser JavaScript console (right-click in the browser > Inspect to open developer tools,then click the Console tab).

Check browser network panel

Open the browser developer network page (right-click in the browser > Inspect to open developer tools,then click the Network tab).

If you are new to the network page, check out this great introduction to the Chrome developer toolsNetwork panel.

• Check the waterfall diagram at the top and the Waterfall column in the list of network requests toquickly identify high-latency requests.
• Clicking on a request will open up a panel that provides additional details about the request.
  • If a GraphQL request is taking a long time, you should obtain its Jaeger trace ID by inspectingthe Headers tab of this panel and finding the X-Trace or x-trace response header value. Onceyou've obtained this trace ID, look it up in Jaeger.
• You can check Preserve log to preserve the list of requests across page loads and reloads.

Check resource usage

Access Prometheus and examine the following metrics:

Memory: process_resident_memory_bytes is a gauge that tracks memory usage per backend process.

• Example: process_resident_memory_bytes{app="indexed-search"} shows memory usage for eachindexed-search instance.

CPU: process_cpu_seconds_total is a counter that tracks cumulative CPU seconds used .

• Example: rate(process_cpu_seconds_total{app="sourcegraph-frontend"}[1m]) shows average CPU usagefor each sourcegraph-frontend instance over the last minute.

Disk: gitserver_disk_free_percent is a gauge that tracks free disk space on gitserver.

Check end-user stats

Go to /site-admin/usage-statistics to view daily, weekly, and monthly user statistics.

To drill down (e.g., into sub-daily traffic, visits per page type, latencies, etc.), accessGrafana and visit the Sourcegraph Internal > HTTP dashboard page, whichincludes the following panels:

• QPS by Status Code
• QPS by URL Route
• P90 Request Duration (request latency at the 90th percentile)

Grafana contains ready-made dashboards derived from Prometheus metrics. Any chart in Grafanamentioned here can be viewed in Prometheus by clicking the dropdown menu next to the Grafana paneltitle > Edit > copying the expression in the Metrics field.

Check error rates

Access Grafana and view the following charts:

• Folder: Sourcegraph Internal > Dashboard: HTTP > Chart: QPS by Status Code
  • This shows request rates by HTTP status code for end-user requests.
• Folder: General
  • This contains dashboards for each core service in Sourcegraph. Examine each for high-levelmetrics important to the health of each service.

Collect a Jaeger trace

If you are looking for the trace associated with a specific request,

• Find the trace ID in the HTTP response in the browser developer tools "Network" tab.
• Access Jaeger and look up the trace ID.

If you do not have a specific request or cannot find the trace ID,

• Access Jaeger.
• Search for a matching span by setting the appropriate fields in the sidebar.

2 ways: start with a span ID, or manually locate your span by searching the Jaeger GUI

Examine logs

If you are using the single Docker container or Docker Compose deployment option, logs are printedto stdout and stderr. You should be able to access these using your infrastructure provider'sstandard log viewing mechanism.

If you are using Kubernetes,

• Retrieve logs with kubectl logs $POD_ID.
• Tail logs with kubectl logs -f $POD_ID.
• If a pod container died, you can access the previous container logs with kubectl logs -p $POD_ID. This can be useful for diagnosing why a container crashed.
• You can tail logs for all pods associated with a given deployment: kubectl logs -f deployment/sourcegraph-frontend --container=frontend --since=10m

Examine Go net/trace

Each core service has an endpoint which displays traces using Go'snet/trace package.

To access this data,

1. First ensure you are logged in as a site admin.
2. Go to the URL path /-/debug. This page should show a list of links with the names of each coreservice (e.g., frontend, gitserver, etc.)
3. Click on the service you'd like to examine.
4. Click "Requests`. This brings you to a page where you can view traces for that service.

• You can filter to traces by duration or error state.
• You can show histograms of durations by minute, hour, or in total (since the process started)

On older versions of Sourcegraph on Kubernetes, the /-/debug URL path may be inaccessible. If thisis the case, you'll need to forward port 6060 on the main container of a given pod to access itstraces. For example, to access to traces of the first gitserver shard,

1. kubectl port-forward gitserver-0 6060
2. Go to http://localhost:6060 in your browser, and click on "Requests".

Copy configuration

Go the the URL path /site-admin/report-bug to obtain an all-in-one text box of all Sourcegraphconfiguration (which includes site configuration, code host configuration, and globalsettings). This lets you easily copy all configuration to an issue report (NOTE: remember to redactany secrets).

Collect instance stats

The following statistics are useful background context when reporting a performance issue:

• Number of repositories (can be found on the /site-admin/repositories page, search for "repositories total")
• Size distribution of repositories (e.g., are there one or more large "monorepos" that contain most of the code?)
• Number of users and daily usage stats from /site-admin/usage-statistics