Sourcegraph exports OpenTelemetry data to a bundled OpenTelemetry Collector instance. This service can be configured to ingest, process, and then export observability data to an observability backend of choice. This approach offers a great deal of flexibility.
Sourcegraph’s OpenTelemetry Collector is deployed with a custom image,
sourcegraph/opentelemetry-collector, and is configured with a configuration YAML file.
sourcegraph/opentelemetry-collector is configured to not do anything with the data it receives, but exporters to various backends can be configured for each signal we currently support - currently, only traces data is supported.
Refer to the documentation for an in-depth explanation of the parts that compose a full collector pipeline.
For more details on configuring the OpenTelemetry collector for your deployment method, refer to the deployment-specific guidance:
Sourcegraph traces are exported in OpenTelemetry format to the bundled OpenTelemetry collector. To learn more about Sourcegraph traces in general, refer to our tracing documentation.
sourcegraph/opentelemetry-collector includes the following exporters that support traces:
Basic configuration for each tracing backend type is described below. Note that just adding a backend to the
exporters block does not enable it - it must also be added to the
Refer to the next snippet for a basic but complete example, which is the default out-of-the-box configuration:
receivers: otlp: protocols: grpc: http: exporters: logging: loglevel: warn sampling_initial: 5 sampling_thereafter: 200 service: pipelines: traces: receivers: - otlp exporters: - logging # The exporter name must be added here to enable it
To reduce the volume of traces being exported, the collector can be configured to apply sampling to the exported traces. Sourcegraph bundles the probabilistic sampler as part of its default collector container image.
If enabled, this sampling mechanism will be applied to all traces, regardless if a request was explictly marked as to be traced.
Refer to the next snippet for an example on how to update the configuration to enable sampling.
exporters: # ... processors: probabilistic_sampler: hash_seed: 22 # An integer used to compute the hash algorithm. Note that all collectors for a given tier (e.g. behind the same load balancer) should have the same hash_seed. sampling_percentage: 10.0 # (default = 0): Percentage at which traces are sampled; >= 100 samples all traces service: pipelines: # ... traces: #... processors: [probabilistic_sampler] # Plug the probabilistic sampler to the traces.
Exporters send observability data from OpenTelemetry collector to desired backends. Each exporter can support one, or several, OpenTelemetry signals.
This section outlines some common configurations for exporters - for more details, refer to the official OpenTelemetry exporters documentation.
Backends compatible with the OpenTelemetry protocol (OTLP) include services like Honeycomb and Grafana Tempo. OTLP-compatible backends typically accept the OTLP gRPC protocol, but they can also implement the OTLP HTTP protocol.
OTLP gRPC backends
Refer to the
otlp exporter documentation for all available options.
exporters: otlp: endpoint: otelcol2:4317 tls: cert_file: file.cert key_file: file.key otlp/2: endpoint: otelcol2:4317 tls: insecure: true
OTLP HTTP backends
Refer to the
otlphttp exporter documentation for all available options.
exporters: otlphttp: endpoint: https://example.com:4318/v1/traces
Refer to the
jaeger exporter documentation for all options.
Most Sourcegraph deployment methods still ship with an opt-in Jaeger instance - to set this up, follow the relevant deployment guides, which will also set up the appropriate configuration for you:
If you wish to do additional configuration or connect to your own Jaeger instance, the deployed Collector image is bundled with a basic configuration with Jaeger exporting.
If this configuration serves your needs, you do not have to provide a separate config - the Collector startup command can be set to
/bin/otelcol-sourcegraph --config=/etc/otel-collector/configs/jaeger.yaml. Note that this requires the environment variable
$JAEGER_HOST to be set on the Collector instance (i.e. the container in Kubernetes or Docker Compose):
exporters: jaeger: # Default Jaeger gRPC server endpoint: "$JAEGER_HOST:14250" tls: insecure: true
Refer to the
googlecloud exporter documentation for all available options.
If you run Sourcegraph on a GCP workload, all requests will be authenticated automatically. The documentation describes other authentication methods.
exporters: googlecloud: # See docs project: project-name # or fetched from credentials retry_on_failure: enabled: false