Continuous integration SOC2/GN-105 SOC2/GN-106

Sourcegraph uses a continuous integration and delivery tool, Buildkite, to help ensure a consistent build, test and deploy process. Software changes are systematically required to complete all steps within the continuous integration tool workflow prior to production deployment, in addition to being peer reviewed.

Sourcegraph also maintains a variety of tooling on GitHub Actions for continuous integration and repository maintainence purposes.

Buildkite pipelines

Tests are automatically run in our various Buildkite pipelines. Pipeline steps are generated using the pipeline generator.

To see what checks will get run against your current branch, use sg:

sg ci preview

To learn about making changes to our Buildkite pipelines, see Pipeline development.

Pipeline steps

A complete reference of all available pipeline steps is not yet available (#30203). This section contains a high-level documentation about what runs in our pipeline.

Soft failures

SOC2/GN-106

Many steps in Sourcegraph’s Buildkite pipelines allow for soft failures, which means that even if they fail they do not cause the entire build to be failed.

In the Buildkite UI, soft failures currently look like the following, with a triangular warning sign (not to be mistaken for a hard failure!):

soft fail in Buildkite UI

We use soft failures for the following reasons only:

  • Steps that determine whether a subsequent step should run, where soft failures are the only technical way to communicate that a later step should be skipped in this manner using Buildkite.
  • Regular analysis tasks, where soft failures serve as an monitoring indicator to warn the team responsible for fixing issues.
  • Temporary exceptions to accommodate experimental or in-progress work.

You can find all usages of soft failures with the following queries:

All other failures are hard failures.

Image vulnerability scanning

Our CI pipeline scans uses Trivy to scan our Docker images for security vulnerabilities.

Trivy will perform scans upon commits to the following branches:

  1. main
  2. branches prefixed by main-dry-run/
  3. branches prefixed by docker-images-patch/$IMAGE (where only a single image is built)

If there are any HIGH or CRITICAL severities in a Docker image that have a known fix:

  1. The CI pipeline will create an annotation that contains links to reports that describe the vulnerabilities
  2. The Trivy scanning step will soft fail. Note that soft failures do not fail builds or block deployments. They simply highlight the failing step for further analysis.

We also run separate vulnerability scans for our infrastructure.

Pipeline health

Maintaining Buildkite pipeline health is a critical part of ensuring we ship a stable product - changes that make it to the main branch may be deployed to various Sourcegraph instances, and having a reliable and predictable pipeline is crucial to ensuring bugs do not make it to production environments.

To enable this, we address flakes as they arise and mitigate the impacts of pipeline instability with branch locks.

Branch locks

buildchecker is a tool responding to periods of consecutive build failures on the main branch Sourcegraph Buildkite pipeline. If it detects a series of failures on the main branch, merges to main will be restricted to members of the Sourcegraph team who authored the failing commits until the issue is resolved - this is referred to as a “branch lock”. When a build passes on main again, buildchecker will automatically unlock the branch.

Authors of the most recent failed builds are responsible for investigating failures. Please refer to the Continuous integration playbook for step-by-step guides on what to do in various scenarios.

Flakes

A flake is defined as a test or script that is unreliable or non-deterministic, i.e. it exhibits both a passing and a failing result with the same code. In other words: something that sometimes fails, but if you retry it enough times, it passes, eventually.

Tests are not the only thing that are flaky - flakes can also encompass sporadic infrastructure issues and unreliable steps.

Flaky tests

Typical reasons why a test may be flaky:

  • Race conditions or timing issues
  • Caching or inconsistent state between tests
  • Unreliable test infrastructure (such as CI)
  • Reliance on third-party services that are inconsistent

If a flaky test is discovered, immediately use language-specific functionality to skip a test and open a PR to disable the test:

If the language or framework allows for a skip reason, include a link to the issue track re-enabling the test, or leave a docstring with a link.

Then open an issue to investigate the flaky test (use the flaky test issue template), and assign it to the most likely owner.

Flaky steps

If a step is flaky we need to get the build back to reliable as soon as possible. If there is not already a discussion in #buildkite-main create one and link what step you take. Here are the recommended approaches in order:

  1. Revert the PR if a recent change introduced the instability. Ping author.
  2. Use Skip StepOpt when creating the step. Include reason and a link to context. This will still show the step on builds so we don’t forget about it.

An example use of Skip:

--- a/enterprise/dev/ci/internal/ci/operations.go
+++ b/enterprise/dev/ci/internal/ci/operations.go
@@ -260,7 +260,9 @@ func addGoBuild(pipeline *bk.Pipeline) {
 func addDockerfileLint(pipeline *bk.Pipeline) {
        pipeline.AddStep(":docker: Lint",
                bk.Cmd("./dev/ci/docker-lint.sh"),
+               bk.Skip("2021-09-29 example message https://github.com/sourcegraph/sourcegraph/issues/123"),
        )
 }
Flaky infrastructure

If the build or test infrastructure itself is flaky, then open an issue with the team/devx label and notify the Developer Experience team.

Also see Buildkite infrastructure.

Pipeline development

The source code of the pipeline generator is in /enterprise/dev/ci.

To test the rendering of the entire pipeline, you can run env BUILDKITE_BRANCH=TESTBRANCH go run ./enterprise/dev/ci/gen-pipeline.go and inspect the YAML output. To change the behaviour set the relevant BUILDKITE_ environment variables.

Pipeline operations

Pipeline steps are defined as Operations that apply changes to the given pipeline, such as adding steps and components.

(:[_] *bk.Pipeline) patternType:structural repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/ci/operations\.go

Within an Operation you will typically create one or more steps on a pipeline with AddStep, which can be configured with options of type SteptOpt.

(:[_]) StepOpt patternType:structural repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/buildkite/buildkite\.go

Operations are then added to a pipeline from GeneratePipeline.

For most basic PR checks, see Developing PR checks for how to create your own steps!

For more advanced usage for specific run types, see Developing run types.

Developing PR checks

To create a new check that can run on pull requests on relevant files, check the changed.Files type to see if a relevant affectsXyz check already exists.

Affects type:symbol select:symbol.function repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/ci/changed

If not, you can define a new one on the changed.Files type.

Then, you can add a new check to CoreTestOperations. Make sure to follow the best practices outlined in docstring.

For more advanced pipelines, see Run types.

Developing run types

There are a variety of run types available based on branch prefixes. These generate special-purpose pipelines. For example, the main-dry-run/ prefix is used to generate a pipeline similar to the default main branch. See RunType for the various run types available, and examples for how to add more.

GeneratePipeline can leverage RunType when generating pipelines, for example:

RunType.Is(:[_]) OR case :[_]: patternType:structural repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/ci/pipeline\.go

For simple PR checks, see Creating PR checks.

Buildkite infrastructure

Also see Flaky infrastructure, Continous integration infrastructure, and the Continuous integration changelog.

Pipeline setup

To set up Buildkite to use the rendered pipeline, add the following step in the pipeline settings:

go run ./enterprise/dev/ci/gen-pipeline.go | buildkite-agent pipeline upload
Managing secrets

The term secret refers to authentication credentials like passwords, API keys, tokens, etc. which are used to access a particular service. Our CI pipeline must never leak secrets:

  • to add a secret, use the Secret Manager on Google Cloud and then inject it at deployment time as an environment variable in the CI agents, which will make it available to every step.
  • use an environment variable name with one of the following suffixes to ensure it gets redacted in the logs: *_PASSWORD, *_SECRET, *_TOKEN, *_ACCESS_KEY, *_SECRET_KEY, *_CREDENTIALS
  • while environment variables can be assigned when declaring steps, they should never be used for secrets, because they won’t get redacted, even if they match one of the above patterns.

GitHub Actions

buildchecker

buildchecker, our branch lock management tool, runs in GitHub actions - see the workflow specification.

To learn more about buildchecker, refer to the buildchecker source code and documentation.

Third-party licenses

We use the license_finder tool to check third-party dependencies for their licenses. It runs as a GitHub Action on pull requests, which will fail if one of the following occur:

  • If the license for a dependency cannot be inferred. To resolve:
    • Use license_finder licenses add <dep> <license> to set the license manually
  • If the license for a new or updated dependency is not on the list of approved licenses. To resolve, either:
    • Remove the dependency
    • Use license_finder ignored_dependencies add <dep> --why="Some reason" to ignore it
    • Use license_finder permitted_licenses add <license> --why="Some reason" to allow the offending license

The license_finder tool can be installed using gem install license_finder. You can run the script locally using:

# updates ThirdPartyLicenses.csv
./dev/licenses.sh

# runs the same check as the one used in CI, returning status 1
# if there are any unapproved dependencies ('action items')
LICENSE_CHECK=true ./dev/licenses.sh

The ./dev/licenses.sh script will also output some license_finder configuration for debugging purposes - this configuration is based on the doc/dependency_decisions.yml file, which tracks decisions made about licenses and dependencies.

For more details, refer to the license_finder documentation.