Deploying Sourcegraph executors on linux machines


In order to run executors on your machine, a few things need to be set up correctly before proceeding.

  • Executors only support linux-based machine with amd64 processors
  • Docker has to be installed on the machine (curl -fsSL | sh)
  • Git has to be installed at a version >= v2.26

If Firecracker isolation will be used: (recommended)

  • The host has to support KVM (for AWS that means a metal instance, on GCP that means enabling nested virtualization)
  • The following additional dependencies need to be installed:
    • dmsetup
    • losetup
    • mkfs.ext4
    • iptables
    • strings (part of binutils)


Once dependencies are met, you can download the executor binary and start configuring your machine:

Step 0: Confirm that virtualization is enabled (if using Firecracker)

KVM (virtualization) support is required for our sandboxing model with Firecracker. The following command checks whether virtualization is enabled on the machine (it should print something):

$ lscpu | grep Virtualization

Virtualization:                  VT-x
Virtualization type:             full

On Ubuntu-based distributions, you can also use the tool kvm-ok available in the cpu-checker package to reliably validate KVM support on your host:

# Install cpu-checker
$ apt-get update && apt-get install -y cpu-checker

# Check for KVM support
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Step 1: Download the latest executor binary

Below are the download links for the latest release of executors:

Note: Executors need to match the version of Sourcegraph they’re running against. Latest will usually only work for you when you run the latest version of Sourcegraph.

Download and setup the executor binary:

# Plug in the version of your Sourcegraph instance, like v4.1.0.
# Before Sourcegraph 3.43.0, tagged releases of executors are not available and you should default to using "latest" instead.
# Using latest is NOT recommended, because it might be incompatible with your Sourcegraph version.
curl -sfLo executor${SOURCEGRAPH_VERSION}/linux-amd64/executor
chmod +x executor
# Assuming /usr/local/bin is in $PATH.
mv executor /usr/local/bin

Step 2: Setup environment variables

The executor is configured through environment variables. Those need to be passed to it when you run it (including for install, validate and test-vm), so add these to your shell profile, or an environment file. Only EXECUTOR_FRONTEND_URL, EXECUTOR_FRONTEND_PASSWORD and EXECUTOR_QUEUE_NAME are required.

Env var Description Example value
EXECUTOR_FRONTEND_URL The external URL of the Sourcegraph instance. required
EXECUTOR_FRONTEND_PASSWORD The shared secret configured in the Sourcegraph instance site config under executors.accessToken. required our-shared-secret
EXECUTOR_QUEUE_NAME The name of the queue to pull jobs from to. Possible values: batches and codeintel required batches
EXECUTOR_USE_FIRECRACKER Whether to isolate jobs in virtual machines. Requires ignite and firecracker. Linux hosts only. (default value: “true”) true
EXECUTOR_MAXIMUM_NUM_JOBS Number of virtual machines or containers that can be running at once. (default value: “1”) 1
EXECUTOR_MAXIMUM_RUNTIME_PER_JOB The maximum wall time that can be spent on a single job. (default value: “30m”) 30m
EXECUTOR_JOB_MEMORY How much memory to allocate to each virtual machine or container. A value of zero sets no resource bound (in Docker, but not VMs). (default value: “12G”) 12G
EXECUTOR_JOB_NUM_CPUS How many CPUs to allocate to each virtual machine or container. A value of zero sets no resource bound (in Docker, but not VMs). (default value: “4”) 4
EXECUTOR_DOCKER_REGISTRY_MIRROR_URL The address of a docker registry mirror to use in firecracker VMs.
EXECUTOR_FIRECRACKER_DISK_SPACE How much disk space to allocate to each virtual machine. (default value: “20G”) 20G
EXECUTOR_FIRECRACKER_BANDWIDTH_EGRESS How much bandwidth to allow for egress packets to the VM in bytes/s. (default value: “524288000”) 524288000
EXECUTOR_FIRECRACKER_BANDWIDTH_INGRESS How much bandwidth to allow for ingress packets to the VM in bytes/s. (default value: “524288000”) 524288000
EXECUTOR_KEEP_WORKSPACES Whether to skip deletion of workspaces after a job completes (or fails). Note that when Firecracker is enabled that the workspace is initially copied into the VM, so modifications will not be observed. (default value: “false”) true
EXECUTOR_MAX_ACTIVE_TIME The maximum time that can be spent by the worker dequeueing records to be handled. (default value: “0”) 100m
EXECUTOR_NUM_TOTAL_JOBS The maximum number of jobs that will be dequeued by the worker. (default value: “0”) 100
EXECUTOR_DOCKER_HOST_MOUNT_PATH The target workspace as it resides on the Docker host (used to enable Docker-in-Docker). /workspaces
EXECUTOR_QUEUE_POLL_INTERVAL Interval between dequeue requests. (default value: “1s”) 1s
EXECUTOR_CLEANUP_TASK_INTERVAL The frequency with which to run periodic cleanup tasks. (default value: “1m”) 1m
EXECUTOR_VM_PREFIX A name prefix for virtual machines controlled by this instance. (default value: “executor”) executor
EXECUTOR_VM_STARTUP_SCRIPT_PATH A path to a file on the host that is loaded into a fresh virtual machine and executed on startup. /
NODE_EXPORTER_URL The URL of the node_exporter instance, without the /metrics path.
EXECUTOR_FIRECRACKER_IMAGE The base image to use for virtual machines. sourcegraph/executor-vm:insiders
EXECUTOR_FIRECRACKER_KERNEL_IMAGE The base image containing the kernel binary to use for virtual machines. sourcegraph/ignite-kernel:5.10.135-amd64
EXECUTOR_FIRECRACKER_SANDBOX_IMAGE The OCI image for the ignite VM sandbox. sourcegraph/ignite:v0.10.5
DOCKER_REGISTRY_NODE_EXPORTER_URL The URL of the Docker Registry instance’s node_exporter, without the /metrics path. http://localhost:9000
SRC_LOG_LEVEL upper log level to restrict log output to (dbug, info, warn, error, crit) (default value: “warn”) warn
# Example:
export EXECUTOR_QUEUE_NAME=batches

Step 3: Configure your machine

To be able to run workloads in isolation, a few dependencies need to be installed and configured. The executor CLI can do all of that automatically.

To run all of the required setup steps, just run

executor install all
# OR run the following, to see how to install/configure components separately.
executor install --help

Step 4: Validate your machine is ready to receive workloads

All set up! Before letting the executor start receiving workloads from your Sourcegraph instance, you might want to verify your setup. Run the following command:

executor validate

If any issues are found, correct them before proceeding.

If you use our sandboxing model with Firecracker (recommended), you can also verify that the executor is able to spin up the isolation VMs properly. For that, use the following command:

# Optionally provide a repo to clone into the VMs workspace, to verify that cloning works properly as well.
executor test-vm [ --revision=main]

This should succeed and print a command to use to attach to the guest VM. If it is able to spin up properly, that is usually good indication that everything is set up properly. If you need to debug some issue further, this can be helpful as you can exec into the guest VM from here.

Step 5: Make executor a systemd service (optional, systemd-based distributions only)

To make sure that the executor runs post-boot of the machine, you might want to wrap it in a systemd service. This is an example of how that could look like:

cat <<EOF >/etc/systemd/system/executor.service
Description=Sourcegraph executor



# Create environment file (this can also be sourced in your shell)
cat <<EOF >/etc/systemd/system/executor.env

systemctl enable executor

Step 6: Start receiving workloads

If you use the systemd service, simply run systemctl start executor, otherwise run executor run. Your executor should start listening for jobs now! All done!