Data freshness

Searches scoped to specific repositories are always up-to-date. Sourcegraph automatically fetches repository contents with any user action specific to the repository and makes new commits and branches available for searching and browsing immediately.

Unscoped search results over large repository sets may trail latest default branch revisions by some interval of time. This interval is a function of the number of repositories and the computational resources devoted to search indexing.

Max file size

By default, files larger than 1 MB are excluded from search results. Use the search.largeFiles keyword to specify files to be indexed and searched regardless of size. Regardless of where you set the search.largeFiles environment variable, Sourcegraph will continue to ignore binary files, even if the size of the file is less than the limit you set.

Exclude files and directories

You can exclude files and directories from search by adding the file .sourcegraph/ignore to the root directory of your repository. Sourcegraph interprets each line in the ignore file as a glob pattern. Files or directories matching those patterns will not show up in the search results.

The ignore file is tied to a commit. This means that if you committed an ignore file to a feature branch but not to your default branch, then only search results for the feature branch will be filtered, while the default branch will show all results.


# .sourcegraph/ignore
# lines starting with # are comments and are ignored
# empty lines are ignored, too

# ignore the directory node_modules/

# ignore the directory src/data/

# ** matches all characters, while * matches all characters except /
# ignore all JSON files

# ignore all JSON files at the root of the repository

# ignore all JSON files within the directory data/

# ignore all data folders

# ignore all files that start with numbers

Our syntax follows closely what is documented in the linux documentation project. However, we distinguish between * and **: While ** matches all characters, * matches all characters except the path separator /.

Note that invalid globbing patterns will cause an error and searches over commits containing a broken ignore file will not return any result.

Shard merging

Shard merging is a feature of Zoekt that enables the combination of smaller index files, or shards, into one larger file, a compound shard. This can reduce memory costs for Zoekt webserver. This feature is particularly useful for customers with many small and rarely updated repositories, and can result in a significant reduction in memory. Shard merging can be enabled by setting SRC_ENABLE_SHARD_MERGING="1" for Zoekt indexserver.

Shard merging can be fine-tuned by setting ENV variables for Zoekt indexserver:

Env Variable Description Default
SRC_VACUUM_INTERVAL Run vacuum this often, specified as a duration 24 hours
SRC_MERGE_INTERVAL Run merge this often, specified as a duration 8 hours
SRC_MERGE_TARGET_SIZE The target size of compound shards in MiB 2000
SRC_MERGE_MIN_SIZE The minimum size of a compound shard in MiB 1800
SRC_MERGE_MIN_AGE The time since the last commit in days. Shards with newer commits are excluded from merging. 7
SRC_MERGE_MAX_PRIORITY The maximum priority a shard can have to be considered for merging, specified as a float value 100.0

When repostiories receive udpates, Zoekt reindexes them and tombstones their old index data. As a result, compound shards can shrink and be dismantled into individual shards once they reach a critical minimum size. These individual shards are then considered for future merge operations.

Shard merging can be monitored via the "Compound shards" panel in Zoekt's Grafana dashboard.

RE2 Regular Expressions

The Sourcegraph search language supports RE2 syntax. If you're used to tools like Perl which uses PCRE syntax, you may notice that there are some features that are missing from RE2 like backreferences and lookarounds. We choose to use RE2 for a few reasons:

  • It makes it possible to build worst-case linear evaluation engines, which is very desirable for building a production-ready regex search engine.
  • It's well-supported in Go, allowing us to take advantage of a rich ecosystem (notably including Zoekt)
  • Our API and tooling makes it straightforward to use Sourcegraph with other tools that provide facilities not built in to the search language.

As an example of how you can use Sourcegraph tooling with other tools, we can use jq (which supports Perl regexes) along with src to post-filter search results. In this case, we want to use backreferences to find go functions take a single pointer argument and return a non-pointer of the same type as the input.

re2_regex='func \w+\(\w+ \*\w+\) \w+'
pcre2_regex='func \w+\(\w+ \*(\w+)\) \1'

src search --json --stream -- "/$re2_regex/" \
  | jq '
    # Filter to only content events
    select(.type == "content")
    # Flatten to a single object per match
    | {content: .chunkMatches[].content} + del(.chunkMatches)

    # Select only matches that match the PCRE regex
    | select(.content | test($ARGS.positional[0]))
  ' --args "$pcre2_regex"