Have a look around, our code is on GitHub.
Sourcegraph server is a collection of smaller binaries. The development server, dev/start.sh, initializes the environment and starts a process manager that runs all of the binaries. See the Architecture doc for a full description of what each of these services does. The sections below describe the dependencies you need to run dev/start.sh
.
Sourcegraph has the following dependencies:
docker-machine
The following are two recommendations for installing these dependencies:
Install Homebrew.
Install Docker for Mac.
optionally via brew
brew cask install docker
Install Go, Node Version Manager, PostgreSQL, Redis, Git, NGINX, golang-migrate, Comby, and SQLite tools with the following command:
brew install go yarn redis postgresql git gnu-sed nginx golang-migrate comby sqlite pcre FiloSottile/musl-cross/musl-cross
Install the Node Version Manager (nvm
) using:
NVM_VERSION="$(curl https://api.github.com/repos/nvm-sh/nvm/releases/latest | jq -r .name)" curl -L https://raw.githubusercontent.com/nvm-sh/nvm/"$NVM_VERSION"/install.sh -o install-nvm.sh sh install-nvm.sh
After the install script is finished, re-source your shell profile (e.g.,
source ~/.zshrc
) or restart your terminal session to pick up the nvm
definitions. Re-running the install script will update the installation.
Note: nvm
is implemented as a shell function, so it may not show up in
the output of which nvm
. Use type nvm
to verify whether it is set up.
There is also a Homebrew package for nvm
, but it is unsupported by the
nvm
maintainers.
Install the current recommended version of Node JS by running the following from the working directory of a sourcegraph repository clone:
nvm install nvm use --delete-prefix
After doing this, node -v
should show the same version mentioned in
.nvmrc
at the root of the sourcegraph repository.
Note: Although there is a Homebrew package for Node, we advise using nvm
instead, to ensure you get a Node version compatible with the current state
of the sourcegraph repository.
Configure PostgreSQL and Redis to start automatically
brew services start postgresql brew services start redis
(You can stop them later by calling stop
instead of start
above.)
Ensure psql
, the PostgreSQL command line client, is on your $PATH
.
Homebrew does not put it there by default. Homebrew gives you the command to run to insert psql
in your path in the "Caveats" section of brew info postgresql
. Alternatively, you can use the command below. It might need to be adjusted depending on your Homebrew prefix (/usr/local
below) and shell (bash below).
hash psql || { echo 'export PATH="/usr/local/opt/postgresql/bin:$PATH"' >> ~/.bash_profile } source ~/.bash_profile
Open a new Terminal window to ensure psql
is now on your $PATH
.
Add package repositories:
# Go sudo add-apt-repository ppa:longsleep/golang-backports # Docker curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # Yarn curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
Update repositories:
sudo apt-get update
Install dependencies:
sudo apt install -y make git-all postgresql postgresql-contrib redis-server nginx libpcre3-dev libsqlite3-dev pkg-config golang-go musl-tools docker-ce docker-ce-cli containerd.io yarn # install golang-migrate (you must move the extracted binary into your $PATH) curl -L https://github.com/golang-migrate/migrate/releases/download/v4.7.0/migrate.linux-amd64.tar.gz | tar xvz # install comby (you must move the extracted binary into your $PATH) curl -L https://github.com/comby-tools/comby/releases/download/0.11.3/comby-0.11.3-x86_64-linux.tar.gz | tar xvz # nvm (to manage Node.js) curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.2/install.sh | bash # in repo dir: install current recommendend version of Node JS nvm install
Configure startup services
sudo systemctl enable postgresql sudo systemctl enable redis-server.service
(optional) You can also run Redis using Docker
In this case you should not enable the redis-server.service
from the previous step.
dockerd # if docker isn't already running docker run -p 6379:6379 -v $REDIS_DATA_DIR redis # $REDIS_DATA_DIR should be an absolute path to a folder where you intend to store Redis data
You need to have Redis running when you start the dev server later on. If you have issues running Docker, try adding your user to the docker group, and/or updating the socket file persimissions, or try running these commands under sudo
.
You need a fresh Postgres database and a database user that has full ownership of that database.
Create a database for the current Unix user
# For Linux users, first access the postgres user shell sudo su - postgres
createdb
Create the Sourcegraph user and password
createuser --superuser sourcegraph psql -c "ALTER USER sourcegraph WITH PASSWORD 'sourcegraph';"
Create the Sourcegraph database
createdb --owner=sourcegraph --encoding=UTF8 --template=template0 sourcegraph
Configure database settings in your environment
The Sourcegraph server reads PostgreSQL connection configuration from the PG*
environment variables.
Add these, for example, in your ~/.bashrc
:
export PGPORT=5432 export PGHOST=localhost export PGUSER=sourcegraph export PGPASSWORD=sourcegraph export PGDATABASE=sourcegraph export PGSSLMODE=disable
You can also use a tool like envdir
or a .dotenv
file to
source these env vars on demand when you start the server.
For more information about data storage, read our full PostgreSQL Guide page.
Migrations are applied automatically.
This is the easy way - just launch Docker.app and wait for it to finish loading.
The Docker daemon should be running in the background, which you can test by
running docker ps
. If you're on OS X and using docker-machine
instead of
Docker for Mac, you may have to run:
docker-machine start default eval $(docker-machine env)
git clone https://github.com/sourcegraph/sourcegraph.git
cd sourcegraph ./dev/start.sh
This will continuously compile your code and live reload your locally running instance of Sourcegraph.
Navigate your browser to http://localhost:3080 to see if everything worked.
Noticing problems with node_modules/
or package versions? Try
running this command to clear the local package cache.
yarn cache clean rm -rf node_modules web/node_modules yarn cd web yarn
This means the frontend
server failed to start, for some reason. Look through
the previous logs for possible explanations, such as failure to contact the
redis
server, or database migrations failing.
While developing Sourcegraph, you may run into:
frontend | failed to migrate the DB. Please contact [email protected] for further assistance:Dirty database version 1514702776. Fix and force version.
You may have to run migrations manually. First, install the Go migrate CLI, then run dev/migrate.sh up
If you get something like error: Dirty database version 1514702776. Fix and force version.
, you need to roll things back and start from scratch.
dev/migrate.sh drop dev/migrate.sh up
If you receive errors while migrating, try dropping the database
dev/drop-entire-local-database.sh dev/migrate.sh up
If you see this error when opening the app:
500 Internal Server Error template: app.html:21:70: executing "app.html" at <version "styles/styl...>: error calling version: open ui/assets/styles/app.bundle.css: no such file or directory
that means Webpack hasn't finished compiling the styles yet (it takes about 3 minutes).
Simply wait a little while for a message from webpack like web | Time: 180000ms
to appear
in the terminal.
./dev/start.sh
may ask you to run ulimit to increase the maximum number
of available file descriptors for a process. You can make this setting
permanent for every shell session by adding the following line to your
.*rc
file (usually .bashrc
or .zshrc
):
# increase max number of file descriptors for running a sourcegraph instance. ulimit -n 10000
On Linux, it may also be necessary to increase sysctl -n fs.inotify.max_user_watches
, which can be
done by running one of the following:
echo 524288 | sudo tee -a /proc/sys/fs/inotify/max_user_watches # If the above doesn't work, you can also try this: sudo sysctl fs.inotify.max_user_watches=524288
If you ever need to wipe your local database and Redis, run the following command.
./dev/drop-entire-local-database-and-redis.sh
See testing.md for details.
On first install, the program will use quite a bit of bandwidth to concurrently download all of the Go and Node packages. After packages have been installed, the Javascript assets will be compiled into a single Javascript file, which can take up to 5 minutes, and can be heavy on the CPU at times.
After the initial install/compile is complete, the Docker for Mac binary uses about 1.5GB of RAM. The numerous different Go binaries don't use that much RAM or CPU each, about 5MB of RAM each.
If you notice heavy battery and CPU usage running gulp --color watch
, please first double check that Spotlight is not indexing your Sourcegraph repository, as this can lead to additional, unnecessary, poll events.
If you're running macOS 10.15.x (Catalina) reinstalling the Xcode Command Line Tools may be necessary as follows:
rm -rf /Library/Developer/CommandLineTools
xcode-select --install
sourcegraph/sourcegraph
’s root directory and run rm -rf node_modules
./dev/start.sh
)How to debug a program with Visual Studio Code:
Requires "Debugger for Chrome" extension.
/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222
start chrome.exe –remote-debugging-port=9222
chromium-browser --remote-debugging-port=9222
(ui) http://localhost:3080/*
debug configurationInstall Delve:
xcode-select --install go get -u github.com/go-delve/delve/cmd/dlv
Then install pgrep
:
brew install proctools
Make sure to run env DELVE=true dev/start.sh
to disable optimizations during compilation, otherwise Delve will have difficulty stepping through optimized functions (line numbers will be off, you won't be able to print local variables, etc.).
Now you can attach a debugger to any Go process (e.g. frontend, searcher, go-langserver) in 1 command:
dlv attach $(pgrep frontend)
Delve will pause the process once it attaches the debugger. Most used commands:
b cmd/frontend/db/access_tokens.go:52
to set a breakpoint on a line (bp
lists all, clearall
deletes all)c
to continue execution of the programCtrl-C
pause the program to bring back the command promptn
to step over the next statements
to step into the next function callstepout
to step out of the current function callCtrl-D
to exitWe use Go modules to manage Go dependencies in this repository.
The Sourcegraph repository relies on code generation triggered by go generate
. Code generation is used for a variety of tasks:
./server/internal/middleware/*
packages)To generate everything, just run:
./dev/generate.sh
Note: Sometimes, there are erroneous diffs. This occurs for a few reasons, none of which are legitimate (i.e., they are tech debt items we need to address):
If you think a diff is erroneous, don't commit it. Add a tech debt item to the issue tracker and assign the person who you think is responsible (or ask).
Running Sourcegraph on Windows is not actively tested, but should be possible within the Windows Subsystem for Linux (WSL). Sourcegraph currently relies on Unix specifics in several places, which makes it currently not possible to run Sourcegraph directly inside Windows without WSL. We are happy to accept contributions here! :)
Sometimes you will want to develop Sourcegraph but it just so happens you will be on a plane or a
train or perhaps a beach, and you will have no WiFi. And you may raise your fist toward heaven and
say something like, "Why, we can put a man on the moon, so why can't we develop high-quality code
search without an Internet connection?" But lower your hand back to your keyboard and fret no
further, for the year is 2019, and you can develop Sourcegraph with no connectivity by setting the
OFFLINE
environment variable:
OFFLINE=true dev/start.sh