Restoring Postgres databases in Kubernetes

Expected use case: Restoring a previous snapshot for disaster recovery

For restoring Postgres databases in our Kubernetes deployments you should have backups of both the primary db and the codeintel-db. The primary db contains all user information.

To restore persistent volumes you need to

  1. Understand how your Kubernetes cluster provides Persistent Volume (PV) Storage GKE , EKS , AKS
  2. Scale down all pods that are using the PersistentVolumeClaim (PVC)
  3. Backup the current PVCs
  4. Create new PVs that reference the restored snapshots


  1. Backup the PVCs we will be operating on kubectl get pvc pgsql codeintel-db codeinsights-db -oyaml > backupPVC.yaml

  2. Ensure that you have snapshots of the necessary PVs (this depends on your deployment environment) kubectl get pv $(kubectl get pvc pgsql codeintel-db codeinsights-db -o=jsonpath='{.items[*].spec.volumeName}')

  3. Restore the snapshots to new disks that meet the requirements to be consumed by your cluster (same region, etc).

  4. Scale down all deployments to zero (note: both approaches involve downtime, the first approach is preferred)

    kubectl scale deployment  -l deploy=sourcegraph --replicas=0

    or (if you are unable to scale deployments to their previous values easily)

    kubectl scale deployment pgsql codeinsights-db codeintel-db  --replicas=0
  5. Next, you need to replace the existing PVC and PV.

    1. First, delete the PVC kubectl delete pvc pgsql

    2. Then create the new PV & PVC. The options used to configure the PV will be specific to every provider. “`yaml apiVersion: v1 kind: PersistentVolume metadata: name: example-pgsql-volume spec: storageClassName: “sourcegraph” accessModes:

      • ReadWriteOnce capacity: storage: 200Gi claimRef: namespace: ns-sourcegraph name: pgsql

        You will need to customize the fields below most likely

        gcePersistentDisk: pdName: disk-restored-from-snapshot

        fsType: ext4

        apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pgsql namespace: ns-sourcegraph spec: storageClassName: “sourcegraph” accessModes:

      • ReadWriteOnce resources: requests: storage: 200Gi “ 1. Ensure PVCs become Boundkubectl get pvc -w 1. (Optional) Commit PVs and PVCs back to deployment repo into thebase/${deployment}directory. 1. Scale deployments back up 1. If you use our deployment scripts, after committing the change you may run./kubectl-apply-all.shagain to restore all deployments to their set number of replicas. 1. Otherwise,kubectl scale deployment pgsql codeinsights-db codeintel-db –replicas=1` 1. After ensuring Sourcegraph is functional, you may delete the previous PVs. If the Reclaim Policy is not set to Delete you will need to manually delete the disk from your provider as well.