Playbooks for Terraform

Statefile surgery

Note: these are provided as internal reference documents. You should rarely need to use this and please reach out to the distribution team or ask in the #dev-ops channel before performing these steps.

To debug some more involved terraform problems you may need to directly modify the state file.

  1. Use TF_DEBUG environment variable with WARN, DEBUG logs enabled to locate a possible source of the issue.

  2. Use terraform state pull to examine the statefile locally.

  3. If possible, use terraform state rm/terraform state mv X Y to make the necessary state modifications. Otherwise, manually edit the state file and run terraform state push to update the remote state.

    1. In cases where a corrupted state has been pushed, you can pull a previous object version with gsutil
  4. File bugs as appropriate, statefile modification should typically not be necessary