Skip to main content

Auto-release stale Terraform state locks on run failure

Feature RequestsStacks

When a run fails unexpectedly, stale Terraform state locks are sometimes left behind. Request to have this condition detected and automatically clean up locks that were created as part of a run.

Status: ❌ Rejected6 comments

Log in to comment and vote

Comments6

  • Jonah Kowall changed status to ❌ Rejected
    Team•

    Jun 12

    Pinned

    Sorry, there's no way for us to get the lock ID programmatically in a consistent way, so this is not really possible. That's why I'm closing the feature request out. It's a valid request, but not technically feasible.

  • Jonah Kowall

    Team•

    Jun 12

    Thanks for raising this. We've thought hard about it and we're not going to automatically release state locks on run failure, and I want to be transparent about why.

    The only time a run failure leaves a lock behind is abnormal termination (a worker killed by OOM or timeout, a crash, or a network partition to your state backend). In that situation Spacelift cannot prove the original process is actually dead. The dangerous case is a network partition where the worker lost its connection to us but is still alive and still writing to your backend. If we auto-released the lock there, you'd get two concurrent writers and a corrupted state file, which is far worse and harder to recover from than a stuck lock. This is the same reason Terraform and OpenTofu keep force-unlock manual and gated behind a specific lock ID rather than automating it.

    To clear an orphaned lock today: open the failed run's logs, copy the lock ID from the Error acquiring the state lock output, confirm the run that created it has actually terminated, then run a Task on the stack with terraform force-unlock -force <LOCK_ID> (or tofu force-unlock -force <LOCK_ID>). This only removes the lock; it does not touch your infrastructure. Run a plan afterward to confirm state is consistent before your next apply.

  • Violet Lizard

    •

    May 22

    To suggest a change / way to handle this request a bit: Locks shouldn’t be auto-released, but when a Statefile / Stack gets into a “Locked” state then having a button to unlock that would be the ideal.

    Getting into situations where a Stacks end up in a bad locked state is common when handling many stacks and the means by which to work around this today is to have a user go into tasks and run a force-unlock command against individual state lock ID. This is cumbersome and requires multiple steps.

    Spacelift providing an “Unlock” button when in a locked state would be a major quality of life improvement.

  • Black Breeze

    •

    Mar 31

    Thanks for raising that, folks. IIRC this is quite tricky but I’m going to get the dev team to explore options.

    • Black Breeze

      •

      Mar 31

      Guys, without going too deep into the woods the reality is that there’s no way of doing it with Terraform or OpenTofu that’s not hacky or brittle.

      So please don’t expect a quick fix. Don’t get me wrong, your request is totally valid. But the best way of going about it is to raise it with the OpenTofu team to see if they’d like to consider improving the experience of programmatic lock management.

  • Azure Tamarin

    •

    Mar 25

    This is such an essential QOL of feature (optional togglable) because of the fact that this ties in so deeply as the underlying functionality of Terraform / OpenTofu and state management, especially if state is managed custom (e.g. in AWS S3).