Stack Dependencies for Unconfirmed runs causes backlog of queued runs

We have a few stacks that utilise Stack Dependencies to pass outputs of one stack into another.

This was initially a very good idea, to maintain a single source of truth, and syncing stacks together whenever upstream stacks changed.

However after using this feature for several months it has proved to be more inconvenient and impractical in practice.

The biggest pain points are as follows

• Stacks that depend on other stacks “lock up“ if the upstream stack is in an unconfirmed state. This leads to a build up of Queued Runs on the downstream stack that aren’t easily visible as being “stuck“ without going into the Stack details (in the Stack Overview the state shows as “Finished”

• Stack Dependencies introduces heavy coupling between stacks. If an upstream stack is in a Failed state sometimes it is not possible to run a downstream stack, because a variable may be unavailable.

• Stack Dependencies are hard to configure in Terraform. Especially across Space boundaries. This hampers our adoption of it too

The first couple of pain points are quite an issue, and I would like to provide some feedback for it. We have thought of moving back to Shared Contexts and relying on them to “link“ stacks together. the outputs of one stack can be written to a context, and any downstream stack can attach the context and use variables from the context.

This still maintains a source of truth, and some form of loose coupling (where downstream stacks would read the context on the next drift or tracked run)

I would like to propose that Stack Dependencies have the following features:

• A Stack dependency should be more lenient on the state of the upstream stacks

• Stack Dependencies can be configured via labeling, or some other means which makes terraforming easier

• Stack Dependencies work more like Contexts. Perhaps every Stacks Outputs are written to a “Managed Context“. A Managed Context would contain all of the Output valuess of a Stack (or a configurable subset of outputs), and would be managed by Spacelift.

  • Downstream Stacks can attach to the Managed Context of an Upstream Stack, and source the variables from it.