Add a third priority level lower than the default one
Most of the times with our changes we trigger ~5-10 stacks and with rather fast feedback. But somewhat regularly we execute upgrades/migrations that trigger hundreds of stacks, for those we don’t expect fast feedback we are happy with those taking a while as we only have 5 workers.
The issue arises when we have both simultaneously, and if the ones triggered early are part of the 100s migration/upgrade batch then the rest will take longer to get feedback.
An approach is to prioritize the small batches. But it is not really efficient for us to prioritize every small batch that comes after a big batch.
Instead we would like to mark the big batches as `deprioritized` so that the of the stacks to be picked up would be:
- prioritized
- regular/default
- deprioritized
- Workaround
- Problem
Log in to comment and vote
Comments9
Natalia Gazda
Sep 11
Documentation about the feature can be accessed here → https://spacelift-io.slack.com/archives/C031E6V9Y3V/p1789131800647059
Happy using :)
Silver Milkshake
Aug 22, 2025
@Marcin Wyszynski
Thanks for the update, unfortunately in our particular case drift detection runs were not the main reason to think of worker pool contention, and thus our major pain point would remain after this change. We will be looking in how to work around it and take at least some time before rising it again as I understand if this is what aligns best with the rest of requests on worker pool contention.
Black Breeze
Aug 21, 2025
We tried to address the main root cause of the worker pool contention - drift detection runs. Now you will be able to allocate a maximum number of workers for a given worker pool that are allowed to execute drift detection runs, leaving others to take on higher priority work.
While we understand this will not address every possible scenario or a corner case, we believe it’s the optimal compromise between complexity and control.
Jade Grape
Jul 16, 2025
I like this idea. I can definitely see a use case for something like a “batch maintenance” priority, where you deliberately set something to go last-in-line. This could be particularly useful for drift detection runs - you never want to stomp on something that a user is waiting on, and some tasks can absolutely be told to wait.
Natalia Gazda
Jul 11, 2025
Hi @Saul Ivan Rivas Vega Thanks for sharing this, I totally understand.
A few things I’d love to understand better:
Would you want to mark them as deprioritized via UI, API, or maybe Terraform?
Would you expect visibility into queue order? Or do you just want to trust that the fast ones go first?
How often does it happen that you have those two simultaneously?
Silver Milkshake
Jul 11, 2025
Hi @Natalia Gazda Thank yo for taking a look!
Would you want to mark them as deprioritized via UI, API, or maybe Terraform?
In the UI would work best for us.
Would you expect visibility into queue order? Or do you just want to trust that the fast ones go first?
Would say mainly the latter but the former is a nice one too.
How often does it happen that you have those two simultaneously?
Almost every time we carry out a big migration/update. Not every day we have a big one, but we have teams triggering small fast changes frequently throughout the day.
Natalia Gazda
Jul 24, 2025
Alright, thanks for the input! Let me check with the team and get back to you :)