Allow to check past/historic states

Hey @Natalia Gazda, hope the following answers help you better understand the problem and expected solution. If after this comment there’s something unclear, don’t hesitate to keep making questions. I’ll be happy to answer them.

How often do you run into situations where you need to compare historic state to current state?

Unfortunately more often than I would like. Usually when there’s something weird happening, or an incident is ongoing. Sometimes also for postmortems. Let’s say 1 or 2 times per month, but these times involve several checks on several stacks.

When it happens, what are you usually trying to figure out—which resources changed, why they changed, or something else?

Mainly which resources changed, and also how they changed (which attributes changed, and be able to compare past value to current value, or to the next value after certain apply in the past). Plans sometimes are not specifying all the attributes that are going to be set, and applies also don’t show that. If other plans&applies happened after, currently there’s no way to know exactly what happened there.

Would simply viewing the old state be enough, or do you prefer to diff it against the current one? OR is there anything else you’d really like to see?

Yep, being able to download it or view it is enough. Diff could be a nice to have, but in the past I found the diff hard to understand (object are not always in the same place, so the diff is a mess). So summing up, I would say the must have is the state download per event (I say event because that’s the nomenclature used at the “State history“ tab), view it inside Spacelift is a nice to have, and being able to navigate the “IaC management“ tab in different points of time (by selecting the commit/event/date) would be the super nice to have.

Would it be only useful for mostly troubleshooting failed applies, or also the compliance / audit history would matter?

I believe right now I’m biased because I needed that for troubleshooting an incident, to know what happened and to know when certain attribute from a resource changed its value (or to know if it was changed or not). That said, I believe that for compliance / audit history could also be useful.

In that situation what was your workaround?

Unfortunately there was no workaround. Past applied states were not available, and plans were not showing the value for the particular attribute I needed to check. I could only check the last state of that attribute. What I did was to create another resource using the same provider and check the result hoping the past situation was the same, but I only had hope without being able to have evidences. Since this incident seemed to come from a 3rd party provider issue, by collecting our assumptions and some evidences added to an investigation from their side concluded it was their fault, and that indeed the attribute changed on their side and made our state change during the refresh step at the beginning of the apply.

That said, I remember TFC was including a special section before the plan where it showed the resources changed outside TFC (if any), so it was easy to know when something was going on, and to know certain change was not fault of (or triggered by) the code/provider/whatever on our side. I’ve been told this is missed here, just FYI.