Skip to main content

Automatic cleanup of old worker binaries and Docker images on private workers

Feature Requests

Long-lived private workers accumulate data that is never cleaned up:

  • /opt/spacelift/binaries/worker/ keeps a new hash-named folder for every worker release; old versions are never removed

  • Unused Docker runner images accumulate under /var/lib/docker

We'd like built-in cleanup in the launcher, for example:

  • Keep only the current worker binary plus the last N versions

  • Prune unused Docker images older than a configurable age, run only while the worker is idle/drained so active runs are never affected

  • Configuration via environment variables (e.g. SPACELIFT_LAUNCHER_BINARY_RETENTION, SPACELIFT_LAUNCHER_DOCKER_PRUNE_AGE)

Native rotation in the Azure workerpool module, similar to the AWS module's instance refresh/lifecycle handling, would also solve this.

Workaround
Problem
Status: ⌛ Waiting for User1 comment

Log in to comment and vote

Comments1

  • Jonah Kowall

    Team•

    Oct 26

    Thanks for the detailed write-up, Jason, the disk-usage measurements and the note about SPACELIFT_LAUNCHER_DISK_USAGE_THRESHOLD looping on a persistent OS disk.

    A few things that may help today, and where we land:

    Docker images: your docker image prune -a -f --filter "until=168h" cron is the right fix for now. To avoid the race with active runs, gate it on the worker being drained, you can drain via the UI Drain button or the workerDrainSet mutation, run the prune, then undrain. Running it as a scheduled task against an idle/drained worker avoids re-imaging the whole VMSS instance.

    Worker binaries: you're correct that nothing prunes /opt/spacelift/binaries/worker/ today, and that the persistent-disk reboot behavior means the threshold setting alone won't reclaim space.

    On the ask itself: the most useful and lowest-risk parts are (1) having the launcher retain only the last N worker binaries and (2) bringing the Azure VMSS workerpool module up to the same lifecycle/rotation behavior the AWS module already offers via instance refresh. We're capturing those for our workers team. The native rotation in the Azure module is likely the cleanest long-term fix for the persistent-disk problem you hit, since it resets the OS disk the way your manual drain/reimage does today.

    We're going to hold on building idle-gated Docker-image pruning into the launcher for now, pruning tied to the run lifecycle adds coordination risk, and image hygiene is generally better handled at the VM/module layer with the drain-gated cron above. If you're seeing specific pain there that the workaround doesn't cover, let us know and we'll factor it in.

    Would launcher binary retention plus Azure module rotation parity cover your use case?