So today I discovered that there's a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn't live inside any source control. This morning it entered a terminal state, and because it overwrites its state there's no way to revert it.
I'm currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
This is almost exactly what happened to me on Monday, resulting in a fifteen hour day.
My particular jenga piece was an Access query that none of my predecessors had deigned to document or even tell me about... but was critical to run monthly or you had obsolete data embedded deep within multi-million dollar reports.
Thank god I don't work on salary anymore, or I'd be really upset.
I stopped reading at “Access” and just wept a silent tear for you.
Oh god Access.
You have my condolences.