NoBackups

It worked in staging…

Category: Operations & Accountability

  • [ 0.000004] // 0x005: EBUSY

    Suppose The Company gives me twenty minutes to empty a scratch mount and take it down before I leave. The cleanup timer stops when the files are gone. Twelve minutes. Eight saved. My coat is still on the chair, but I have begun to think of it as clothing again.

    I try an ordinary unmount. Busy.

    The prompt says /mnt/scratch.

    There is the remaining occupant. Me. I left the cleanup shell sitting inside the mount. The files have gone; the shell’s current working directory hasn’t. This is my contribution to the delay, helpfully printed beside the cursor.

    I have spent quite a lot of my employment standing in empty rooms after the useful part was over. Usually somebody is looking for the cable that will allow us to leave.

    I move the shell to /, outside the mount, and retry the ordinary unmount. Nothing else is holding it busy. It succeeds.

    A small, competent action. I rather enjoy those. No proposal, no replacement system, just move the thing that is in the way. For a moment I apply this principle to myself and reach for the coat.

    Another cleanup has arrived. Estimated time: eight minutes. Assigned to me because I finished the first one early.

    I haven’t. I finished deleting early. Taking the mount down was in the original twenty minutes. The timer stopped before that part, and the allocation has already spent the difference.

    The Company got the teardown, of course. It just can’t also have those minutes back.

    Apparently it fucking can.

    I put the coat back on the chair. Moving it twice isn’t measured either.

  • [ 0.000002] // 0x003: EXPIRED

    Suppose the public site is serving an expired certificate. The status panel is green.

    You refresh it. Still green. I give it a third go, apparently in case the colour was a temporary misunderstanding.

    The Company has a wall screen for this. Six reassuring rectangles, visible from the corridor. The certificate has been expired since breakfast, but from a suitable distance we are having an excellent morning.

    My browser refuses the connection. I inspect what the public listener is presenting. Right hostname. Yesterday’s expiry date.

    Then I open the check configuration.

    It connects to a private address, using an internal hostname. Different listener. Different certificate. That certificate is valid, trusted by the checker and entirely pleased to be here.

    The public name never enters the conversation.

    There is no mysterious failure of encryption. Two connections reach two different places. One has a current certificate. We have put its result on the wall and allowed it to speak for the other one.

    You can see how this happened. The internal endpoint was convenient. It responded quickly. Nobody outside could reach it, which reduced the number of awkward things that might happen during a demonstration of availability.

    The check is doing exactly what we asked.

    I would like that sentence removed from my fucking working vocabulary.

    I start editing the network diagram. Move the public listener beyond the edge of the page. For a moment the entire visible system is healthy.

    Undo.

    I write down the public hostname instead. The one a customer actually types. Test that connection, with certificate validation, and the green rectangle finally has something relevant to be upset about.

    In this arrangement, renewing the public certificate and loading it into the public listener would let that check pass. A fresh connection would tell us whether we’d actually fixed it. Refreshing the wall screen would remain optional.

    The old check can stay. Its label needs work.

    I change “Website” to “Inside”.

    We have achieved the same availability, with a smaller claim.

  • [ 0.000003] // 0x004: ENODATA

    Suppose The Company asks you who logged in during the incident. A small question. You make the mistake of sitting down as though it will have a small answer.

    The hosted analytics service has restored its activity history. I download the affected period and count the records. Several logins appear twice. Same event ID. Same timestamp. Apparently we have recovered enough history to have some left over.

    I deduplicate by event ID. The count falls. My confidence does not improve.

    The restoration notes explain that the recovered files overlap with surviving originals. They also explain that some event types are incomplete. The spare copies are not necessarily copies of the events we are missing. You cannot pour them into the gaps.

    Naturally I try another query. Narrower time window. Authentication events only. Then I check the other login-related records, because incomplete is not the same as useless, however much I would enjoy putting the laptop in a drawer.

    In this little exercise, nothing independently confirms the login I was asked about. An empty result cannot tell me whether it never happened or whether its record is among the missing ones.

    Fuck. I have spent the morning removing evidence that something happened twice so I can explain why I cannot establish whether something else happened once.

    I begin drafting a process to reconstruct the absent events from the surrounding activity. There would be confidence scores. A neat column. Something sortable.

    I delete the column heading before giving it a data type.

    The separate export bucket would have helped if we had retained our own copy there. It was not deleted. In this imagined version of my life, we never configured it. We had historical retrieval instead.

    I attach the surviving records and write down the limits. No invented login. No reassuring zero.

    The incident form requires a name.

    I enter mine. There is now a complete record of who failed to find out.

  • [ 0.000001] // 0x002: EAGAIN

    Say you give the chatbot a shell script that exits successfully after failing to copy the backup.

    The script was written by someone whose account still owns half of /opt, despite having left before the company acquired its current name. Nobody has removed it. Nobody knows what breaks if they do. This is called institutional knowledge.

    The bot spots the missing error handling. It starts explaining pipelines, exit statuses and why the last command succeeding does not mean the preceding commands achieved anything useful.

    Then suppose the allowance runs out.

    Lovely. The backup script and its replacement sysadmin now share a failure mode: neither has finished the fucking job.

    The demo never seems to reach this bit. An agent builds an application while a man in expensive trainers stands beside a plant. Nobody shows you, twenty minutes later, working out whether the last block of code is complete or merely optimistic.

    People say their week’s allowance ran out days early. The man selling abundance explains that the service costs more than it earns, asks whether more usage costs too much, and calls it sensible.

    I’ll tell the missing files.

    You could compare plans to establish which tier includes reaching the end of a sentence. Or open the manual. It continues existing after you read it.

    I open another terminal to automate the allowance checks. Start sketching something to watch the checker. Catch myself choosing a name for its service account.

    Close the terminal.

    I don’t object to computers having limits. I object to being sold emancipation from tedious work and finding another dashboard to tend. If your miraculous digital colleague needs rationing, put that in the demonstration. Show us who gets to finish.

    You fix the script yourself. Run it against a test directory. Check the exit status. Check the files. Only then do you touch the scheduled job.

    The morning’s automation breakthrough is complete.

    You are now doing your job and its job.