NoBackups

It worked in staging…

Category: Monitoring & Reliability

  • [ 0.000006] // 0x007: ESTABLISHED

    Imagine The Company’s dispatch screen showing sixty-four established connections. Beside it, a label needs a delivery address. I have the customer number. Apparently this is the difficult half.

    Sixty-four is a reassuring number. I have included it in reports. More than once, if we’re admitting things. It suggests enough spare capacity for somebody else to be the problem.

    The connection I need has been idle. The network path has since gone silent, but the client still lists the socket as established. No usable answer has been required of it lately. We have that in common, although only one of us gets counted as capacity.

    I could check the connection count again. There is a particular comfort in asking a question whose answer you already know how to put in a report. I have been known to adjust the column widths before sending it, which is still attention to detail if nobody specifies the detail.

    Instead I ask the application for this customer’s delivery address over that existing connection. One read-only lookup. No booking, no update, no opportunity to send sixty-four parcels while finding out whether anything is listening.

    The lookup has an application deadline. It expires without a usable answer. That ends this attempt’s wait; it doesn’t tell me what happened at the far end, or require the socket to disappear politely from the list.

    Keepalive needs enabling, and its settings matter. It wouldn’t promise an address anyway. This client’s wait for one is shorter than its transport’s patience with silence.

    I return to the label. Customer number, parcel weight, blank address. The capacity report still has sixty-four connections to offer The Company. I have one attempted address lookup and no address.

    There is plenty of room on the label for the connection count. The courier will still want a street.

  • [ 0.000002] // 0x003: EXPIRED

    Suppose the public site is serving an expired certificate. The status panel is green.

    You refresh it. Still green. I give it a third go, apparently in case the colour was a temporary misunderstanding.

    The Company has a wall screen for this. Six reassuring rectangles, visible from the corridor. The certificate has been expired since breakfast, but from a suitable distance we are having an excellent morning.

    My browser refuses the connection. I inspect what the public listener is presenting. Right hostname. Yesterday’s expiry date.

    Then I open the check configuration.

    It connects to a private address, using an internal hostname. Different listener. Different certificate. That certificate is valid, trusted by the checker and entirely pleased to be here.

    The public name never enters the conversation.

    There is no mysterious failure of encryption. Two connections reach two different places. One has a current certificate. We have put its result on the wall and allowed it to speak for the other one.

    You can see how this happened. The internal endpoint was convenient. It responded quickly. Nobody outside could reach it, which reduced the number of awkward things that might happen during a demonstration of availability.

    The check is doing exactly what we asked.

    I would like that sentence removed from my fucking working vocabulary.

    I start editing the network diagram. Move the public listener beyond the edge of the page. For a moment the entire visible system is healthy.

    Undo.

    I write down the public hostname instead. The one a customer actually types. Test that connection, with certificate validation, and the green rectangle finally has something relevant to be upset about.

    In this arrangement, renewing the public certificate and loading it into the public listener would let that check pass. A fresh connection would tell us whether we’d actually fixed it. Refreshing the wall screen would remain optional.

    The old check can stay. Its label needs work.

    I change “Website” to “Inside”.

    We have achieved the same availability, with a smaller claim.

  • [ 0.000000] // 0x001: NULL

    [ 0.000000] // 0x001: NULL

    Somewhere in a database you forgot to back up in 2017, a value lives. Or rather, it doesn’t.

    It is NULL, sitting in a table that hasn’t been vacuumed since a language version change was considered news. For seven years, it did nothing. A harmless gap in the records.

    Then a monitoring script, written by a developer who has since pivoted to organic sourdough, hit a new runtime environment. It expected an integer. It found NULL.

    The script crashed. The monitor went dark. The alerting service failed to trigger because its own heartbeat depended on another API undergoing scheduled maintenance.

    At 3:00 AM, staring at a garbled trace, you realise you were meant to be a sheep farmer.

    Sheep are predictable. They don’t require routing updates. They don’t have nested dependencies on startups. When a sheep stops moving, you don’t check the logs for a firmware mismatch; you accept that the entity has reached its End of Life and move on. Sheep don’t need reboots.

    But you aren’t in a field. You are in a terminal. That unbacked-up row from 2017 is still there, waiting for your next deployment.

    It worked in staging though, didn’t it?