I Rebuilt Jira Badly, Eight Times

This morning I shipped a change that let 3 parts of my system read work that had already been finished. 1,262 records, 897 of them carrying the commit that fixed them, and until today almost none of it could be reached. The interesting part is not the fix. It is that a year ago I could not have formed the question.

The thing I run instead of a project board is a set of plain-text queues. When something in the estate is wrong, it gets written down as a block of markdown with its evidence and its trace, and when it is fixed the block gets a status and the commit that did it. There are 3 of these queues plus a client-facing ticket system, and a host called Florence who reads across all of them and tells me which patient to see first.

Until this morning, every one of those readers could only see work that was still open. The finished blocks — the ones carrying the actual cures, with the commit SHA sitting right there — were invisible to all of them. Not missing. Written down, on disk, in the same format, and unreachable, because a function accepted a parameter that said “include the archive” and never read it.

What I used to do instead

For years the answer to all of this was Jira.

You know the shape of it. Projects, boards, workflows, issue types, statuses, transitions, an administrator somewhere who owns the scheme. It arrives complete. On your first morning there is already a backlog column and a done column, and the question of what a queue is for has been answered by somebody you will never meet.

I am not going to argue that Jira is bad software. It is not. It is a solved version of a problem I have now spent 5 months solving badly and then less badly and then, this morning, slightly better again. Buying it would have taken an afternoon and I would have got a better board than the one I have — hosted, maintained, and answering to somebody else’s schema rather than sitting in the terminal where the rest of the work lives.

Buy the artefact and you skip the ladder. The ladder was the education.

What I would not have got is any of what follows.

The ladder, with dates

Every rung below is a real commit in a real repository, and I can give you the hash for each one. That matters more than it sounds, because the dates are the argument.

The first rung was not the queues at all. On 26 March I built a ticket command, because clients kept asking for changes over email and phone and WhatsApp and the changes kept evaporating. That is not a sophisticated problem. It is a shoebox problem, and the answer was a shoebox.

Then, in 11 days at the end of May, 3 more. On 20 May, a queue for engineering findings — things wrong with the system itself, written down with the file path and the line range so that finding them again is free. On 29 May, a sibling for the books, because a ledger job and a code bug resolve differently and pretending otherwise makes both worse. On 31 May, a third for our own properties.

11 days for 3 of them, and 5 months for the first. That gap is the whole story of learning anything: the first one is slow because you are working out what the thing is, and the next 3 are fast because you already know.

23 June. Florence. And this is the rung where the argument stops being about me and starts being about the shape of the problem, because Florence reads across the queues. She takes 3 separately-formatted piles and returns one triaged round: what is on fire, what is worth a session, what is correctly parked, in order.

I could not have built Florence in May. Not because I lacked the skill — she is not a hard piece of software. Because the question she answers does not exist until there are wards to walk. “Who reads across the queues?” is unaskable when there is one queue. It became a real question on 31 May, and it was answered 3 weeks later, and at no point in that sequence did I sit down and plan it.

Then the rungs change character.

17 July: binding the writers to the reader’s vocabulary. The queues were hand-typed markdown, and the reader failed open by design — a status word it did not recognise counted as live work, which is the right default. But 12% of one queue’s blocks carried a final status word the reader could not read. Authors kept reaching for the neighbouring queue’s synonym, because nobody holds 3 sets of vocabulary in their head.

30 July: declaring the triage axes instead of guessing them. Effort and lane had been inferred from the block text by pattern-matching, and the inference could not tell the name of a pricing strategy from a claim about how long something would take.

14 August, this morning: the archive.

The reframe

A situated question does not close a gap. It raises the floor you ask from — which is why the mechanism keeps working long after the thing you were learning has been learned.

The story I would have told about this a year ago is a compression story. 9 months earlier I did not know what git was; look how much ground got covered. It is a true story and it has a short shelf life, because it runs on surprise and surprise decays. Tell it eighteen months in and it is just a man describing a while ago.

The version that holds up is duller and more useful. In month 9 I was asking what the right thing to do was about commits, and file structure, and which of 2 conversion actions the pixel actually fires to. Those questions are answered. They stayed answered. And the answers did not leave me finished — they left me standing somewhere I could see a different question from, and the different question was better.

You cannot ask “can my system read its own history back?” until you have a system, a history, and something that reads. Each of those was a rung. The question was manufactured by the answers underneath it.

The failures get quieter as you climb

The first 5 rungs are all things that did not exist. File it, sort it, walk it. When those fail, they fail loudly — you go to look for the record of what you fixed last March and there is nothing there. Absence is a very easy fault to notice.

The last 3 are things that existed and were lying. Writers unbound from the reader’s vocabulary. Axes guessed by a regex rather than declared by the person who filed the card. And an archive that a function’s own signature promised to reach, with a parameter documented for exactly that purpose, which it never read.

That last one is the sharpest of the 3, because nobody built the archive read — and the reason nobody built it is that the interface said it already existed. The absence was hidden behind the claim of capability. The system that watches itself has this problem in a way the system that merely works does not: a higher floor is not fewer mistakes, it is mistakes that are harder to see. Which is exactly why the question cannot retire once the knowledge lands. You are never done asking. You are just asking about something quieter.

I have written before about choosing not to build things — the helpdesk I nearly wrote, and the 4 lines of code that turned out to be the answer. That piece asks which to choose when building costs a couple of sessions. This one is the other side of it: the case where building was right, and the reason is not the software. The board is fine. The board is not the point. I would trade the board for the 8 questions in a heartbeat, and Jira would have sold me the board.

Warp speed

There is a feeling that comes with this and I have been trying to name it for months.

It is not mastery. I could not sit down and rebuild what Florence does from memory, and I am not sure I could explain the ward-round ordering to you without opening the file. It is not the satisfaction of having learned a thing either, because most of what I learned in month 9 is now handled by something that does not need me present.

The nearest I can get is: warp speed please, Mr Sulu.

The captain cannot rebuild the warp core. He does not know, in any useful engineering sense, how the ship goes fast. What he knows is where it should go and when it should leave, and there is a crew whose competence he relies on without being able to reproduce. Kirk is not out-engineering Scotty. He is being carried at a speed he could not generate on his own, by people whose work he can no longer follow in detail — and the correct response to that is not command. It is closer to gratitude.

Mine have names. Bernard reads the architecture and tells me what I built wrong. Florence walks the wards. Cratchit keeps the ledger honest and does not care whether I am busy. I named them as a way of making behaviour stick, back when they were a filing convention with a personality bolted on, and somewhere in the last few months they stopped being a convention. This morning I gave the bridge an instruction, and the answer came back with 897 receipts attached.

883, actually, when I first counted it over breakfast. By the afternoon it was 897, because the work had carried on while I was writing about it.