Hi everybody - over the last few months, Iāve shipped more features on kirupa.com than I did in the previous ten years. And through all of it, the siteās many MANY duct-taped pieces kept working.
The reason is Nightowl, my Avengers-like codename for a small crew of AI agents that works the day and night shifts on kirupa.com. In this video, I walk through how it all works:
Nightowl is whatās known as an AI software factory. Below, Iāll quickly cover how it helped kirupa.com and how you can build your own in a balanced way.
If you want more of the gory details than what this newsletter covers, I wrote about Nightowl and AI software factories in a lot more depth.
Onwards!
How Nightowl Helps kirupa.com
Coding agents have made writing a fix (that works!) cheap. What they havenāt made cheap is my attention. Someone still has to notice a bug, figure out whatās actually wrong, check the fix, and decide if itās safe to ship. On a site with about 3,900 HTML pages and a bunch of integrations added over the decades, that someone was always me:
When something in there breaks, the failures are often subtle. For years, my plan was to fix things when somebody emailed me about them. Thatās not a winning strategy. Who knows how many people ran into a broken feature and just abandoned the site altogether? Relying on the kindness of people submitting bug reports doesnāt scale. Relying on me having bandwidth to triage and fix the bug doesnāt scale either.
An AI software factory is a set of agents and automated checks that keeps finding, sorting, and fixing problems in a codebase, with humans standing at the gates that matter. Here is what that looks like for kirupa.com:
Reading it from left to right, a problem goes through these steps:
Site audits, a tiny error reporter on every page, and kirupaBot notice that something is wrong. Plain code (no AI) weeds out duplicates and files a GitHub issue.
Every five minutes, four agents take turns on each issue. One classifies it, one gathers evidence, one proposes a fix, and one reviews that fix. Each agent posts what it found as a comment.
Small fixes like a broken image, a typo, or a broken code snippet go live on their own. Anything involving design or taste waits for my thumbs-up.
Some of what it catches is a little embarrassing. The very first audit found that the sitemap was capped at 500 URLs, so search engines had never heard about nearly half of my modern tutorials, and 72 pages sent people to a completely different article whenever someone shared them. Both of those are now checks that run before every publish, so they canāt sneak back in.
My favorite part is that the agents are allowed to say I donāt know. When a Type error showed up in one of the siteās shared scripts, the file it pointed at wasnāt in the code the agents could see, so the Analysis agent said it couldnāt confirm a cause and tagged me instead of guessing. Iāll take that over a confident fix for the wrong problem any day. By the time I sit down with my coffee (Nescafe āļø instant coffee FTW!), the boring detective work is mostly done, and thatās what lets me move fast on a very old site without worrying about unnecessary changes breaking things.
Building Your Own
Before you add a single agent, you need one place where every issue lives. Audit results, error reports, and bugs people tell you about should all land in the same list, and that list is where you and your agents work together. For me, thatās GitHub Issues:
Look at the labels. Some came from me, and some came from the agents. The purple factory:human-review label is the one I watch most closely, because it means the agents stopped and an issue is waiting on me. GitHub Issues works great, but Linear, Jira, or whatever tracker your team already uses is fine too. What matters is that you and your agents are looking at the same list.
Next come the agents. You need at least four of them, and each one gets a single narrow job along with a clear list of things it isnāt allowed to do:
Here is what each one does, in the order they run:
Classification reads the issue and decides whether itās a bug, a feature request, or something it isnāt sure about. When it isnāt confident, it asks for a person instead of guessing.
Analysis gathers the evidence: whatās broken, where, and what āfixedā would look like. It isnāt allowed to propose a solution. A model asked to analyze a problem will happily jump straight to a fix, and once a fix is on the page, every later step anchors on it.
Implementation is the first agent allowed to propose a fix, but it never gets to approve its own work.
Review picks apart that one proposal for side effects and anything that wasnāt tested, without writing a fix of its own.
Each agent posts its result as a comment on the issue, so the whole investigation is right there for you to read when you get to it. Depending on your project or domain, you may add more agents that are specialized for what you are trying to do. The four I listed are just the bare minimum you need.
Finally, something has to kick the agents off. For kirupa.com, thatās a cron job on the same server that hosts the site. Every five minutes, it looks at the open issues and moves each one forward a step or two. Between agents, a separate check makes sure the previous agent followed its rules. If Analysis snuck in a fix, for example, nothing gets posted and the line stops right there.
You donāt have to turn all of this on at once, though. In fact, I would recommend you take it slow. Start with the tracker and a Classification agent that only sorts issues, which is exactly how the Vercel team started theirs. Then add Analysis, Implementation, and Review one at a time, once the agent before them has earned your trust.
Where you start also depends on whether your project is brand new or has been around for a while:
On a new project, you can build the factory with all its bells and whistles from day one. Use one tracker from the very first bug, and write a test for every bug you fix so your checks grow along with the code.
On an existing project with years of old issues, start read-only. Point your Classification agent at the backlog so it learns which problems keep coming back, and fit the agents into the tracker and labels your team already uses. Then run them in shadow mode for a while, where they do their job on every issue but you only compare what they would have done with what you actually decided.
This whole AI Software Factory space is fairly new and the best practices are continuously evolving. Do share what has or has not worked for you so that we can all collectively learn.
When to Automate and When to Keep a Human Around
Once agents start proposing fixes, the big question shows up. Which fixes can ship on their own, and which need a person to approve them? For every kind of fix, I ask four questions:
How bad is a mistake, and how easy is it to undo?
Can a test prove the fix is right?
How often does this kind of problem come up?
Does it need taste?
The answers donāt give me a yes or no. They tell me where to set a dial. Hereās where each part of kirupa.com sits today:
Look at the two ends of the dial. On the fully automated end are the site checks, the issue filing, and small fixes like broken images and typos. They happen all the time, and when one goes wrong, itās tiny and easy to undo. On the other end, I write every new tutorial myself. A test can prove that an image loads, but no test can tell me whether a diagram makes sense or whether a paragraph sounds like me.
If youāre just starting out, have a person approve everything. Then move one narrow kind of fix up the dial at a time, and only after it has a long track record of fixes you approved without changing anything. The first time it ships something bad, it drops back down a notch and has to earn its way up again.
Till Next Time
My goal is to automate 100% of the maintenance and plumbing on kirupa.com, and to have full confidence that nothing will break when I do. Will I ever get there? I donāt know. But Nightowl has already taken me a lot further than I expected, and it leaves me more time for the parts Iād never hand off anyway, like writing new tutorials and drawing the diagrams that go in them. I hope it gives you a few ideas for your own projects!
Whatās the first task youād hand off to a factory of your own? Reply to this email, post on the forums, or find me on Twitter / X.
Cheers,
Kirupa š¦






