Table of Contents
Reporter and Editor
A journalist friend of mine once told me that in the standard reporting pipeline, you always have an editor. The original writers are often too verbose and rambling, and it is the editor’s job to filter out the noise.
Journalists often have a beef with editors for that reason. Editors have their own vision and the ultimate power over the final text.
I find this framing useful when thinking about LLMs and text generation.
Commits as Inputs
My commit messages are now mostly LLM-generated, and my recent commit history is rather verbose. My average commit message used to be a one-liner, probably comfortably below 80 chars. Not anymore. Now my commit messages are longer than the actual code diff.
That makes it trivial to feed the commits into an LLM and ask it to summarize everything into a draft report. The goal is to get rid of the repetitive, exhaustive job of manually reviewing every commit and turn the verbosity into an asset rather than a liability.
We publish two kinds of monthly reports for BTC Map. One is the consolidated report on the BTC Map Blog, and the other is my personal report, published here on this blog.
I started feeding an LLM a time period and a list of git repos, then asking it to produce a draft changelog. My job shifted from journalist to editor. I take the draft, shorten it, rearrange what survives, and remove the slop.
Trouble in Paradise
Did the report quality improve? The answer is: yes and no.
For my personal reports, quality did improve because I tend to forget details and clankers have a good memory. I still spend roughly as much time preparing my reports as before, but my reports are better both in terms of grammar and content. The boring part is faster, so I can shift my focus to readability.
I cannot say the same about our consolidated reports though. I suspect the reason is that they are just unedited LLM outputs. I am not just speculating here, it is trivial to see. LLMs cannot figure out which changes are meaningful, which changes are inappropriate for the format, and which ones are just noise.
My personal report is supposed to be technical, since my contributions are mostly code-focused. Our consolidated reports target a completely different audience, so they used to be focused on new partnerships, usage statistics, and so on. But because both reports are fed the same commit history, they include all kinds of irrelevant noise and nitpicky details derived from raw commit messages.
Conclusion
I find LLMs useful for report generation, but here is my personal rule: it is not a good idea to publish something you did not even bother to read a few times and fully conceptualize yourself.
If I am not ready to read, fact-check, and polish my own text, why would someone else read it? Respect for the reader is fundamental. New fancy tools don’t change anything in principle.
This morning, I saw someone on Twitter defining AGI as “Actually Generating Income.” I think it is a good framing. We see an explosion in textual output, but do we see an explosion in quality? The answer so far is a definitive no.