When I’m going on long walks, I like to keep a paper notebook. This is something I’ve been doing since my Very First Long Walk back in 2018.

A picture of a closed paper notebook with the text 'E5 2018 / Innsbruck-Venedig / Gebirge Tagesbuch' labelled in black handwriting.

I wrote the whole thing in German; Fernwanderwege (long distance hiking trails) are the best German language schools you could ask for.

On the first few pages, I’ll write my hike itinerary (from/to, accommodation, distance, food stops), and the rest is a journal. I like journalling on paper, and I like being mostly offline when I’m walking; I’m trying to disconnect from the internet a little.

A page from a notebook with a handwritten table, with an itinerary showing dates, distances, and start and stop locations.

the first pages are always the neatest.

I also like being able to search my journal, and have it backed up. I’m very good at losing things. Scanning and OCRing the journal would be really nice. There’s just a few problems.

First: my handwriting is terrible. My dad once described it “as if a dying spider limped through a puddle of ink”, which I think is unfair (to dying spiders).

A handwriting sample with the start of each line cut off.

Here I am getting mad about a poorly-signposted five kilometer detour due to highway works and trying to decipher my own writing a day later (it says ‘signpost anything clearly’).

Second: My journals are a bit of a stress test for what counts as a ‘real word’. Place names are sometimes a little crazy. Also, I make spelling mistakes sometimes (also sometimes on purpose!), and write in a mix of UK and US English, and sometimes the occasional word that’s only in my brain in German.

A handwriting sample with three variant spellings of the word 're-remembered'.

re-remembered. remememberd? reremembered?

Third: My journals are a mess of scribbles with words scratched out sometimes and with the occasional e.g. arrow symbol to denote the way a particular bird sings:

A handwriting sample that says 'One goes →↗ like a whip sound.'

I don’t think I ever found out which bird this was.

I tried a few OCR solutions, including Transcribus, a new app called Flowstate, and OlmOCR. But they all struggled with my handwriting.

I had heard that the LLMs are pretty good at transcribing handwriting into text. But I don’t just want to just blindly trust the output without checking it, so I built a little app around proofing and correcting.

It is (imaginatively) named Notebook Tools.

How it works

Here’s a video:

Basically:

Why LLMs?

They have an idea of a world model which is uniquely powerful for figuring out what the hell I’m writing about. Because they’re trained on ~everything, they can figure out likely words from context in ways that aren’t possible with other models. This works great with place names – “Cotswolds Way” and “Doddington” implies that that messy word is likely “Tormarton” and that scribble at the end of the sentence is likely “Old Sodbury”. It means when I talk about the pub I’m staying in and scrawl “Dave, the Hauswirt” – a German word that might not even really belong in context1 – the LLM can still figure it out.

A screenshot from the app, with the text 'Dave, the Hauswirt'. Hauswirt is highlighted for correction.

Flagged as unusual, but still accurately transcribed.

You also see evidence of the world model in other, weirder, ways:

A screenshot showing a correction overlaid on top of handwriting. The handwriting clearly shows the word 'reheated', but the LLM-suggested alternative is 'toasted'.

I am sorry but there is no way this says ’toasted'.

Anthropic APIs for transcription

I am using the Anthropic API for this instead of just using a consumer-grade Claude.ai account. I couldn’t just plug in to my existing subscription because of some restrictions I don’t understand. This has implications on cost (it’s worse) and privacy (it’s better by default). The worst thing is that the UX for enabling transcription is kinda terrible as a result. Let’s talk about each of these in turn.

Cost

Transcription costs about US$0.05 - US$0.10 per photo, but each photo can be roughly a two-page notebook spread (two 13 x 21cm pages). The cost depends on the model in use: the less-powerful Sonnet costs about $0.05 per spread and the more-powerful Opus costs about $0.10. I found Opus was better enough to warrant the extra cost, but I guess Sonnet will work fine for people with halfway-legible handwriting. I added a little option switch for this:

A screenshot: 'Opus: More accurate, and handles formatting better, but slower and more expensive. Best if you're in a hurry and want to skip proofreading, or your handwriting is terrible. Sonnet: Not quite as accurate (but still really good). Faster, half the price.'

I am team Opus

Part of why it’s expensive is the production of bounding-boxes and alternatives. If you just get the LLM to dump text, you can get away with much cheaper transcription, but the bounding boxes make the transcription UI and the exported PDFs really nice.

A screenshot of the app in 'Transcript' mode. The left half shows a photo of a notebook page, the right half shows the transcript. The currently hovered line and suggestion in the transcript highlights a corresponding bounding box on the notebook photo.

I love that mousing over a line on one view highlights it on the other.

Privacy

Privacy is a concern; I feel a little uncomfortable about the fact that I’m shipping my journal to a company whose business model craves human-authored text (but not uncomfortable enough; and I guess I think that Anthropic are one of the better ones). Stuff that’s sent over the APIs is apparently not used for training and theoretically mostly deleted after 30 days.

Login is a pain

This is the big one. Currently, to use transcription in the app, you need an API key of some kind, and that means:

This really bothers me, because it means that Notebook Tools isn’t something that I’d ever really productionise / distribute outside of developer circles. If I wanted to distribute more broadly and make the UX halfway reasonable for ‘regular people’, I’d probably have to:

I don’t want to have to deal with any of that. It feels like there’s something powerful about the idea of being able to “BYO machine intelligence”, instead of an LLM provider being something that every app needs to deal with (and, probably, bill for) individually.

Other models?

The post that sparked this idea explicitly mentioned Gemma 4 31B because it’s way closer to ‘commodity’: you can run it in loads more places, and much more cheaply. Maybe it’d be good enough for me? I prototyped Gemma 4 running on Tinfoil (which runs in a secure enclave and therefore has strong privacy guarantees!) and found that it was able to transcribe a two-page spread for less than one US cent ($0.01). For my handwriting, it was a little less accurate than Sonnet again, and much worse at (a) place names (b) identifying areas of uncertainty and alternatives. But at a fraction of the cost of the Anthropic models, it’s probably not a bad option for most people.

It’s got a slightly lower max image resolution than Opus / Sonnet, which might be a contributing factor – at some point, I’ll (maybe) experiment with splitting a spread into two individual pages before sending to the LLM, and see if that helps with accuracy.

Gemma 4 seemed to not be able to fathom that there’d be more than one interpretation of a single word though. Here’s some junk from the ‘reasoning’ where I’d written the word Dorfromantik2:

Wait, "Dortromantic" is really weird. Let me look again. "Dortromantic"? "Do romantic"? "Dornromantic"? Looking at the 'r' and 't'... it looks like "Dortromantic". Let me check if that's a word in some dialect. No. Maybe "Dreamromantic"? No. Maybe "Dormant-romantic"? "Dormant" as in sleeping. "Dormant romantic somehow". That makes more sense. "Dormant romantic". Let's see if it's "Dormant". The 'r' is clear, then 'm' then 'a' then 'n' then 't'. Yes, "Dormant romantic". Let me re-examine. The 'm' might be 'rt'. "Dortromantic". Let's look at the letters. D-o-r-t-r-o-m-a-n-t-i-k. It ends in 'k' or 'c'? Looks like 'k'. "Dortromantik". German? "Romantik" is German for romantic. "Dort" in German means "there". "Dort romantik" = "there romantic". That's very likely since the author mentions "Kempner Hütte" and "ES" (maybe Europäischer Fernwanderweg?) and "full english" (which they are comparing with something else). "Dort romantik somehow". "Dort romantik" isn't quite right grammatically in German, but a learner might write it. Or maybe it's just "Dort romantic". Let's look at the letters again. D-o-r-t-r-o-m-a-n-t-i-k. Yes, it looks like "Dortromantik".

I wonder how it would change if I’d written the prompt in German.

Development approach

100% of the code in the repo was written by Claude Opus 5.5. I’ve skim read every change, and banned the agent harness from making its own commits. It is definitely faster than making software by hand (and weirdly more capable; there’s loads of tasks where I’d just have no idea of how to get started, like edge detection). You get more ‘software’ for the same amount of effort.

I’m trying to work at a level of abstraction that’s higher than writing code, but in the weeds about all the details, which pretty much precludes ‘one-shotting’ or ‘slot-machining’ anything. The problem, is of course:

I tried a few new (to me) ‘knowledge management’ things in an attempt to make working with the agents more… useful? maintainable? In particular, I’ve got a decision log stored as a set of markdown files in decisions/, and an evolving list of forward-looking tasks in tasks/. I mostly write the tasks myself, but sometimes we’ll3 spin parts of the current task out into future tasks if it seems like the scope on the current task is creeping, or the task is more complex than I initially believed (it is software development, this happens constantly). The decisions/ are mostly agent-maintained based on conversations that we have when planning a feature. The idea is to capture who made certain choices and why, so that e.g. I can tell if a certain behaviour exists because I / the LLM explicitly had a reason for it, or if it exists by accident. I’m hoping that this’ll help me avoid Chesterton’s Fence-style situations in the future. But I think it’s also I think a way of wrestling with the fact that I want the effort to be seen, and the existence of the code doesn’t prove much anymore.

When implementing tasks/, I’ll typically read through the task file and add stuff to it, especially for things that I’ve got a lot of opinions about, and then kick off work with a prompt in plan mode like this:

/plan
Hello, today we're working on `tasks/missing-page-export.md`,
please ask lots of questions to resolve ambiguity. ty

Then, a bunch of back and forth on decisions and eventually execution. I’ll occasionally add a task for a refactor if I’ve spotted something design-ish that’s frequently getting in the way, and sometimes ask the agent explicitly to look at the list of upcoming tasks and identify refactor candidates that will make future work easier. Or ask, “what did you get stuck on this session? Are there any changes that we should make to reduce friction in future?”

Overall, it’s worked pretty well, and I haven’t had many problems with maintainability. But the project is still very small, and also greenfield.

Building just for yourself

Both before LLMs and after them, the temptation has always been Scope Creep. Figuring out how to cut scope, deciding ahead of time that a project is going to be just for you and then actually sticking to that and calling it finished earlier – giving the system enough grace to make it fun to use, but treating code changes as part of that use – is all the battle.

Building a demo – or something that’s useful on just one machine, in just one point in time, on just a limited dataset – is about 20% of the work. Making it distribution-grade is the other 80%.

There are lots of examples of that ‘other 80%’ work in this project:

This is maybe all unnecessary, but I’ve still put a lot of thought into small details because they make things ergonomic for future me. I will forget how this all works in 6 months, probably, and so the more the tool is ‘complete’ (with tooltips! and inline help! and install instructions! And a cute dock icon!) The more likely it is to be useful to me in the future.

Also, just deep in my bones, I like making things that are polished. I’m convinced that it’s possible to take an anti-slop (or at least, min-slop?) approach to LLM-driven-development, and I’m trying to illustrate that approach in this project.

That’s it

That’s all for now! The software is on Github, it works on recent MacOS.


  1. I thought this meant the owner / innkeeper / proprietor, but now that I’m writing about it, I wonder if I’ve learned it incorrectly from context and/or gotten confused with Gastwirt. But it doesn’t matter! The LLM still figured it out. ↩︎

  2. Dorfromantik: A “nostalgic feeling for the countryside”, and also a great board / videogame. ↩︎

  3. I’m a bit uncomfortable using the word ‘we’ to refer to development that I’m undertaking with an agent harness, but don’t really have any good alternatives yet. ↩︎