Weeknotes
Weeknotes for the week finishing Friday, 12th June 2026.
Two weeks’ worth of weeknotes this week.
Meetings aside, I’ve spent much of my time trying to understand why DuckDB ran out of memory during a long-running process. The problem started when I was taking annual leave, so my first task was to understand what happened, when it happened, and how the team responded.
On 20th May, DuckDB ran out of memory when ingesting about 40GB of prescribing data; the process was terminated by the OOM Killer. The team made several judicious configuration changes, but couldn’t confidently determine the cause of the problem. This was because the feedback loops were long — the process normally takes around three hours to complete — and because the team needed to ship new features at the same time. Consequently, it became hard to separate the effects of configuration changes from the effects of new features changes.
I started by checking out each commit in a range, running the process, and recording the maximum amount of physical RAM it used. Surely, a smoking gun?
No such luck (and don’t call me Shirley). The jump was because the team increased DuckDB’s memory limit. For unrelated reasons the process failed just before and just after, hiding the commit amongst several other commits.
Next, I cloned the production server. I ran the process as it existed on 20th May and as it existed on 22nd April, when it previously ran to completion. Again, I was hoping for a smoking gun; again no such luck. In both cases, the process ran to completion. The graphs of memory usage over time were very similar.
I tried several other cases, covering an upgrade to DuckDB from 1.5.1 to 1.5.2, and seeing whether reducing the number of threads would reduce memory usage, as the DuckDB docs suggest. Again, the graphs of memory usage over time were very similar.
I’m going to think about the problem next week. However, it seems likely that whilst the process didn’t change, the environment did. We run two web apps on the production server; although we think their memory usage is low, we don’t know their memory usage is low.
Time’s up!