Fixing Postgres’s biggest mistakes

Postgres is one of the greatest databases out there. It’s the default choice for almost every startup I talk to. Calling its design choices “mistakes” is a little unfair, since most of them made total sense on 1990s hardware. Still, in 2026, if you run Postgres long enough without a dedicated database person, you’ll hit the same four problems over and over. I wrote about them back in May in The four horsemen behind thousands of Postgres outages:

  1. VACUUM and transaction ID wraparound
  2. Connection limits and query parallelism
  3. Bad query plans
  4. Missing jsonb statistics

When I wrote that post, pgrust (our reimplementation of Postgres for modern hardware) was still early, and most of our plans to fix those four problems were still ideas on a whiteboard.

Since then, pgrust hit a 100% pass rate on Postgres’s upstream regression suite, and none of the four horsemen are whiteboard ideas anymore. We’ve already shipped our fix for one of them, and we have internal prototypes for the other three that we’re wiring into pgrust right now. Here’s how we’re fixing all four.

1. VACUUM and Transaction ID Wraparound

VACUUM is probably the number one cause of Postgres outages I hear about. Suppose you run a harmless looking update on a busy table:

UPDATE users SET last_seen_at = now() WHERE id = $1;

Postgres doesn’t overwrite last_seen_at in place. Instead, Postgres writes a brand new copy of the entire row and marks the old row copy as dead. If that new copy lands on a different page, Postgres also has to insert new entries into every index on users, even when none of the indexed columns changed. Later on, a background VACUUM job has to scan your tables and indexes, physically clean up those dead rows, and reclaim old transaction IDs before they wrap around and Postgres shuts down your database to prevent corruption.

This creates a nasty operational trap on tables with heavy writes. Run VACUUM too aggressively and it’ll eat up I/O that your live queries need. Run it too slowly and dead rows pile up, your tables and indexes bloat, queries slow down scanning half empty pages, and you’ll creep toward transaction ID wraparound.

Back in May, I wrote about looking into storage designs that don’t need VACUUM at all. Since then, we’ve built an internal prototype of a new storage engine around an undo log and a lock free buffer pool. It draws on ideas from Umbra, OrioleDB, LeanStore, HyPer, and InnoDB, with a few twists of our own after benchmarking on real hardware.

Instead of scattering dead row copies across your table, we keep the newest version of each row right in place (like HyPer and Umbra) and push older versions into compact undo deltas for active snapshots and rollbacks. Once active transactions move past an old version, that undo space gets reused directly without scanning your tables. Updating an unindexed column like last_seen_at doesn’t touch secondary indexes, dead tuples don’t bloat your pages, and there’s no transaction ID wraparound cliff.

The early numbers from our standalone prototype are pretty wild. On sysbench with durable commits, the prototype ran many times faster than stock Postgres with lower tail latency. Right now, we’re wiring that engine into pgrust as a Table Access Method and testing it on real SQL workloads.

2. Connection Limits and Query Parallelism

When you spin up a Postgres instance, you have to pick a hard max_connections limit up front. Changing it requires restarting your database. If traffic spikes past that limit, Postgres won’t let new clients connect and your app goes down. Most teams end up running PgBouncer in front of Postgres just to survive connection spikes.

Postgres has this problem because it spawns a separate OS process for every client connection. Processes are expensive to spin up and expensive to keep around. That’s painful in two ways. It caps how many concurrent connections you can hold, and it makes parallelizing work within a single query expensive enough that Postgres only bothers spinning up parallel worker processes on big, long running queries.

We tackled this in two steps.

First, we architected pgrust from day one as a single process database using threads. One big reason C Postgres stuck with processes for decades is fault isolation, since a stray pointer bug in one C process is less likely to corrupt another process’s memory. Because pgrust is written in Rust, we get memory safety at compile time without paying for OS process isolation. Queries run on a morsel-based shared work stealing thread pool, which makes parallelizing work both across connections and within individual queries cheap.

Second, we wanted to go further than OS threads for connection scaling. Threads are much lighter than processes, though holding tens of thousands of mostly idle connections from serverless functions or microservices still wastes a lot of RAM.

To fix that properly, we’ve built an internal prototype of lightweight fibers in user space. Whenever a session has to wait on a client socket between statements, a row lock, a WAL commit flush, or a page read from disk, the fiber suspends at that wait point and lets another session run on the same core.

When a connection sits idle between queries, it returns its execution stack to a pool and uses 1/10th the memory of an idle thread backend, while context switches run orders of magnitude faster than OS thread switches. You can hold tens of thousands of connections directly against pgrust without PgBouncer in front.

3. Bad Query Plans

Postgres’s query planner is smart, right up until it isn’t. Most of the time it picks a great plan. When it gets a plan wrong, though, a fast query can suddenly run hundreds of times slower. Worse yet, stock Postgres doesn’t give you planner hints out of the box. When a bad plan hits production, you have very few levers to pull. At a previous company I worked at, we had to turn off nested loop joins globally because Postgres kept picking them and tanking query latency.

A big reason this happens is how Postgres estimates row counts across filters and joins. Suppose you run a query like this:

SELECT count(*)
FROM orders o
JOIN customers c ON c.id = o.customer_id
JOIN shipments s ON s.order_id = o.id
WHERE c.country = 'CA'
  AND c.currency = 'CAD'
  AND o.status = 'disputed';

Postgres keeps single column histograms and mostly assumes conditions across columns and tables are independent. In real data, country and currency are heavily correlated, so multiplying their selectivities together badly underestimates how many customers match. Across several joins, those underestimates compound by orders of magnitude. Once Postgres thinks an intermediate join will return two rows when it actually returns two hundred thousand, a routine ANALYZE can flip a fast hash join into a disastrous nested loop join.

Back in May, an adaptive query planner was the most theoretical item on my list. Since then, we’ve built internal prototypes that tackle bad plans both before and after a query runs.

The key idea powering the adaptive planner is a feedback loop. Whenever a query finishes, pgrust compares the planner’s estimated row counts against the actual row counts observed at each join and learns a correction weight where the estimate missed badly.

When we first built that feedback loop, we actually saw performance get worse. After some research, we found we hit a well-known trap: “fleeing from knowledge to ignorance.” Suppose the planner underestimates two candidate join orders, picks Plan A, and discovers at runtime that Plan A actually produces way more rows than expected. On the next run, the planner now knows Plan A’s true cost, while Plan B is still wildly underestimated. So the planner “flees” from the plan it just measured to the unmeasured, still underestimated Plan B! On one Join Order Benchmark query in our early tests, that made the query run over 10x slower after feedback than before.

4. Missing jsonb Statistics

Postgres makes it super easy to store and query semi structured data with jsonb. It also makes it super easy to shoot yourself in the foot once that table grows.

At Heap, we used jsonb heavily and it was a massive query planning headache. Imagine a query filtering on a jsonb property and joining against another table:

SELECT e.user_id, e.created_at
FROM events e
JOIN accounts a ON a.id = e.account_id
WHERE e.properties->>'plan' = 'enterprise'
  AND e.created_at >= now() - interval '7 days';

For normal columns like created_at, Postgres collects rich statistics during ANALYZE, including histograms, most common values, and distinct counts. For keys and values inside e.properties, stock Postgres collects none of that. When you filter on a jsonb field, Postgres falls back to a hardcoded magic number and guesses that 0.1% of rows will match.

The real match rate might be almost every row in the table or almost none of them. Postgres has no idea either way. As soon as you combine jsonb filters with joins, that 0.1% guess causes the planner to pick terrible join orders and wrong indexes all the time.

Unlike the first three horsemen, you don’t have to wait on our internal prototypes for this one. We shipped native jsonb statistics in pgrust v0.4-beta. During ANALYZE, pgrust inspects the keys and values inside your jsonb columns and builds real statistics on your JSON paths instead of guessing 0.1%. Because the planner actually understands the distribution of data inside your jsonb documents, it can pick sane join orders and index scans out of the box without forcing you to maintain manual expression indexes for every path you filter on. Depending on your workload, jsonb queries can run orders of magnitude faster right away.

Try an Early Version or Follow Along

To be clear, pgrust is still in beta today, so don’t put data you can’t afford to lose on it yet. Native JSONB statistics and our thread-based execution model are already out in pgrust v0.4-beta, and we’re wiring our new storage engine, user-space fibers, and adaptive query planner into pgrust next.

If you’re interested in trying an early version of any of the pieces I covered here, shoot me an email at malis@pgrust.com. Running against real test workloads and replicas helps us a ton. Even if you just want to chat about gnarly Postgres problems you’re running into in production, I’d love to hear from you.

Thanks for reading. If you want to support pgrust, the best way to help out is to give us a star on GitHub. You can also follow along and reach us here:

Leave a Reply

Your email address will not be published. Required fields are marked *