Picking Parallel Consumer up again

TL;DR: Parallel Consumer is getting picked up again. My fork at astubbs/parallel-consumer is the project’s active home going forward. Same code, renewed commitment to maintenance, PRs being reviewed, releases being cut.

The short version

I started Parallel Consumer in May 2020 while at Confluent. It became a Confluent Labs project, I gave the Kafka Summit Europe 2021 talk introducing it, and it grew to 548 stars by mid-2023. I left Confluent at the end of February 2023, and since then the Confluent repository has been quiet.

I’m picking it up again. Not forking it away from anyone — just making sure the project has someone committed to maintaining it.

Why it exists

Traditional Kafka consumers scale by adding partitions. If you want 100 messages processed in parallel, you need 100 partitions (or 100 consumer instances). That’s wasteful — you pay for partition overhead whether you need it for ordering or not.

Parallel Consumer flips this: one consumer instance processes many messages concurrently, while still preserving ordering where it matters (by key, by partition, or not at all — you pick). Higher throughput, lower latency, less infrastructure.

What “active home” means

I’m not putting down the Confluent version. The code is identical — the fork just has someone actively maintaining it. Specifically:

• PRs get reviewed
• Releases get cut
• Issues get triaged
• Questions get answered

If that’s valuable to you, point your dependency at the fork. If you’re happy with the original, keep using it.

Why this is a good thing

Kafka needs small, focused libraries like this. The standard consumer API is decades old at this point and the gap between “what Kafka gives you” and “what most teams actually want” is wide. Parallel Consumer filled part of that gap, and I want to make sure it stays maintained so teams keep getting value from it.

More practically: I’ve kept using it on side projects and want the bug fixes and improvements I’m making to flow back out to everyone else.

The star count detective work

One weird side note from the research for the project page: the Confluent repository’s accumulated star count got reset at some point in 2024. I went looking for the historical numbers to preserve the record, and the Wayback Machine came through:

• Early 2023: 479 stars
• August 2023: 548 stars (proof)
• 2024: reset to 123
• Recovering since

Not important, but it was a fun bit of internet archaeology. The full timeline is on the project page.

What’s next

• Tidying up the backlog
• Cutting a snapshot release to establish the new maintenance cadence
• Reviewing open PRs

Feedback, issues, and PRs welcome at github.com/astubbs/parallel-consumer.