Email Deduplication at 40,000 Emails Per Minute: No Locks, No Bottlenecks

When you're sending 40,000 emails per minute through 52 parallel workers, you'd think the hardest problems would be obvious from the start - queue throughput, API rate limits, infrastructure scaling. But one of the most unexpectedly difficult challenges we faced while building our multi-tenant Mautic platform had nothing to do with speed. It was about making sure the same person doesn't receive the same email twice.

Email deduplication sounds simple. It isn't. Especially when you can't afford to slow anything down.

Why Duplicates Happen in the First Place

To understand the problem, you need to understand how large organizations manage contacts in marketing automation systems.

Consider a network of public libraries. A single person - let's call her Anna - might be registered as a member at three different branches. In the marketing automation system, Anna exists as three separate contacts: one per branch. Each contact has the same email address, but different metadata - different branch, different registration date, different activity history.

Now the organization wants to send a newsletter to all active members. The campaign targets a segment built on contact records, not unique email addresses. Anna matches the segment three times - once for each branch membership. Without deduplication, Anna gets three identical emails in her inbox.

At small scale, this is mildly annoying. Anna might think the library needs better IT. At the scale we operate - hundreds of thousands of contacts per campaign, multiple campaigns running simultaneously across multiple tenants - it becomes a deliverability disaster.

Why Duplicates Are More Than Just Annoying

Email service providers (ESPs) and inbox providers like Gmail, Outlook, and Yahoo track sending patterns with remarkable granularity. When someone receives the same email multiple times from the same sender, several bad things happen:

Spam signals increase. ISPs see the same content hitting the same inbox repeatedly. That's a pattern they associate with poorly managed lists, which is a pattern they associate with spam.

Bounce rates get artificially inflated. If Anna's mailbox is full, three sending attempts generate three bounces instead of one. Your bounce rate triples for a single recipient, dragging down your sender reputation.

Unsubscribe and complaint rates spike. People who receive duplicate emails are more likely to hit "report spam" - and spam complaints are one of the most damaging signals for domain reputation.

Multi-tenant amplification. On our platform, all tenants share a single sending domain. One tenant's duplicate problem doesn't just hurt that tenant - it degrades the shared domain reputation that all tenants depend on. A careless campaign from one organization can impact deliverability for hundreds of others.

Getting deduplication right isn't a nice-to-have. It's a deliverability requirement.

The Naive Solutions (And Why They Fall Apart)

Before explaining what we built, let's walk through the approaches we considered and rejected.

Pre-scan and deduplicate the entire segment. Before sending, scan all contacts in the campaign segment, identify duplicates by email address, and remove them. This works logically, but it breaks our two-stage pipeline architecture. Our Stage 1 is designed to complete in seconds - it scans the segment, creates lightweight batches of contact IDs, and pushes them to the queue. Adding a full deduplication pass at this stage means loading and comparing potentially hundreds of thousands of email addresses before a single message is queued. That delays campaign start time from seconds to minutes, defeating one of our core design goals.

Central deduplication service with locks. Set up a Redis-backed service that every worker checks before sending. Worker picks up a batch, reads the email addresses, checks against the central dedup registry, sends only new ones. The problem: with 52 workers processing batches simultaneously, a central locking mechanism becomes the bottleneck of the entire pipeline. Every worker blocks while waiting for the lock. Your 40,000-emails-per-minute throughput drops to whatever the lock service can sustain.

Database-backed deduplication table. Insert every sent email address into a database table, check before sending. Same bottleneck problem as the lock service, plus you now have a database table growing by hundreds of thousands of rows per campaign that needs to be queried under heavy write load.

Redis set without locks. Better than a database - Redis is fast. But you've still introduced a centralized dependency. If Redis hiccups during peak sending, all 52 workers stall. And at our throughput, even Redis SISMEMBER calls add measurable latency when multiplied across every email in every batch.

Every naive solution trades throughput for correctness. We needed both.

Our Approach: Deduplication Without Centralization

We implemented deduplication within the batch processing pipeline itself, as a natural part of how batches are assembled and dispatched. The key design constraint was absolute: no central lock, no sequential processing gate, no single point of contention.

Each worker processes its batches independently while the system guarantees global uniqueness across all workers. The deduplication logic is woven into the pipeline architecture rather than bolted on as an external check.

We're intentionally keeping the implementation details high-level here - the specific mechanism is part of our competitive advantage and represents significant R&D investment. What we can share is the design philosophy: deduplication should be as fast as the pipeline it protects. If your dedup layer adds latency, you've just moved the bottleneck from the queue to the dedup service.

The approach was one of the most demanding engineering challenges on the entire project. Making it work correctly was one thing. Making it work correctly at 40,000 emails per minute with 52 parallel workers and zero performance degradation - that required multiple iterations and careful testing with realistic data.

Proving It Works: The Stress Test

We validated the deduplication system during our full-scale stress test: 1 million synthetic emails, all personalization tokens active, 52 parallel workers running at maximum throughput.

The results:

| Metric | Result | |---|---| | Emails processed | 1,000,000 | | Duplicate emails sent | 0 | | Performance impact of dedup logic | Not measurable | | Pipeline throughput | ~40,000 emails/minute | | Workers processing simultaneously | ~52 |

Zero duplicates. No measurable performance overhead. The deduplication logic runs at pipeline speed - it's effectively invisible in the performance profile.

We tested with deliberately duplicate-heavy data: contacts that shared email addresses across multiple tenants and multiple branches within tenants. The kind of data that would have caused massive duplication in a system without proper dedup.

Lessons Learned

Deduplication is a first-class architectural concern. We originally underestimated its complexity. It's tempting to treat dedup as a post-processing filter - something you add after the pipeline is built. In a parallel system, that's a recipe for either correctness bugs or performance regression. Design for it from the start.

"Simple" problems become distributed systems problems. Checking if you've seen an email address before is trivial on a single machine. Doing it across 52 machines processing different batches simultaneously, with no central coordinator, is a distributed consensus problem. Recognizing that distinction early saved us from building something that worked in development and failed in production.

Test with realistic data that includes duplicates. Our early tests used synthetic data with unique email addresses. Everything looked fine. It was only when we generated data that reflected real-world patterns - contacts registered at multiple branches, shared email addresses across organizational units - that the dedup challenges became apparent. Your test data should be at least as messy as your production data.

The best deduplication is invisible. Recipients never know it happened. They just get one email instead of three. The organization never sees inflated send counts. Deliverability metrics stay clean. That invisibility is the whole point - dedup should protect the system silently, without adding operational overhead or user-visible complexity.

The Bigger Picture

Email deduplication is one piece of a larger deliverability architecture that includes bounce handling, sender reputation management, and per-tenant isolation. Each of these systems must work in concert to maintain the trust that ISPs place in our sending domain.

At 40,000 emails per minute, every architectural decision compounds. An extra millisecond per message across a million emails is over 16 minutes of added latency. A 0.1% duplicate rate across a million emails is 1,000 duplicate sends - enough to trigger spam filters and damage reputation scores.

The lesson we keep coming back to: at scale, the problems that seem trivial in design documents are the ones that consume the most engineering effort in production. Deduplication was supposed to be a checkbox item. It became one of the most important engineering investments on the platform.

Building a High-Volume Email Platform?

If you're dealing with email deduplication challenges at scale, or building marketing automation infrastructure that needs to handle millions of messages without compromising deliverability, we've been through the hard parts. We can help you avoid the pitfalls and get to production faster.

[Book a free consultation](https://www.droptica.com/contact/) to discuss your email infrastructure architecture.