RabbitMQ & Async Messaging
The most common cause of a system feeling slow is not slow code — it is fast code waiting on something else. A message queue is how you stop a third-party API's bad afternoon from becoming your outage.
Non-blocking
External calls never block a save
Retry + DLQ
Failures recovered, not lost
Decoupled
Services fail independently
Absorbs
Traffic spikes without loss
Ask why an application feels slow and the answer is usually not the application. It is a payment gateway taking four seconds, an e-way bill API being unresponsive, an email provider rate-limiting, a PDF being generated inline, or a report aggregating a million rows while a user waits with a spinner. Every one of those is work that did not need to happen before the user got a response.
A message broker fixes that structurally. The request does the minimum required to be correct — validate, persist, acknowledge — and everything else becomes a message on a queue processed by workers. The user gets an immediate response. If the downstream service is slow or down, messages wait; when it recovers, they are processed. Nothing is lost and nothing was blocked.
We use RabbitMQ across the systems where this matters most: factory systems where a shop-floor save must never wait on a government API, financial platforms where a gateway timeout must not become a duplicate charge, and education platforms where result computation for a whole cohort cannot compete with learners' requests.
How Queue Implementations Go Wrong
A badly implemented queue is worse than none — it loses work silently.
No dead-letter queue
A message that cannot be processed is retried forever or discarded. Either way the failure is invisible: the queue looks healthy while work quietly disappears and nobody finds out until a customer asks.
Consumers that are not idempotent
Delivery guarantees are at-least-once, so a message will occasionally be delivered twice. A consumer that assumes exactly-once sends a duplicate email, or worse, a duplicate payment.
Acknowledging before the work is done
Acknowledging on receipt rather than on successful completion means a worker crash loses the message permanently. This is the defect that turns a queue into a silent data-loss mechanism.
Using a database table as a queue
Polling a table with SELECT FOR UPDATE works until throughput rises, then it becomes a lock-contention problem that slows down the same database serving user requests.
Messaging Capabilities
The patterns we implement, and the operational work that makes them trustworthy.
Background Job Processing
Report generation, exports, PDF and document creation, bulk imports and image or video processing moved out of the request path so a user never waits on them.
Third-Party API Buffering
Calls to payment gateways, e-way bill and GST services, SMS, email and WhatsApp providers queued with retry, so an unavailable provider delays one feature instead of failing a transaction.
Retry, Backoff & Dead-Letter
Exponential backoff on transient failures, a bounded retry policy, and a dead-letter queue with alerting so genuinely failed work is visible and replayable rather than lost.
Event-Driven Architecture
Publishing domain events so services react without direct coupling — an order dispatched, a payment settled, a batch completed — with topic exchanges routing to whoever cares.
Notification Fan-Out
One event producing email, SMS, push and in-app notifications through independent consumers, so a failing SMS provider does not block the email nobody else depends on.
Spike Absorption
Bursts — an exam cohort, a sale launch, a bulk upload — accepted immediately and processed at a sustainable rate, instead of overwhelming the database and failing everything.
Celery & Django Integration
Celery workers over RabbitMQ with scheduled tasks, task routing by priority, result handling and monitoring — the pattern behind most of our Django systems.
Queue Monitoring & Alerting
Queue depth, consumer lag, processing rate and dead-letter volume as monitored metrics with alerts, so a stalled consumer is detected by the system rather than by a customer.
Microservice Communication
Asynchronous inter-service messaging for systems moving away from a monolith, with contracts defined so a change on one side does not silently break the other.
Getting Asynchronous Processing Right
Decide what genuinely must happen before the response
The design question is not “what can we make asynchronous?” but “what must be complete and durable before we tell the user this succeeded?” Usually that is validation and persisting the core record — and very little else. Sending the confirmation email, generating the PDF, notifying the warehouse, calling the tax API and updating the analytics store can all happen afterwards.
Drawing that line correctly is most of the work. Put too much behind the queue and users see stale state and get confused. Put too little and you have kept the coupling you were trying to remove. We work through it per operation with the business, because the answer depends on what the user reasonably expects to be true the moment the screen says “saved”.
- Only validation and durable persistence stay synchronous
- The line drawn per operation, with the business, not by default
- User-visible state kept honest about what is still in progress
- Nothing user-facing depends on an external service responding in time
Assume every message will arrive twice
RabbitMQ guarantees at-least-once delivery. A network blip during acknowledgement, a worker restart mid-processing, or a redelivery after a timeout will all cause a message to be processed more than once. This is not an edge case to be surprised by; it is the contract.
So consumers are written to be idempotent. Every message carries an identifier, processing records that identifier, and a repeat is recognised and skipped. Where the operation genuinely cannot be repeated — sending money, generating a numbered document — idempotency is enforced at the database level with a unique constraint rather than by an application-level check that can race. It is the same discipline described on our fintech page, applied to messaging.
- Idempotent consumers as the default, not an optimisation
- Message identifiers recorded so redelivery is recognised
- Database-level uniqueness where an operation must not repeat
- Acknowledge on successful completion, never on receipt
Failure handling is the whole point
A queue that drops work it cannot process is worse than no queue, because the failure is invisible. The queue looks healthy, the dashboards look fine, and a customer discovers three weeks later that their document was never generated.
So every consumer has a bounded retry policy with exponential backoff for transient failures, and anything that exhausts it goes to a dead-letter queue that is monitored and alerted on. Dead-lettered messages retain enough context to be understood and replayed once the underlying cause is fixed. Queue depth and consumer lag are monitored metrics with thresholds, because a queue growing steadily is an early warning that something downstream is failing — usually well before users notice.
- Bounded retries with exponential backoff
- Dead-letter queues monitored, alerted and replayable
- Queue depth and consumer lag as first-class monitored metrics
- Failed work visible and recoverable rather than silently lost
When you do not need a message broker
Introducing a queue adds a moving part, an operational responsibility and a new class of bug. That is worth it above a threshold and not below it:
- If everything completes in milliseconds, skip it. A simple CRUD application with no external calls and no heavy processing gains complexity and nothing else.
- Django and Celery with Redis may be enough. For straightforward background jobs at modest volume, Redis is simpler to run than RabbitMQ. Move to RabbitMQ when you need routing, durability guarantees or real throughput.
- Kafka is a different tool for a different problem. If you need an event log that can be replayed from the beginning and retained for analytics, that is Kafka's territory, not RabbitMQ's.
- Do not use it to hide a slow database. Queuing work that is slow because of a missing index just moves the problem. Fix the query first.
What We Pair It With
Applications
Data
Platform
Edge
Operations
Frequently Asked Questions
It stops slow or unreliable work from blocking your users. Instead of a request waiting on a payment gateway, an e-way bill API, a PDF generator or a bulk report, the request does the minimum needed to be correct and queues the rest for workers to process. The user gets an immediate response, and if a downstream service is unavailable the work waits and completes when it recovers rather than failing the transaction.
Related Expertise
Docker & Kubernetes
Scaling workers on queue depth, and running brokers in production.
Python & Django
Celery over RabbitMQ — the pattern behind most of our backends.
Kong API Gateway
Handling inbound webhooks reliably at the edge.
Manufacturing ERP
Why a shop-floor save must never wait on a government API.
Fintech
Idempotency and reliable payment processing in depth.
Custom Software Development
How architecture work is scoped alongside features.
Users Waiting on Something They Shouldn't Be?
Slow exports, third-party APIs timing out, notifications holding up a save, imports that lock the system. Tell us where the waiting happens and we'll show you what moving it off the request path actually takes.
Serving startups, factories and enterprises across India, the US, UK, Australia & Europe.