In the first article, we removed the polling tax.
Instead of asking the database every 15 seconds:
“Did anything change?”
we captured relevant mutations atomically using a Transactional Outbox.
The legacy ERP remained untouched.
The database captured the mutation.
The Outbox recorded the event.
Then we faced the next problem.
How do we move those events out of MySQL without going back to polling?
In the second article, we solved that problem by following the MySQL Binary Log through the replication protocol.
We introduced:
GTID-based recovery
a lightweight Go BinLog streamer
event filtering
bounded buffers
checkpointing
controlled backpressure
But we now have another problem.
The streamer is producing events that still look too much like the database that created them.
And that is dangerous.
Because a database schema is an implementation detail.
A public event contract should not be.
This article is about the next architectural boundary:
Turning database changes into stable, transport-independent events and introducing a durable message broker between producers and consumers.
The goal is to reach a point where the ERP can continue evolving internally while downstream systems consume a stable event contract.
The Problem After the Streamer#
At the end of the previous article, we had:
The streamer can now consume the database change stream.
But what should it publish?
One tempting approach is to forward the database mutation almost exactly as it arrived.
For example:
{
"table": "fktscxx001",
"operation": "UPDATE",
"before": {
"stock": 10
},
"after": {
"stock": 9
}
}
It works.
Until someone renames a column.
Or changes the table.
Or moves the data to another database.
Or splits one table into three.
Or replaces MySQL entirely.
Then every consumer becomes coupled to the database schema.
That is not event-driven architecture.
That is database schema propagation over a network.
The Contract Boundary#
The database schema belongs to the database.
The event schema belongs to the event platform.
Those are two different things.
We want this:
The Go streamer therefore needs to do more than decode BinLog events.
It needs to translate infrastructure-level changes into domain-level events.
For example:
The consumer should not care whether StockUpdated came from:
MySQL,
PostgreSQL,
an ERP,
a Kafka topic,
a batch process,
or an API.
It should only care that a contractually defined event has occurred.
Why CloudEvents?#
This is where CloudEvents becomes useful.
CloudEvents is a specification designed to provide a common way of describing event data through standardized metadata, making event producers, routers and consumers less dependent on the specific format used by each event source. CloudEvents is now a CNCF Graduated project and provides SDKs for multiple languages, including Go.
The important idea is not “another JSON format”.
It is standardized event metadata.
A CloudEvent identifies things such as:
what happened,
where it came from,
which event instance this is,
which CloudEvents specification version is being used,
and optionally when it happened and other extension attributes.
At the core, the event starts to look like this:
{
"specversion": "1.0",
"type": "com.example.erp.stock.updated",
"source": "/erp/inventory",
"id": "8b7b9a80-2f2e-4a77-8f5e-0a81b8a0d4d1",
"time": "2026-09-10T12:30:45Z",
"datacontenttype": "application/json",
"data": {
"article_id": "12345",
"previous_stock": 10,
"current_stock": 9,
"warehouse_id": "MAD-01"
}
}Now the consumer sees an event contract rather than a MySQL row mutation.
Database Schema vs. Event Schema#
This distinction is one of the most important ideas in the whole series.
Consider this database table:
A consumer should not have to know any of that.
Instead it receives:
The ERP can change its internal representation while the external contract remains stable.
This is what allows the event layer to become an actual architectural boundary.
The First Failure: Direct HTTP#
Now imagine we skip the broker.
The Go streamer receives an event and calls the consumers directly:
It looks simple.
It is also where the temporal coupling problem reappears.
What happens when WMS is healthy but Analytics is down?
What happens when the Web platform takes 5 seconds to respond?
What happens when all three are unavailable?
The streamer now has to decide:
And suddenly our small CDC process has become a message broker.
Except it does not have the durability guarantees, operational tooling or consumption semantics of a dedicated messaging system.
Backpressure Becomes the Real Problem#
Imagine the BinLog produces: 3,000 events/sec
while the downstream consumer can process only: 500 events/sec
The difference is: +2,500 events/sec
That backlog has to exist somewhere.
Without a broker:
The queue grows until memory becomes the bottleneck.
That is exactly what we want to avoid.
The Broker as a Temporal Boundary#
The message broker changes the equation.
Now the architecture becomes:
The broker introduces temporal decoupling.
The producer and consumer no longer have to be available at exactly the same time.
The producer can publish.
The broker retains the message according to its configured durability and retention semantics.
The consumer processes it when it is able to do so.
That is a fundamentally different relationship.
RabbitMQ or NATS JetStream?#
At this point, we have several reasonable choices.
For a lightweight architecture, two interesting candidates are:
RabbitMQ
and
NATS JetStream
They are not identical products and they should not be treated as interchangeable in every workload.
The architectural requirement is what matters:
We need a broker that can provide durable asynchronous delivery and explicit acknowledgement semantics appropriate to the workload.
RabbitMQ supports publisher confirms on the producer side and consumer acknowledgements on the consumer side. RabbitMQ documents these mechanisms as the foundation for at-least-once delivery, while also warning that retransmission after failures can produce duplicates.
That has an immediate architectural consequence:
At-least-once delivery means consumers must be prepared for duplicates.
We will return to that in the next article.
CloudEvents Before the Broker#
The critical sequence is:
We do not want:
We want:
That distinction gives us an explicit contract boundary.
Building a CloudEvent in Go#
The CloudEvents project maintains an official Go SDK for creating, serializing and transmitting CloudEvents. The current SDK supports CloudEvents 1.0 and its current v2 module requires Go 1.24+.
Instead of maintaining our own CloudEvent structure, the production implementation can use the official SDK.
A simplified example:
package main
import (
"time"
cloudevents "github.com/cloudevents/sdk-go/v2"
)
type StockUpdated struct {
ArticleID string `json:"article_id"`
WarehouseID string `json:"warehouse_id"`
PreviousStock int `json:"previous_stock"`
CurrentStock int `json:"current_stock"`
}
func BuildStockUpdated(
id string,
data StockUpdated,
) cloudevents.Event {
event := cloudevents.NewEvent()
event.SetID(id)
event.SetSource("/erp/inventory")
event.SetType("com.example.erp.stock.updated")
event.SetTime(time.Now().UTC())
_ = event.SetData(
cloudevents.ApplicationJSON,
data,
)
return event
}This is better than manually recreating the CloudEvents envelope because the official SDK already provides the model and serialization support.
The Event ID Is Not Just Metadata#
Remember the id field?
It becomes extremely important.
For example:
{
"specversion": "1.0",
"type": "com.example.erp.stock.updated",
"source": "/erp/inventory",
"id": "outbox-8fbc8f5d",
"data": {
"article_id": "12345",
"current_stock": 9
}
}That identifier gives downstream systems something they can use for:
deduplication,
observability,
tracing,
auditing,
retries,
idempotency.
And that leads directly into the next problem.
Publishing to RabbitMQ#
A simplified RabbitMQ publishing flow looks like:
The publisher confirm tells the producer that RabbitMQ has taken responsibility according to the broker’s confirmation semantics. Consumer acknowledgements, in turn, let the consumer indicate when it has successfully handled a delivery.
But there is an important caveat.
A confirmation does not magically make the entire distributed workflow exactly-once.
Failures can produce duplicates.
For example:
RabbitMQ explicitly documents this possibility when publishers retransmit messages after failures with outstanding confirmations.
This is why at-least-once delivery must be combined with idempotent processing.
RabbitMQ Consumer Flow#
The consumer side should also use explicit acknowledgements.
Conceptually:
The important ordering is:
Not:
RabbitMQ’s documentation specifically recommends acknowledging deliveries only after the consumer has completed the required work; manual acknowledgements also allow consumers to limit the number of in-flight messages with prefetch.
NATS JetStream#
The same architecture can be implemented with NATS JetStream.
Conceptually:
The important point is the same:
the consumer no longer has to be available at the exact moment the producer publishes the event.
The broker becomes the temporal boundary.
That is the architectural property we need.
The Producer Must Not Wait for the Consumer#
This sounds obvious.
It is surprisingly easy to violate.
The following architecture is dangerous:
because the producer now owns downstream availability.
The preferable model is:
The producer publishes.
The consumers consume.
Those are separate responsibilities.
The Complete Architecture#
At this point, the complete pipeline looks like this:
Now we have a clean chain of responsibility:
Every layer has a distinct job.
What We Actually Gained#
The result is not simply “we added RabbitMQ.”
We gained four important properties.
1. Schema Decoupling#
Consumers no longer depend directly on MySQL table structures.
2. Temporal Decoupling#
The producer and consumer no longer need to be available at the same time.
3. Transport Independence#
The event contract can move through different transports without changing the domain meaning of the event.
4. Controlled Backpressure#
The broker provides a durable buffering boundary instead of forcing the CDC process to retain an unlimited backlog in memory.
These properties are more important than the specific broker technology.
The New Failure Modes#
Of course, every abstraction introduces new failure modes.
We have now moved from:
to:
That means we now have to reason about:
duplicate delivery,
consumer retries,
ordering,
poison messages,
dead-letter queues,
broker outages,
network partitions,
consumer crashes,
checkpoint advancement,
and idempotency.
And this is exactly where distributed systems stop being a diagramming exercise.
Because the real question is no longer:
“Can we move the message?”
The real question becomes:
“Can we move the message reliably when everything around it is failing?”
The Golden Rule#
The lesson from the first three articles can now be summarized as:
Keep database transactions local, convert database changes into stable domain events, and introduce a durable asynchronous boundary before the network of consumers.
The full evolution looks like this:
The database remains responsible for transactional truth.
The CDC layer is responsible for transporting change.
The event contract is responsible for semantic stability.
The broker is responsible for temporal decoupling.
The consumer is responsible for business processing.
That separation is what makes the architecture manageable.
What Comes Next?#
We now have:
Post 1: Transactional Outbox + MySQL Trigger
→ Capture changes atomically.
Post 2: MySQL BinLog + GTID + Go Streamer
→ Consume changes without polling.
Post 3: CloudEvents + Durable Messaging
→ Standardize and distribute those changes without tightly coupling producer and consumer availability.
But one problem remains.
Actually, it is several problems hiding inside one:
What happens when the same event arrives twice?
What happens when a consumer crashes after processing the payment but before acknowledging the message?
What happens when a network failure makes the publisher unsure whether RabbitMQ accepted the event?
What happens when events arrive out of order?
What happens when one event repeatedly fails processing?
Those are not edge cases.
They are normal conditions in distributed systems.
The next article will focus on idempotency, retries, ordering, dead-letter queues, and failure recovery.
Because at-least-once delivery is not the end of the architecture.
It is where the interesting part begins.