In the previous article, we removed the polling tax.

Instead of asking the database every 15 seconds:

“Did anything change?”

we changed the architecture so that every relevant mutation could produce an event atomically through a Transactional Outbox.

The legacy ERP remained untouched.

The database captured the change.

The Outbox recorded the event.

But that created the next problem:

How do we move those events out of MySQL without going back to polling?

That is where the MySQL Binary Log becomes interesting.

In this article, we will build a lightweight Go-based streamer that connects to MySQL through the replication protocol, consumes BinLog events continuously, uses GTID-based positioning for recovery, and feeds a bounded in-memory pipeline without turning the database into another polling source.

The implementation described here was designed around a target workload of approximately 3,000 messages per second, with a measured memory footprint below 100 MB.

The Problem After the Outbox#

The previous architecture looked like this:

We had successfully created a reliable event boundary.

But the question mark remained.

How do we consume the Outbox?

The obvious answer is:

SELECT *
FROM outbox
WHERE status = 'pending'
ORDER BY created_at
LIMIT 1000;

Then run it again.

And again.

And again.

We would have simply recreated the architecture we had just removed.

Polling is still polling, even if the table is called outbox.

We needed a different mechanism.

The Binary Log Changes the Game#

MySQL already maintains a sequential history of changes through its Binary Log.

Instead of asking the database to repeatedly query the Outbox table, we can consume the database’s replication stream.

The architecture changes from:

to:

The database is no longer being interrogated for changes.

The consumer is following the existing replication stream.

MySQL’s replication protocol is specifically designed to transmit binary-log events to replicas. A Go implementation can use that same protocol to behave as a lightweight replication client rather than a traditional application polling the database. The go-mysql library explicitly provides a replication package for this use case and supports GTID-based synchronization.

Why GTIDs Matter#

There are two fundamentally different ways to remember where we were.

The first is the traditional file-and-position model:

binlog.000123:4567

It works.

But the position is tied to a particular binary-log file and offset.

If the source changes during a failover, the physical coordinates may no longer be the most useful identity for resuming the stream.

GTIDs solve a different problem.

A GTID identifies a transaction independently of the physical file position.

MySQL documents GTID-based replication specifically as a way to simplify replication management and failover by avoiding dependence on log-file and position coordinates.

Conceptually:

Instead of saying:

“Continue reading from byte 4,567 in binlog.000123.”

we can say:

“Continue from this transaction history.”

That distinction becomes extremely useful when the topology changes.

The Streamer Architecture#

The resulting architecture looks like this:

There are four important components:

BinLog Syncer

Maintains the connection to MySQL and consumes replication events.

Event Filter

Ignores events that are irrelevant to the CDC pipeline and keeps only the changes we care about.

Bounded Buffer

Prevents downstream backpressure from causing unbounded memory growth.

Checkpoint Store

Persists the position state required to resume processing after a restart.

The key architectural constraint is:

The streamer must never allow downstream slowness to turn into unbounded memory consumption.

Why a Bounded Buffer?#

A common mistake when building streaming systems is to optimize for throughput first and memory second.

That usually ends with something like:

events = append(events, event)

forever.

It works until it doesn’t.

If the consumer is slower than the producer, the queue grows.

Eventually:

A bounded queue introduces a hard memory limit:

The queue can become full.

That is not a bug.

That is a signal.

Once the buffer reaches its limit, the system must apply a deliberate backpressure policy rather than silently consuming all available memory.

For a production system, that policy might mean blocking, batching, dropping only explicitly non-critical data, or transferring pressure downstream.

For transactional CDC, the correct behavior is generally to preserve the event and control throughput rather than silently lose data.

Connecting to MySQL#

The Go ecosystem has a useful building block for this problem:

github.com/go-mysql-org/go-mysql/replication

The library exposes a BinlogSyncer, allows you to specify a unique server ID, and supports both file/position and GTID-based synchronization.

A minimal configuration looks like this:

package main

import (
	"log"

	"github.com/go-mysql-org/go-mysql/replication"
)

func main() {
	cfg := replication.BinlogSyncerConfig{
		ServerID: 1001,
		Flavor:   "mysql",
		Host:     "127.0.0.1",
		Port:     3306,
		User:     "cdc_user",
		Password: "cdc_password",
	}

	syncer := replication.NewBinlogSyncer(cfg)

	log.Println("BinLog syncer initialized:", syncer)
}

The ServerID is important because the client is participating in the replication protocol as a replication consumer. The go-mysql documentation explicitly requires a unique server ID for the syncer.

In production, of course, credentials should come from a secret manager or environment rather than being embedded in source code.

The Replication User#

The replication client also needs an appropriate MySQL account.

MySQL documents REPLICATION SLAVE as the privilege used by a replication account to request updates from a replication source. Although MySQL has renamed much of its replication terminology from “slave” to “replica”, the privilege name remains REPLICATION SLAVE in current MySQL documentation.

A dedicated account is preferable to using an administrative account:

CREATE USER 'cdc_user'@'10.%'
IDENTIFIED BY 'strong-secret';

GRANT REPLICATION SLAVE ON *.*
TO 'cdc_user'@'10.%';

The important principle is least privilege.

This process does not need to update application tables.

It needs to consume replication data.

Starting From a GTID Set#

The crucial part of the implementation is the starting point.

The go-mysql replication package supports:

StartSyncGTID(...)

which allows the streamer to request binary-log events starting from a GTID set rather than a raw file/position coordinate.

A simplified implementation looks like this:

package main

import (
	"log"

	"github.com/go-mysql-org/go-mysql/mysql"
	"github.com/go-mysql-org/go-mysql/replication"
)

func startFromGTID(syncer *replication.BinlogSyncer, gtidString string) {
	gtidSet, err := mysql.ParseGTIDSet(
		mysql.MySQLFlavor,
		gtidString,
	)
	if err != nil {
		log.Fatalf("parse GTID set: %v", err)
	}

	streamer, err := syncer.StartSyncGTID(gtidSet)
	if err != nil {
		log.Fatalf("start GTID sync: %v", err)
	}

	_ = streamer
}

This is conceptually the important transition:

MySQL’s GTID lifecycle associates GTIDs with committed transactions in the binary log, and the gtid_executed set represents the transactions executed on the server.

Reading the Stream#

Once the stream is established, the consumer continuously receives events.

A simplified loop:

for {
	ev, err := streamer.GetEvent(context.Background())
	if err != nil {
		log.Printf("stream error: %v", err)
		continue
	}

	handleEvent(ev)
}

The important architectural property is that GetEvent is not repeatedly issuing SQL queries against the Outbox.

The process is consuming the replication stream.

That means the data path is now:

The difference from polling is fundamental.

There is no: SELECT … loop.

Filtering the Outbox Events#

The BinLog contains much more than our Outbox changes.

The database may be generating events for:

  • orders,

  • customers,

  • payments,

  • logs,

  • inventory,

  • internal tables,

  • and many other transactions.

The streamer therefore needs a filtering layer.

Conceptually:

The actual implementation should inspect the parsed event type and table metadata rather than pretending every BinLog event is an Outbox record.

For row-based logging, MySQL writes row-change events to the binary log, which makes row-level CDC a natural fit for this style of event extraction. MySQL 8.0 uses row-based logging by default, and MySQL recommends row-based logging for new replication setups.

A simplified processing structure might look like:

switch event := ev.Event.(type) {
case *replication.RowsEvent:
	// Inspect table and row mutations.
	// Keep only the Outbox changes.
	_ = event
}

The production implementation then needs to handle the complete set of event types required by the chosen MySQL configuration.

The Checkpoint Problem#

Streaming data is easy.

Restarting safely is much harder.

Imagine the streamer processes:

Then the process crashes.

Where should it restart?

If the checkpoint was written before D was successfully processed, we may process D again.

If the checkpoint was written after the process claimed D was handled but before the downstream operation actually succeeded, we may lose it.

This is where checkpoint semantics become more important than raw throughput.

The pipeline should look like:

The ordering matters.

Process first. Persist the checkpoint after successful processing.

Otherwise the checkpoint can move ahead of the actual business operation.

That is how seemingly reliable CDC systems accidentally create data loss.

GTID-Based Recovery#

With GTIDs, the restart problem becomes easier to reason about.

The streamer can persist the GTID state representing the transaction history it has successfully consumed and then request the stream from that state after restarting.

MySQL’s GTID mechanism is explicitly designed to identify transactions independently of physical binary-log coordinates and to support automatic positioning during replication and failover.

Conceptually:

The checkpoint is therefore not simply:

binlog.000123:4567

It represents transactional progress.

Why BoltDB?#

For a lightweight standalone binary, we wanted checkpoint storage to have very little operational overhead.

A full external database would have introduced another dependency:

That is unnecessary for a small amount of local state.

A local embedded store such as BoltDB can provide durable checkpoint storage without introducing another network service.

The important principle is not BoltDB itself.

It is:

The checkpoint store should be simpler than the system it is checkpointing.

The state is small.

The operational footprint should therefore remain small.

A More Complete Streamer Skeleton#

Putting the pieces together:

package main

import (
	"context"
	"log"
	"time"

	"github.com/go-mysql-org/go-mysql/mysql"
	"github.com/go-mysql-org/go-mysql/replication"
)

type BinlogStreamer struct {
	syncer *replication.BinlogSyncer
}

func NewStreamer() *BinlogStreamer {
	cfg := replication.BinlogSyncerConfig{
		ServerID: 1001,
		Flavor:   "mysql",
		Host:     "127.0.0.1",
		Port:     3306,
		User:     "cdc_user",
		Password: "cdc_password",
	}

	return &BinlogStreamer{
		syncer: replication.NewBinlogSyncer(cfg),
	}
}

func (s *BinlogStreamer) Start(gtidString string) error {
	gtidSet, err := mysql.ParseGTIDSet(
		mysql.MySQLFlavor,
		gtidString,
	)
	if err != nil {
		return err
	}

	streamer, err := s.syncer.StartSyncGTID(gtidSet)
	if err != nil {
		return err
	}

	for {
		ctx, cancel := context.WithTimeout(
			context.Background(),
			2*time.Second,
		)

		ev, err := streamer.GetEvent(ctx)
		cancel()

		if err != nil {
			continue
		}

		if !isOutboxEvent(ev) {
			continue
		}

		if err := processEvent(ev); err != nil {
			log.Printf("processing failed: %v", err)
			continue
		}

		// Persist the checkpoint only after
		// successful event processing.
		persistCheckpoint(ev)
	}
}

func isOutboxEvent(
	ev *replication.BinlogEvent,
) bool {
	// Production implementation:
	// inspect event type, table metadata,
	// schema and row changes.
	return true
}

func processEvent(
	ev *replication.BinlogEvent,
) error {
	// Push to bounded buffer / downstream pipeline.
	_ = ev
	return nil
}

func persistCheckpoint(
	ev *replication.BinlogEvent,
) {
	// Persist GTID / transactional progress.
	_ = ev
}

This is intentionally a skeleton.

A production CDC engine needs considerably more:

  • reconnect handling,

  • retry policies,

  • bounded queues,

  • graceful shutdown,

  • durable checkpoint semantics,

  • schema-change handling,

  • event batching,

  • metrics,

  • tracing,

  • dead-letter handling,

  • backpressure,

  • and exactly what constitutes successful downstream processing.

The interesting part is the architecture.

The implementation details grow from there.

The Memory Constraint#

Now we get to the engineering constraint that drove the implementation.

The streamer should be small enough to run as a dedicated sidecar or service without requiring a large runtime environment.

The pipeline therefore has to avoid:

Instead:

The buffer defines the maximum amount of work waiting in memory.

The worker pool defines the maximum concurrency.

The checkpoint defines durable progress.

Those three controls prevent the stream from becoming an uncontrolled memory accumulator.

What We Measured#

In the implementation described in this series, the target was approximately:

Peak throughput:        ~3,000 messages/sec
Processed volume:      ~350,000 records
Memory usage:           <100 MB
Target CPU:             ~50 mCPU

These numbers describe the benchmark environment and implementation, not a universal performance guarantee for Go, MySQL, or go-mysql.

The important result was that the streamer could maintain the required throughput while keeping its operational footprint small enough to run alongside the existing infrastructure.

That was the real objective.

Not winning a benchmark.

Building a CDC component that could be operated cheaply.

The Resulting Architecture#

At this point, the complete pipeline looks like this:

The architectural evolution is now complete:

And none of it required rewriting the ERP.

The Important Trade-Off#

There is no free abstraction.

Moving from polling to BinLog CDC introduces new complexity.

You now have to understand:

  • replication privileges,

  • binary-log configuration,

  • GTIDs,

  • event parsing,

  • checkpointing,

  • failover,

  • backpressure,

  • duplicate delivery,

  • schema evolution,

  • and operational monitoring.

That complexity is real.

But it is controlled complexity.

Instead of forcing every downstream service to repeatedly query a legacy transactional database, we create a single integration boundary that understands the database’s change stream.

That is a much more scalable architectural boundary.

Polling vs. CDC#

The transformation can now be summarized in one diagram:

The difference is not simply performance.

It is architectural responsibility.

In the polling model, the consumer repeatedly interrogates the database.

In the CDC model, the database’s existing change stream becomes the source of integration events.

The Golden Rule#

The lesson from both articles can now be summarized in one sentence:

Do not make the transactional database repeatedly prove that a change happened. Capture the change once, then stream the fact asynchronously.

In the first article, we captured the mutation atomically using the Transactional Outbox.

In this article, we moved the event out of MySQL using the BinLog replication stream and GTID-based positioning.

The result is:

The critical transaction remains local.

The network remains asynchronous.

The legacy ERP remains untouched.

And the event pipeline can evolve independently from the system that originally produced the data.

What Comes Next?#

We have now solved two different problems.

Post 1: How do we detect changes without polling?

→ Transactional Outbox + MySQL Trigger

Post 2: How do we move those changes without polling again?

→ MySQL BinLog + GTID + Go Streamer

But there is still one major problem.

We have a stream.

We have events.

We have a buffer.

Sooner or later, we need to answer a much harder question:

What happens when the consumer is down, the broker is unavailable, the network fails, or the same event is delivered twice?

That is where idempotency, retries, ordering, backpressure, and failure recovery become the real architecture.

And that is the subject of the next post.

Because a streaming system is not production-ready when it can process 3,000 messages per second.

It is production-ready when it can survive failure without losing the truth.