Jordan Goodman

How Debezium Works

Used by Airbyte, Apache Flink, and Google BigQuery as a core dependency in production, Debezium is an open source distributed platform for change data capture.

Debezium captures committed database changes and turns them into structured events.

It is commonly used to move inserts, updates, and deletes from operational databases into Kafka, data warehouses, search systems, caches, and other downstream platforms.

The basic flow is:

The output looks simple, but reliable change data capture requires more than reading a database log.

Debezium must also handle initial snapshots, transaction order, schema changes, restarts, duplicate delivery, source-log retention, and downstream slowdowns.

The database is the source of truth

Debezium captures changes after the source database accepts them.

It does not capture a button click, an API request, or an ORM method call. It captures the committed result of those operations.

Different databases expose committed changes in different ways:

Debezium provides a connector for each supported database. Each connector understands the source-specific log format and converts it into a more consistent event structure.

Every connector must solve the same general problems:

How Debezium is deployed

Debezium is most commonly run through Kafka Connect.

Kafka Connect manages:

Debezium can also run through Debezium Server or Debezium Engine.

Debezium Server reads from a source connector and sends records directly to a supported destination without requiring a user-managed Kafka Connect deployment.

Debezium Engine embeds a connector inside a Java application. This gives the application more control, but it also makes the application responsible for event handling, offset storage, acknowledgement, and recovery.

The connector is only one part of the delivery system. Reliability also depends on where offsets are stored and whether the destination has safely accepted an event.

The initial snapshot

A database transaction log contains changes. It does not necessarily contain a complete copy of every current row.

When Debezium starts against an existing database, it normally needs to create a baseline before it can process new changes. This baseline is the initial snapshot.

The simplified process is:

The important requirement is that no committed change can be lost between the snapshot and the start of streaming.

Snapshot records typically use the operation code r, meaning that the row was read from the existing table state rather than created by a new insert.

For large tables, Debezium can use incremental snapshots. Instead of copying an entire table in one operation, it reads the table in smaller key ranges while normal CDC continues.

That creates a coordination problem: a row might be updated while its range is being copied. Debezium compares the snapshot rows with live changes during that window and avoids emitting an older snapshot row after a newer update.

Reading the transaction log

After the snapshot, Debezium continuously reads the source database’s change mechanism.

PostgreSQL

PostgreSQL writes changes to the write-ahead log, usually called the WAL.

The WAL exists primarily for durability and crash recovery. PostgreSQL records a change in the WAL before the related table page is written to permanent storage. If the server crashes, PostgreSQL can replay WAL records and recover committed changes.

Debezium does not usually parse raw physical WAL records directly. PostgreSQL logical replication converts WAL activity into logical messages describing table changes.

The main PostgreSQL objects involved are:

The replication slot is operationally important. If Debezium stops reading, PostgreSQL may continue retaining WAL for the slot. If the connector remains offline long enough, retained WAL can consume the database server’s disk.

MySQL

MySQL writes row changes to the binary log, or binlog.

The connector reads events for:

A MySQL source position may include the binlog filename, byte offset, row position, and GTID information.

The connector also needs to know the table definition that applied when each event was written. A column position in an older binlog event may refer to a different schema than the current table definition.

The event pipeline

The source-specific connector reads and decodes database changes, but it does not send each record directly to the destination.

The internal flow is roughly:

The queue separates database reading from destination writing.

This matters because source databases and downstream systems run at different speeds. Debezium may be able to read changes faster than Kafka or another sink can accept them.

The queue has a fixed capacity. When it becomes full, Debezium slows or pauses reading from the source.

This is called backpressure.

Backpressure protects the Debezium process from unlimited memory use, but it does not eliminate the pressure. It moves the problem toward source-log retention and replication lag.

Event structure

A Debezium row event usually has a key and a value.

The key normally comes from the source table’s primary key:

{

“order_id”: 1001

}

The value contains the row before and after the change, along with source metadata:

{

“before”: {

“order_id”: 1001,

“status”: “pending”

},

“after”: {

“order_id”: 1001,

“status”: “shipped”

},

“source”: {

“connector”: “postgresql”,

“database”: “sales”,

“schema”: “public”,

“table”: “orders”,

“transaction_id”: 81293,

“lsn”: 238492936

},

“op”: “u”

}

Common operation codes are:

The source database controls how complete the before image is.

For PostgreSQL, replica identity determines which old values are available for updates and deletes. For MySQL, binlog row-image settings affect the amount of old and new row data included.

Kafka deployments may also produce a tombstone after a delete. A tombstone has the same key as the deleted record and a null value. Kafka log compaction can use it to remove older records for that key.

Offsets

An offset is Debezium’s saved source position.

It answers the question: Where should this connector resume after a restart?

The exact value depends on the database.

PostgreSQL:

MySQL:

A reliable delivery sequence is:

If Debezium publishes an event and crashes before saving the offset, the event may be published again after restart.

This produces at-least-once delivery.

If Debezium saved the offset before the event became durable, a crash could cause the event to be skipped permanently. Debezium is designed to prefer duplicate delivery over silent loss.

Downstream systems should therefore assume that duplicate events are possible.

Common ways to handle duplicates include:

Kafka Connect can provide stronger source-record and offset coordination in supported exactly-once configurations. That guarantee ends at the Kafka boundary. A downstream database still needs its own transactional or idempotent loading logic.

Schema history

An offset tells Debezium where to resume. It does not always tell Debezium how the table was structured at that position.

Assume a table originally contains:

If the connector restarts from a position before the schema change, it must interpret older records using the old two-column structure. Reading only the current three-column schema could decode older data incorrectly.

Debezium maintains schema information as it processes changes. Some connectors also store an internal schema history containing table definitions and schema changes associated with source-log positions.

During restart, Debezium can rebuild the schema that was valid at the saved offset before it resumes reading.

Schema history is connector state, not normal business data. Losing it can make safe recovery impossible even when the source offset is still available.

PostgreSQL logical replication sends relation metadata through the replication stream, so its behavior differs from MySQL and several other connectors. The general requirement remains the same: every log event must be decoded using the correct schema version.

Transactions

A source transaction may update several rows across several tables.

Debezium preserves the source order of those changes and can attach transaction metadata such as:

This allows a consumer to identify which events belonged to the same source transaction.

It does not automatically make the destination transactionally atomic. A sink must explicitly buffer and commit a group of events together if it needs the destination to preserve the same all-or-nothing boundary.

Long-running or very large transactions can increase memory use, temporary storage, and replication latency because the connector may need to retain transaction context until the source commits or rolls back.

Heartbeats and signals

A database may be active while none of the selected tables are changing. In that case, the connector still needs a way to show that it is alive and advance its saved position.

Heartbeat events periodically record the current source position.

They are useful for:

Signals are control messages sent to Debezium. They can request actions such as starting an incremental snapshot without replacing the entire connector configuration.

These features are important in production because CDC connectors need operational controls, not only data extraction logic.

Transformations and serialization

Debezium creates a full change-event envelope. Kafka Connect or another runtime can transform that event before writing it to the destination.

A common transformation extracts only the after state:

{

“order_id”: 1001,

“status”: “shipped”

}

This format is easier for many consumers, but it can remove useful information:

The full Debezium envelope is more useful for auditing, debugging, replay, and transaction reconstruction.

After transformations, the runtime serializes events as JSON, Avro, Protobuf, or another configured format.

Failure Behavior

Debezium is designed to recover from interruptions, but recovery depends on the relationship between the last event delivered, the last source offset saved, and the source log still available.

Event Published Before the Offset Is Saved

An event may reach the destination just before Debezium crashes, but the corresponding source offset may not yet be persisted.

When the connector restarts, it resumes from the older saved offset and emits the event again.

Consequence: Duplicate delivery is possible, so downstream systems should use idempotent writes, primary-key merges, or another deduplication method.

Saved Source Position Is No Longer Available

Debezium resumes from a stored log position. If the source database deletes or recycles the required transaction-log segment before the connector restarts, that position can no longer be read.

Consequence: The connector cannot continue from its saved offset and may require a new snapshot or manual state recovery.

PostgreSQL Replication Slot Falls Behind

A PostgreSQL replication slot prevents the database from deleting WAL that Debezium has not yet confirmed.

If Debezium is offline or processing too slowly, PostgreSQL continues retaining the required WAL.

Consequence: WAL usage grows on the source server and can eventually consume all available disk space.

MySQL Binlog Expires

MySQL retains binary logs for a configured period. If the binlog file or GTID range containing Debezium’s saved position is removed, the connector loses its restart point.

Consequence: Streaming cannot resume from the saved position, and a new snapshot may be required.

Schema History Is Missing

The source offset tells Debezium where to resume, but it does not always provide enough information to interpret the table structure at that point in time.

If the required schema history is missing or corrupted, Debezium may be unable to decode older log records safely.

Consequence: Connector startup may fail rather than risk producing incorrectly mapped data.

Destination Processing Slows Down

If the destination processes events more slowly than Debezium reads them, the connector’s internal queue begins to fill.

Once the queue reaches capacity, Debezium slows or pauses source-log consumption.

Consequence: Replication lag increases, and the source database must retain transaction-log data for longer.

Database Failover

After a database failover, Debezium must reconnect to a server that exposes compatible transaction history and source metadata.

For PostgreSQL, this includes WAL continuity, publications, replication slots, and database identity. For MySQL, GTIDs generally make failover easier than server-specific binlog filenames and offsets.

Consequence: If the replacement server cannot provide the expected log history or identifiers, the connector may be unable to resume without state repair or a new snapshot.

Monitoring

A production Debezium deployment should monitor both the connector and the source database.

Important connector metrics include:

Important source metrics include:

A connector process can be running while no longer making useful progress. The important questions are:

The practical model

Debezium performs six core jobs:

The database-log reader is only one part of the system. The harder work is coordinating snapshots, offsets, schemas, queues, acknowledgements, source retention, and restarts without losing committed changes.