Change Data Capture (CDC)

Overview

Change Data Capture (CDC) files represent row-level mutations after the initial full-load snapshot.

OP Column Semantics

The OP column describes the operation type:

• I = Insert
• U = Update
• D = Delete

Reload / Replace Behavior

In some scenarios, Fortress replaces a row instead of updating it in place. In CDC, this can appear as:

D id=123
I id=123

Treat this sequence as a replace/upsert of the same logical record. The id remains the stable identifier.

This also explains why some D operations may not include a deletedAt value.

Processing Rules

To process CDC files correctly:

1. Process files in filename order using YYYYMMDD-HHMMSSfff.csv.
2. Within each file, process rows in the order provided.
3. If multiple operations exist for the same id in a batch, apply precedence:

D -> U -> I

This precedence ensures reload-style sequences resolve correctly.

Recommended Ingestion Pattern

1. Start with the most recent full-load file for a topic.
2. Apply CDC files in strict chronological filename order.
3. Apply rows in-order within each file.
4. Use operation precedence (D -> U -> I) when multiple ops for the same id exist in one batch.
5. Persist checkpoints by last processed filename for resumable ingestion.