Data Exports Integration

Change Data Capture (CDC)

Overview

Change Data Capture (CDC) files represent row-level mutations after the initial full-load snapshot.

OP Column Semantics

The OP column describes the operation type:

  • I = Insert
  • U = Update
  • D = Delete

Reload / Replace Behavior

In some scenarios, Fortress replaces a row instead of updating it in place. In CDC, this can appear as:

D id=123
I id=123

Treat this sequence as a replace/upsert of the same logical record. The id remains the stable identifier.

This also explains why some D operations may not include a deletedAt value.

Processing Rules

To process CDC files correctly:

  1. Process files in filename order using YYYYMMDD-HHMMSSfff.csv.
  2. Within each file, process rows in the order provided.
  3. If multiple operations exist for the same id in a batch, apply precedence:
D -> U -> I

This precedence ensures reload-style sequences resolve correctly.

  1. Start with the most recent full-load file for a topic.
  2. Apply CDC files in strict chronological filename order.
  3. Apply rows in-order within each file.
  4. Use operation precedence (D -> U -> I) when multiple ops for the same id exist in one batch.
  5. Persist checkpoints by last processed filename for resumable ingestion.