Default directory structure for CDC files on Amazon S3, Google Cloud Storage, Microsoft Azure Data Lake Storage Gen2, and Microsoft Fabric OneLake targets
Database ingestion and replication jobs create directories on Amazon S3, Google Cloud Storage, Microsoft Azure Data Lake Storage Gen2, and Microsoft Fabric OneLake targets to store information about change data processing.
The following directory structure is created by default on the targets:
The following table describes the directories in the default structure:
Folder
Description
connection_folder
Contains the Database Ingestion and Replication objects. This folder is specified in the Folder Path field of the Amazon S3 connection properties, in the Directory Path field of the Microsoft Azure Data Lake Storage Gen2 connection properties, or in the Lakehouse Path field of the Microsoft Fabric OneLake connection properties.
Note:
This folder is not created for Google Cloud Storage targets.
job_folder
Contains job output files. This folder is specified in the Directory field on the Target page of the database ingestion and replication task wizard.
cycle/completed
Contains a subfolder for each completed CDC cycle. Each completed cycle subfolder contains a completed cycle file.
cycle/contents
Contains a subfolder for each CDC cycle. Each cycle subfolder contains a cycle contents file.
data
Contains output data files and schema files for each table.
data/table_name/schema/V1
Contains a schema file.
Note:
Database Ingestion and Replication does not save a schema file in this folder if the output files use the Parquet format.
data/table_name/data
Contains a subfolder for each CDC cycle that produces output data files.
Cycle directories
Database Ingestion and Replication uses the following pattern to name cycle directories:
[dt=]yyyy-mm-dd-hh-mm-ss
The "dt=" prefix is added to cycle folder names if you select the Add Directory Tags check box on the Target page of the database ingestion and replication task wizard.
Cycle contents files
Cycle contents files are located in cycle/contents/cycle_folder subdirectories. Cycle contents files contain a record for each table that has had a DML event during the cycle. If no DML operations occurred on a table in the cycle, the table does not appear in the cycle contents file.
Database Ingestion and Replication uses the following pattern to name cycle content files:
Cycle-contents-timestamp.csv
A cycle contents csv file contains the following information:
•Table name
•Cycle name
•Path to the cycle folder for the table
•Start sequence for the table
•End sequence for the table
•Number of Insert operations
•Number of Update operations
•Number of Delete operations
•For combined load jobs only: Number of Truncate operations
•For combined load jobs only: Number of Insert operations encountered during the initial load phase
•For combined load jobs only: Number of Delete operations encountered during the initial load phase
•Schema version
•Path to the schema file for the schema version
Note:
If the output data files use the Parquet format, Database Ingestion and Replication does not save a schema file at the path that is specified in the cycle contents file. Instead, you can use the schema file in the folder that is specified in the Avro Schema Directory field on the Target page of the database ingestion and replication task wizard.
Completed cycle files
Completed cycle files are located in cycle/completed/completed_cycle_folder subdirectories. A database ingestion and replication job creates a cycle file in this subdirectory after a cycle completes. If this file is not present, the cycle has not completed yet.
Database Ingestion and Replication uses the following pattern to name completed cycle files:
Cycle-timestamp.csv
A completed cycle csv file contains the following information:
•Cycle name
•Cycle start time
•Cycle end time
•Current sequence number at the time the cycle ended
•Path to the cycle contents file
•Reason for the end of cycle
Valid reason values are:
- NORMAL_COMMIT. A commit operation was encountered after the cycle had reached the DML limit or the end of the cycle interval. A cycle can end only on a commit boundary.
- NORMAL_EXPIRY. The cycle ended because the cycle interval expired. The last operation was a commit.
- For combined initial load jobs only: BACKLOG_COMPLETED. The cycle ended because CDC backlog processing completed. The CDC backlog consists of events captured during the initial load phase of the combined job. The backlog includes potential DML changes captured at the beginning or end of the initial load phase and during the transition from the initial load phase to the main CDC incremental processing.
- For combined load jobs only: INITIAL_LOAD_COMPLETED. The cycle ended because the initial load completed.
- For combined load jobs only: RESYNC_STARTED. The cycle ended because the table resync initiated.
Output data files
The data files contain records that include the following information:
•Operation type. Valid values are:
- I for Insert operations.
- U for Update operations.
- D for Delete operations.
- For combined load jobs only: T for Truncate operations.
- For combined load jobs only: X for Delete operations encountered during the initial load phase of a combined load job
- For combined load jobs only: Y for Insert operations encountered during the initial load phase of a combined load job
•Sortable sequence number. In combined initial and incremental load jobs, the sortable sequence number contains a 20-digit prefix that can be used to align rows with the resync version and the load job. The prefix is a combination of the following attributes:
1Incarnation. This nine-digit number is incremented each time the table is resynced. The initial value is 1.
2Schema version. This nine-digit number is incremented each time a schema drift change is propagated for the table. The initial value is 1.
3Phase. This two-digit number changes when transition from unload, to backlog, to CDC is performed. Valid values are:
▪ 00 for Truncation, which is the first data record written during initial load or resync
▪ 01 for a normal insert during initial load or resync
▪ 02 for a change detected during the initial load
▪ 03 for a change detected after the initial load or resync is completed but before the transition back to the main CDC phase
▪ 04 for a change detected during the normal CDC phase
•Data columns
Note:
Insert and Delete records contain only after images. Update records contain both before and after images.