New features and enhancements

Watch the What's New video to learn about the new features and enhancements in the October 2024 release.

The Data Ingestion and Replication Command-Line Interface (CLI) provides a new job metrics command to display statistics for a specified job or a subtask of a job. This enhancement is available for application ingestion and replication jobs and database ingestion and replication jobs.

On the job level, the statistics include records read, records written, and capture progress.

On the subtask level, you can display the following statistics:

You can use the -f field parameter to specify which of these values should be included in the output.

For more information, see the Data Ingestion and Replication Command-Line Interface.

Kerberos authentication is supported for connecting to Oracle sources and targets in database ingestion and replication tasks that use any load type. For CDC sources, it's supported for either the Query-based or Log-based CDC method. It's also supported for connecting to Oracle targets in application ingestion and replication tasks.

When you define an Oracle Database Ingestion connection, select Kerberos in the Authentication Mode field. You must also perform some prerequisite Kerberos configuration tasks.

For more information, see the Oracle Database Ingestion connection in Connectors and Connections.

The October 2024 release of Database Ingestion and Replication includes the following new features and enhancements:

Database ingestion and replication incremental load jobs and combined initial and incremental load jobs can now process DML delete operations on the source as soft deletes on Google BigQuery targets. Database Ingestion and Replication marks the soft-deleted records with a "D" in the INFA_OPERATION_TYPE column on the target without actually deleting the records.

For more information, see Database Ingestion and Replication > Configuring a database ingestion and replication task > Configuring the target > Google BigQuery target properties.

If you use BFILE access to Oracle data and the data is stored in different log locations, Database Ingestion and Replication can now automatically find the logs by querying for all directory objects that begin with ARCHIVELOG_DIR and ONLINELOG_DIR and end with any suffix, for example, ARCHIVELOG_DIR_01, ARCHIVE_LOG_DIR_02, and so on. This feature enables Database Ingestion and Replication to support Oracle standby databases that use log locations different from those on the primary database, replica logs on Amazon RDS database instances, and archive logs in the Fast Recovery Area to which the USE_DB_RECOVERY_FILE_DEST parameter points. You can override the ARCHIVELOG_DIR and ONLINELOG_DIR names by using the custom properties pwx.cdcreader.oracle.database.additional BFILEARCHIVEDIR=<directory_object> and pwx.cdcreader.oracle.database.additional BFILEONLINEDIR=<directory_object>.

If you grant the CREATE ANY DIRECTORY and DROP ANY DIRECTORY privileges to database users who run database ingestion and replication jobs, the directory objects can be created at runtime, as needed. Database Ingestion and Replication performs no cleanup processing on these generated directory objects.

If you use the USE_DB_RECOVERY_FILE_DEST parameter and the CREATE ANY DIRECTORY and DROP ANY DIRECTORY privileges have not been granted, your DBA must create the directory objects daily or weekly, before your database ingestion and replication jobs run, by using a script such as:

If you use the USE_DB_RECOVERY_FILE_DEST parameter and the database user has the CREATE ANY DIRECTORY and DROP ANY DIRECTORY privileges, the directory objects are created as needed at runtime and dropped after 14 days. These directory objects have the naming convention <ARCHIVEDIRNAME>_YYYY_MM_DD.

For more information, see Database Ingestion and Replication > Database Ingestion and Replication sources > Oracle sources.