Database and Application Ingestion and Replication REST API
The REST API resources in this section apply to both database ingestion and replication and application ingestion and replication tasks in the Data Ingestion and Replication service.
•Base URL. Use the serverUrl value from the login response to build the base URL. Add -ing to the end of the first part of the host name (na4 becomes na4-ing). For example, if serverUrl is https://na4.dm-us.informaticacloud.com, the base URL <baseUrl> is https://na4-ing.dm-us.informaticacloud.com.
•URI. Use the following URI for all API calls, where <API name> is the resource that you call, such as task/create:
/dbmi/public/api/v2/<API name>
The full URL for a call is the base URL followed by the URI. For example, use the following URL to create a task:
POST https://na4-ing.dm-us.informaticacloud.com/dbmi/public/api/v2/task/create HTTP/1.1 Content-Type: application/json IDS-SESSION-ID: 123ABC456789defIJK
Get a request body for a new task
A create request needs a request body that defines the task. To get a request body, you can either copy the request body of an existing task, or start from the sample task details.
Copy the request body of an existing task
Perform the following tasks to use the request body of an existing task from the user interface:
1Create and save a task in the user interface.
2Get the task ID. Use the following request:
GET <baseUrl>/dbmi/public/api/v2/task/fetchId? taskName={taskName}&projectId={projectId}&folderId={folderId}
After you copy the request body, change the task name general.name. Then, set the properties that are specific to your source and target. You can remove the properties that you do not need, or send the entire request body. For the property fields and descriptions, see the following topics:
Use these samples as a starting point for the request body when you create a task. Each sample is the response that the task details resource returns for a sample task. The response contains the full task definition in the dbmiTask object.
The samples cover the following tasks:
•Application ingestion and replication tasks that read from Salesforce and write to Snowflake.
•Database ingestion and replication tasks that read from Oracle and write to Snowflake.
To create a task from a sample, copy the dbmiTask object, starting from the type field, and use it as the request body. Update the connection names, the runtime environment, and the project or folder to match your environment.
Application ingestion and replication initial load
Use this sample to create an application ingestion and replication initial load task. The sample task, sf_to_snowflake_full, reads data from Salesforce with the Salesforce Bulk API 2.0 and loads it to Snowflake one time. In the dbmiTask object, general.type is initial.
Application ingestion and replication incremental load
Use this sample to create an application ingestion and replication incremental load task. The sample task, sf_to_snowflake_incremental_full, reads data from Salesforce and replicates changes to Snowflake. In the dbmiTask object, general.type is cdc.
Application ingestion and replication combined load response
Use this sample to create an application ingestion and replication combined load task. The sample task, sf_to_snowflake_combined, reads data from Salesforce and replicates changes to Snowflake. In the dbmiTask object, general.type is combined.
Database ingestion and replication initial load response
Use this sample to create a database ingestion and replication initial load task. The sample task, oracle_to_snowflake_full, reads data from Oracle and loads it to Snowflake one time. In the dbmiTask object, general.type is initial.
Database ingestion and replication incremental load response
Use this sample to create a database ingestion and replication incremental load task. The sample task, oracle_to_snowflake_cdc, uses the query-based change data capture (CDC) method. In the dbmiTask object, general.type is cdc.
Database ingestion and replication combined load response
Use this sample to create a database ingestion and replication combined initial and incremental load task. The sample task, oracle_to_snowflake_combined, loads the source data once and then replicates changes by using the log-based CDC method. In the dbmiTask object, general.type is combined.
Create a database or application ingestion and replication task
Use the task resource to create a database ingestion and replication task or application ingestion and replication task in a specified project and folder.
To create a task, send a POST request to the following resource:
The request payload must include the taskType field to specify the type of task that you want to create:
•appmi. Creates an application ingestion and replication task.
•dbmi. Creates a database ingestion and replication task.
If you omit the value for the taskType field, it defaults to dbmi.
The following payload specifies an application ingestion and replication task:
{ "taskType": "appmi", ... }
The following payload specifies a database ingestion and replication task:
{ "taskType": "dbmi", ... }
Request payload
The following table describes the attributes that you can set within the general object for a database ingestion and replication task or application ingestion and replication task:
Field
Type
Description
name
String
Name of the task.
description
String
An optional description that provides context about the purpose of the task.
location
String
Project or project folder that contains the task definition.
runtimeEnvironment
String
The name of the runtime environment where the task runs.
type
String
Type of load operation that the task performs.
Options are:
- initial. Loads source data read at a single point in time to a target. After the data is loaded, the ingestion and replication job ends.
- cdc. Loads data changes continuously or until the ingestion and replication job is stopped or ends. The ingestion and replication job loads the changes that have occurred since the last time it ran or from a specific start point.
- combined. Performs an initial load of point-in-time data to the target and then automatically switches to replicating incremental data changes made to the same source objects on a continuous basis.
The source, target, and runtime attributes for the task in the request payload depend on the task type and on the specific source and target connector. For the connector-specific source and target attributes, see the Source attributes, Target attributes, and Runtime attributes tables.
Post request and response examples
For sample post requests and responses, including the required payload fields and expected output, see the following examples:
If the task is created successfully, the API returns a 201 Created success code. The response typically includes the following header:
Content-Type: application/json;charset=UTF-8 HTTP/2 201 Created
The response also includes the following information about the task:
Field
Type
Description
location
String
Project and folder name in which the task is created.
taskId
String
ID generated for the newly created task.
frsId
String
FRS reference ID.
name
String
Name of the task.
description
String
Description for the task.
createdBy
String
User ID who created the task.
documentType
String
Type of document: DBMI_TASK or APPMI_TASK based on the taskType value specified in the request.
parentInfo
String
Parent project and structure details.
Failed responses and error codes
If the request fails, the API might return one of the following HTTP error codes:
HTTP status code
Error code
Error message
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed. Access token is missing or expired.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions for this request.
500
CDIR_INTERNAL_ERROR
An unexpected internal error occurred with this job.
503
CDIR_SERVICE_UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when a connection specified in the request does not exist:
{ "error": { "httpStatus": 404, "code": "CDIR_CONNECTION_DOES_NOT_EXIST", "message": "Connection with name Oracle_19RDS_yk-CDC23 does not exist.", "details": [ { "taskName": "ora_2_snflk_api_agust5_case3" } ], "traceId": "b2bca519-8c6c-4a63-a882-9b3f4b6a852b" } }
Source attributes
The following table describes the source connection and task attributes in the request.
Field
Type
Description
Supported source types
connection
String
Name of the connection for the source system that the task uses to read data.
All sources
selectionRules
List
Optional object or table selection rules to select a subset of the source objects or tables for a task. By default, an Include rule that contains only the asterisk (*) wildcard character selects all objects on the source for an application ingestion and replication task or all tables in the source schema for a database ingestion and replication task. To narrow the source objects or tables that the task processes, you can add additional Include rules, Exclude rules, or a combination of both.
The source schema that includes the source tables.
This option applies for database ingestion and replication sources except MongoDB.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
Note:
For MongoDB, to provide the database that stores collections, you must use this schema property instead of database.
journalName
String
For Db2 for i incremental loads in a database ingestion and replication task, the name of the journal that records the changes made to source tables.
Db2 for i
replicationSlotName
String
For PostgreSQL sources in a database ingestion and replication task, the unique name of a PostgreSQL replication slot.
PostgreSQL
replicationPlugin
String
For PostgreSQL sources in a database ingestion and replication task, the replication plugin. Options are:
- pgoutput
- wal2json
Note:
You can specify pgoutput only for PostgreSQL version 10 and later.
PostgreSQL
publication
String
For PostgreSQL sources in a database ingestion and replication task, the publication name used by the pgoutput plugin. Use this parameter only if you specified pgoutput as the replicationPlugin value.
PostgreSQL
restartPointForIncrementalLoad
String
For SAP ECC sources in an application ingestion and replication incremental load task and all sources in a database ingestion and replication incremental load or combined initial and incremental load task, the position in the source change stream or logs from which to start reading changes the first time the job runs.
Options are:
- earliest
- latest
For example: restartPointForIncrementalLoad: earliest, restartPointForIncrementalLoad: latest
For a database ingestion and replication task, default is latest.
For an application ingestion and replication task, the earliest option is not supported. The default is the restartPointForIncrementalLoadTimestamp value.
If you specify this parameter, do not also specify the restarPointForIncrementalLoadPosition or restartPointForIncrementalLoadTimestamp parameter.
SAP ECC, and database ingestion and replication sources
restartPointForIncrementalLoadPosition
Integer
For all sources in a database ingestion and replication incremental load or combined initial and incremental load task, the RBA position in source change stream or logs from which to start reading change records the first time the job runs.
For example:
restartPointForIncrementalLoadPosition: 0
If you specify this parameter, do not also specify the restarPointForIncrementalLoad or restartPointForIncrementalLoadTimestamp parameter.
Database ingestion and replication sources
restartPointForIncrementalLoadTimestamp
Timestamp
For all sources in an application ingestion and replication incremental load task or database ingestion and replication combined initial and incremental load task, the date and time, including AM or PM, in the source change stream or logs from which to start reading change records the first time the job runs. For example: restartPointForIncrementalLoadTimestamp: 2021-08-18 02:50:00 PM
For an application ingestion and replication task, this parameter determines the default behavior.
Note:
If you specify this parameter, do not also specify the restarPointForIncrementalLoad or restartPointForIncrementalLoadPosition parameter.
All sources
cdcInterval
String (time interval)
For Salesforce sources in an application ingestion and replication incremental load or combined load task, the time interval for which the job runs to retrieve change records for CDC.
You can specify cdcIntervalDays, cdcIntervalHours, and cdcIntervalMins parameters.
Salesforce
fetchSize
Integer
For Salesforce sources in an application ingestion and replication initial load or incremental load task, the number of records that the job reads at a time from the source.
The default value for initial load jobs is 50000, and the default value for incremental load jobs is 2000.
If you specified Standard (REST) API for the salesforceAPI parameter, you must change the fetchSize value to 2000.
Salesforce
fetchSizeForInitialLoad
Integer
For Salesforce sources in an application ingestion and replication combined initial and incremental load task, the number of source records that the job reads at a time during the initial unload phase.
The default value is 50000.
If you specified Standard (REST) API for the salesforceAPI field, you must change the fetchSize value to 2000.
Salesforce
fetchSizeForIncrementalLoad
Integer
For Salesforce sources in an application ingestion and replication combined initial and incremental load task, the number of change records that the job can fetch at a time from the source during the incremental phase.
The default value is 2000.
For an initial load task or combined initial and incremental load task, if you specified Standard (REST) API for the salesforceAPI parameter, you must update the fetchSize value to 2000.
Salesforce
includeArchivedAndDeletedRows
Boolean
For Salesforce sources in an application ingestion and replication initial load or combined initial and incremental load task, controls whether the job can read archived and soft-deleted rows from the source during the initial load or unload phase of the combined load.
Default is false.
Salesforce
includeBase64Fields
Boolean
For an application ingestion and replication task with a Salesforce source, controls whether the job can replicate data from source fields that have the Base64 data type.
The default value is false.
Configure this parameter only if you set the salesforceAPI parameter to Standard (REST) API.
Salesforce
maximumBase64BodySize
Integer
For Salesforce sources in an application ingestion and replication task, the maximum body size of Base64 encoded data.
Default is 7 MB.
Pass this parameter only if you set the includeBase64Fields field to true.
Salesforce
includeViews
Boolean
Indicates whether to include views in the table counts and list of table names.
Options are:
- true. Include views.
- false. Do not include views.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
startDate
String
The start date for retrieving source records for the load operation.
Salesforce, Marketo, Microsoft Dynamics 365, SAP (ODP Extractor, OData V2, SAP Mass Ingestion), SAP (ABAP), NetSuite, Workday, ServiceNow, Salesforce Marketing Cloud, Oracle Fusion Cloud
endDate
String
The end date for retrieving source records for the load operation.
Salesforce, Marketo, Microsoft Dynamics 365, SAP (ODP Extractor, OData V2, SAP Mass Ingestion), SAP (ABAP), NetSuite, Workday, ServiceNow, Salesforce Marketing Cloud, Oracle Fusion Cloud,
batchSize
Integer
The batch size used when reading source records.
Salesforce, Marketo, Microsoft Dynamics 365, SAP (ODP Extractor, OData V2, SAP Mass Ingestion), SAP (ABAP), NetSuite, Workday, ServiceNow, Salesforce Marketing Cloud, Oracle Fusion Cloud
mid
String
The unique Member Identification code assigned to your Salesforce Marketing Cloud account.
Salesforce Marketing Cloud
context
String
For SAP sources using the ODP extractor, the context name.
SAP ODP
cdcTechnique
String
For incremental load or combined initial and incremental load tasks, specifies the change data capture method.
Options are:
- CDC Tables. Read data changes directly from the SQL Server CDC tables. This option applies only to SQL Server sources. Default is CDC Tables.
- Log-based. Capture Inserts, Updates, and Deletes in near real time by reading the database transaction logs. Also capture DDL changes based on schema drift settings. For SQL Server sources, data changes are read from the SQL Server transaction log and the enabled SQL Server CDC tables. For Azure SQL Database sources, data changes are read from CDC tables only.
- Query-based. Capture Inserts and Updates by using a SQL WHERE clause that points to a CDC query column. The query column is used to identify the rows that contain the changes made to the source tables since the beginning of the CDC interval.
Oracle, Microsoft SQL Server, MySQL, Db2 for LUW
cdcQueryBasedColumnType
String
For query-based CDC on source types, the column data type of the CDC query column in the source table. The column data type must combine the date and time. For Oracle, use Timestamp.
Oracle, Microsoft SQL Server, MySQL, Db2 for LUW
cdcQueryBasedColumnName
String
For query-based CDC on source types , the case-sensitive name of the CDC query column in the source table. The column must be present in the source table. Maximum length is 70 characters.
Oracle, Microsoft SQL Server, MySQL, Db2 for LUW
cdcQueryBasedDateColumn
String
For query-based CDC on source types that use a separate date column, the date column used to detect changed rows.
Oracle, Microsoft SQL Server, MySQL, Db2 for LUW
cdcQueryBasedTimeColumn
String
For query-based CDC on source types that use a separate time column, the time column used to detect changed rows.
Oracle, Microsoft SQL Server, MySQL, Db2 for LUW
CdcIntervalDays
Integer
The frequency of a query-based change data capture cycle, expressed in days. Sets the days portion of the CDC Interval property.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
CdcIntervalHours
Integer
The frequency of a query-based change data capture cycle, expressed in hours. Sets the hours portion of the CDC Interval field.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
CdcIntervalMins
Integer
The frequency of a query-based change data capture cycle, expressed in minutes. Sets the minutes portion of the CDC Interval field. Default value is 5 minutes.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
lobsEnabled
Boolean
Set this parameter to true if the source contains large object columns that you want to replicate to the target. Supported LOB data types are BLOB, CLOB, NCLOB, LONG, LONG RAW, and XML.
Oracle, PostgreSQL, Db2 for LUW, Db2 for i, SAP ABAP
workdayReplicationType
String
The type of web service to use for data retrieval.
Options are:
- SOAP: Uses SOAP APIs to extract Workday data.
- RaaS: Uses Workday Report-as-a-Service (RaaS) to extract source data from custom objects and fields through custom reports. You can use Workday RaaS only in initial load jobs.
Workday
workdayReportsNumber
Integer
If you use the RaaS API, specify the number of reports you want to extract from the source.
Workday
workdayReportName
String
If you use the RaaS API and you want to extract a single report, specify the name or URL of the custom report you want to read from the source.
Workday
workdayReportsConfigFile
String
If you use the RaaS API and you want to extract multiple reports, specify the path to the CSV file that you created for the list of custom reports that you want to read from the source.
Workday
outputType
String
The format in which you want the data to be stored on the target.
Workday
extractNonDefault
Boolean
Indicates whether extract source fields that do not contain any default value.
Workday
oracleFusionInterfaceType
String
The type of interface used to extract data from the source.
Options are:
- REST. Extracts data from various applications of Oracle Fusion such as ERP, SCM, HCM, Sales, and Services, and transfer data to the target.
- BICC. Extracts bulk data from the source to the target.
Oracle Fusion Cloud
oracleFusionModuleName
String
The name of the Oracle Fusion Cloud Applications Suite module to extract data.
Oracle Fusion Cloud
oracleFusionEnableChildObjects
Boolean
Indicates whether to include child objects from the Oracle Fusion Cloud source in the extraction.
Oracle Fusion Cloud
The following table describes the source advanced attributes in the request:
Field
Type
Description
Supported source types
salesforceAPI
String
For Salesforce sources in an application ingestion and replication task, the type of Salesforce API that you want to use to retrieve the source data. Options are:
- Standard (REST) API. Replicates source fields of Base64 data type.
- Bulk API 2.0. Excludes replication of source fields of Base64 data type. To use Bulk API 2.0, pass the value bulkv2 in the payload.
Default is Bulk API 2.0.
Salesforce
salesforceBulkV2Partitioning
Boolean
For initial and combined load jobs that use the Salesforce Bulk API 2.0, set this parameter to true to use partitioning when reading source data.
Salesforce
salesforceBulkV2RecordsPerPartition
Integer
For initial and combined load jobs that use the Salesforce Bulk API 2.0, the number of records to process per partition when reading data.
Salesforce
salesforceBulkV2MaxParallelPartitions
Integer
The maximum number of partition threads that can be used to query the source for data in parallel during initial load processing or the unload phase of combined jobs that use the Salesforce Bulk API 2.0. Use this property to control the number of source partition queries that can be executed against the source at the same time.
For example, if a table contains data in 100 partitions, all 100 partitions are queried at the same time by default. However, you can use this property to reduce the number of concurrent queries.
Default value is equal to the total number of partitions.
Salesforce
enableSourcePartitioning
Boolean
Controls whether to use multiple partitions to query for the source data to be unloaded in parallel during initial load processing or the unload phase of combined load jobs. Set this property to true to use multiple partitions with one query per partition.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
sourcePartitionCount
Integer
The number of partitions to use for querying for source data in parallel during initial load jobs and the unload phase of combined load jobs, when the Enable Partitioning custom property is selected.
Default is 5.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
jdbcPartitioningTechnique
String
The partitioning technique used to distribute source rows across partitions. Valid values are uniform, which distributes primary keys evenly across partitions, and heuristic, which estimates key ranges for faster calculation.
Default is uniform.
Oracle, Microsoft SQL Server, SAP HANA
sourceMaxParallelPartitions
Integer
The maximum number of partition reader threads that can be used to query the source in parallel during initial load processing or the unload phase of combined jobs.
Oracle, Microsoft SQL Server, SAP HANA
unloadHelperThreadCount
Integer
The number of unload helper threads allocated to an initial load job or the unload phase of a combined job to convert the unloaded data rows into a canonical format that can be passed to the writer.
Default value is 2.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
readEventBatchSize
Integer
The number of payload events written in batch to the internal event queue during CDC processing. When the event queue is implemented as an internal ring buffer, this value is the number of payload events that the reader writes to a single internal buffer slot.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
readerHelperThreadCount
Integer
The number of reader helper threads used during CDC processing to convert change data into a canonical format that can be passed to the target.
Default value is 3.
You can enter a larger value to allow more threads to be available for performing conversion processing in parallel.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
readerInputIsPersisted
Boolean
Set this parameter to true to persistently store transaction data in a disk buffer so that the data can be consumed continually, even when the writing of data to the target is slow or delayed.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
unloadFlashbackDisabled
Boolean
Set this parameter to true to disable the use of Oracle Flashback when fetching data from the database.
Oracle, Microsoft SQL Server, MySQL
salesforceBulkV2Partitioning
Boolean
For initial and combined load jobs that use the Salesforce Bulk API 2.0, set this parameter to true to use partitioning when reading source data.
Salesforce
salesforceBulkV2RecordsPerPartition
Integer
For initial and combined load jobs that use the Salesforce Bulk API 2.0, the number of records to process per partition when reading data.
Salesforce
salesforceBulkV2MaxParallelPartitions
Integer
The maximum number of partition threads that can be used to query the source for data in parallel during initial load processing or the unload phase of combined jobs that use the Salesforce Bulk API 2.0. Use this property to control the number of source partition queries that can be executed against the source at the same time.
For example, if a table contains data in 100 partitions, all 100 partitions are queried at the same time by default. However, you can use this property to reduce the number of concurrent queries.
Default value is equal to the total number of partitions.
Salesforce
readerHistoricalCacheUseCloud
Boolean
Indicates whether the reader historical cache is stored in cloud storage.
Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Teradata, Netezza, Db2 for LUW, Db2 for i, Db2 for z/OS, SAP HANA
Target attributes
The following table describes the target connection and task attributes in the request:
Field
Type
Description
Supported target types
connection
String
Name of the connection for the target system.
All supported targets
schema
String
The target schema where Data Ingestion and Replication creates the target objects or tables.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
bucket
String
The name of an existing bucket container that stores, organizes, and controls access to the data objects that you load to the target.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
directory
String
Name of a root directory to use for storing output files for an incremental load task.
If you select the Connection Directory as Parent option, you can still optionally specify a task target directory. It will be appended to the parent directory to form the root for the data, schema, cycle completion, and cycle contents directories.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
snowflakeRole
String
The Snowflake role assigned to the user to load data to Snowflake.
Snowflake Data Cloud
useTableNameAsTopicName
Boolean
Indicates whether the task writes messages that contain source data to separate topics, one for each source table or object, or writes all messages to a single topic.
Options are:
- true. Write messages to separate table-specific topics.
- false. Write all messages to the single topic that has the name specified in the topicName field.
Kafka
includeSchemaName
Boolean
When you set useTableNameAsTopicName to true for a Kafka target, this setting adds the source schema name in the table-specific topic names. The topic names then have the format schemaname_tablename.
Options are:
- true. Add the source schema name in the table-specific topic names.
- false. Do not add the source schema name in the table-specific topic names.
Kafka
tablePrefix
String
When you set useTableNameAsTopicName to true for a Kafka target, this parameter specifies an optional prefix to add to the table-specific topic names.
For example, if you specify myprefix_, the topic names have the format myprefix_tablename. If you omit the underscore (_) after the prefix, the prefix is prepended to the table name.
Kafka
tableSuffix
String
When you set useTableNameAsTopicName to true for a Kafka target, this parameter specifies an optional suffix to add to the table-specific topic names.
For example, if you specify _mysuffix, the topic names have the format tablename_mysuffix. If you omit the underscore (_) before the suffix, the suffix is appended to the table name.
Kafka
topicName
String
The name of the single Kafka topic to which all messages that contain source data will be written. Use this parameter if useTableNameAsTopicName is set to false.
Kafka
stage
String
The name of the internal staging area that holds the data read from the source before the data is written to the target tables. This name must not include spaces. If the staging area does not exist, it will be automatically created.
Snowflake Data Cloud
applyMode
String
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target.
Options are:
- Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
- Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
After enabling Soft Deletes, any update in the source table during normal or backlog mode results in the deletion of the matching record, insertion of the updated record, and marking of the INFA_OPERATION_TYPE operation as NULL in the target table. Similarly, inserting a record in the source table during backlog mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record.
Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
- Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image. Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
Note:
If you use Query-based CDC method, this property is not applicable.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
outputFormat
String
The format of the output file.
Options are:
- AVRO
- CSV
- PARQUET
Default is CSV.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
parquetFormat
Boolean
If AVRO is specified as the output format in the outputFormat parameter for the target, set this parameter to true to write data in uncompressed Parquet format. Alternatively, you can set the outputFormat parameter to PARQUET and not include this parameter.
Options are:
- true. Write data in uncompressed Parquet format.
- false. Write data in AVRO format.
Note:
If you set this option to true, you must install Visual C++ Redistributable Packages for Visual Studio 2013 on the computer where the Secure Agent runs.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
avroFormat
String
If AVRO is specified as the output format, specifies the format of the Avro schema that will be created for each source table or object.
Options are:
- Avro-Flat. This Avro schema format lists all Avro fields in one record.
- Avro-Generic. This Avro schema format lists all columns or fields from a source table or object in a single array of Avro fields.
- Avro-Nested. This Avro schema format organizes each type of information in a separate record.
Default is Avro-Flat.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
avroSerializationFormat
String
If AVRO is specified as the output format, specify the serialization format of the Avro output files.
Options are:
- Binary
- JSON
Default value is Binary.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
avroSchemaDirectory
String
If AVRO is specified as the output format, specifies the local directory where Data Ingestion and Replication stores Avro schema definitions for each source table or object.
Schema definition files have the following naming pattern: schemaname_tablename.txt
Note:
If you do not specify this directory, no Avro schema definition file is produced.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
fixedDirectoryForEachTable
Boolean
For an initial load task, indicates whether to use the source table or object names as the names of the directories to which the task writes flat files that contain source data for all job runs.
Options are :
- true. Use the source table or object names as the names of the directories for each job run.
- false. Create a new set of directories for each job run by using the following naming pattern: tablename_timestamp.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
fileCompressionType
String
The file compression type to optionally use for AVRO or CSV output files.
Options are:
- Deflate
- gzip
- Snappy
If you do not specify this parameter, the output files are not compressed.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
avroCompressionType
String
The optional Avro compression type if the output format is set to AVRO.
Options are:
- None
- Bzip2
- Deflate
- Snappy
Default value is None, which means no compression is used.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
parquetCompressionType
String
The optional Parquet compression type to use when the output format is set to PARQUET or AVRO by the parquetFormat true parameter.
Options are:
- None
- gzip
- Snappy
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
deflateCompressionLevel
Integer
The compression level of 0-9 to use for Deflate compression type, which is set by the avroCompressionType parameter.
Default is 0.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
addDirectoryTags
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether the task adds the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning.
Options are:
- true. Add the "dt=" prefix to the names of apply cycle directories.
- false. Do not change the names of apply cycle directories.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
directoryTags
String
This field is dependent on the addDirectoryTags field.
For example:
addDirectoryTags: true,
directoryTags: \directory.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
renamingRules
List
Optional rules for renaming target tables that correspond to the source tables or objects.
For example:
- source: "*" target: "*_1"
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Microsoft Azure Synapse Analytics, Flat File, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
dataTypeRules
List
Optional data type mapping rules that override the default source-to-target data-type mapping rules. The default mappings are described under Default Data-Type Mappings in the Application Ingestion and Replication and Database Ingestion and Replication documentation.
For example:
- source: Int target: String
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks
cdcCompatibleFormat
Boolean
Indicates whether to include UNDO data in the output.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
useGenericDatatypeFramework
Boolean
Set this parameter to true to use the generic data type framework for mapping source data types to target data types.
All supported targets
The following table describes the target advanced attributes that you can set for a task:
Field
Type
Description
Supported target types
createUnmanagedTables
Boolean
Set this parameter to true if you want the task to create Databricks target tables as unmanaged tables. After you deploy the task, you cannot change this parameter to switch to managed tables. Default is false, which creates managed tables.
Databricks
unmanagedTablesParentDirectory
String
If you create Databricks unmanaged tables, specify a parent directory in Amazon S3 or Microsoft Azure Data Lake Storage to hold the Parquet files that are generated for each target table when captured DML records are processed.
For example:
/data/unmanaged
Databricks
writerTargetTableFormat
String
The format of the target tables, such as standard or Iceberg.
To enable the use of Snowflake-managed Iceberg tables, set this property to iceberg.
Snowflake Data Cloud
writerTargetTableExternalVolumeName
String
If the target uses the Iceberg table format, the name of the external volume used for the target tables.
Snowflake Data Cloud
writerTargetTableExternalStorageBaseLocation
String
If the target uses the Iceberg table format, the base URL for the external storage location. If you do not specify this value, the base URL from the external volume definition is used.
Snowflake Data Cloud
writerSnowflakeIngestionMethod
String
Set to SNOWPIPE_STREAMING to use the Snowpipe Streaming API to quickly stream rows of data directly to Snowflake Data Cloud target tables with low latency instead of first writing the data to stage files. This option is applicable for all load types.
Ensure that the Snowflake connection uses KeyPair authentication.
Snowflake Data Cloud
writerSnowflakeDeferredMergeInterval
Integer
If you set writerSnowflakeIngestionMethod to SNOWPIPE_STREAMING, you can optionally set the frequency, in seconds, at which change data rows are merged and applied to the Snowflake target tables.
The merge frequency affects how often the stream change data is merged to the Snowflake base table. A Snowflake view joins the stream change data with the base table. Set this value to balance the costs of merging data to the base table with the performance of view join processing.
This field applies to incremental load and combined initial and incremental load tasks.
Valid values are 60 through 604800 seconds. Default is 3600 seconds.
Snowflake Data Cloud
addHeadersToCSVFile
Boolean
If CSV is selected as the output format, indicates whether to add a header row with column names to the CSV output files.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
addOperationType
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the source SQL operation type in the output that the job replicates to the target.
For an incremental load or combined initial and incremental load task, default is true. For an initial load task, default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addOperationTime
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the source SQL operation time in the output that the job replicates to the target. For initial loads, the job always writes the current date and time. Default is false for all load types.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addOperationOwner
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the owner of the source SQL operation in the output that the job replicates to the target.
For initial loads, the job always writes "INFA" as the owner. Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addOperationTransactionId
Boolean
For targets in an incremental load or combined initial and incremental load task, set this parameter to true to add a metadata column that includes the source transaction ID in the output that the job replicates to the target for SQL operations.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
addBeforeImages
Boolean
For targets in an incremental load or combined load task, set this parameter to true to include UNDO data in the output that the job writes to the target.
Default is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
asyncWrite
Boolean
Controls whether to use synchronous delivery of messages to Kafka.
Options are:
- true. Use asynchronous delivery. Data Ingestion and Replication sends messages as soon as possible, without regard for the order in which the changes were retrieved from the source.
- false. Use synchronous delivery. Kafka must acknowledge each message as received before Data Ingestion and Replication sends the next message. In this mode, Kafka is unlikely to receive duplicate messages. However, performance might be slower.
Default is true.
Kafka
producerConfigurationProperties
String
A comma-separated list of key=value pairs to enter Kafka producer properties for Apache Kafka, Confluent Kafka, or Kafka-enabled Event Hubs targets.
If you have a Confluent target that uses Confluent Schema Registry to store schemas, you must specify the following properties:
The number of writer helper threads that are used to convert incoming change data rows or initial unload rows to the output format configured for the target, such as Avro, CSV, or Parquet. Default value is 2.
All supported targets
writerDistributorsCount
Integer
The number of distributors that can run on separate threads in parallel to process data during an initial load job or the unload phase of a combined load job when the Writer Unload Multiple Distributors custom property is set to true. Using parallel distributor threads can improve job performance, particularly for high-volume data transfers. Default value is 3.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Snowflake Data Cloud, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
writerUnloadUseMultipleDistributors
Boolean
Indicates whether multiple distributor threads can be used to process data in parallel during initial load jobs and the unload phase of combined load jobs. Default value is false.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Snowflake Data Cloud, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
auditColumnsPrefix
String
Adds a prefix to the names of the added metadata columns to easily identify them and to prevent conflicts with the names of existing columns. The default value is INFA_.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
jsonFormatType
String
The format type used when writing output data in JSON format.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Kafka, Google Cloud Storage
checkpointsInTarget
Boolean
Set to true if you want the job to save a checkpoint in the Kafka header each time a source transaction commits. When the job is restarted, checkpoint information is retrieved from the Kafka header.
Kafka
addLastReplicatedTime
Boolean
Adds a metadata column that records the timestamp in UTC format at which a record was inserted or last updated in the target table. For initial loads, all loaded records have the same timestamp. For incremental and combined initial and incremental loads, the column records the timestamp of the last DML operation that was applied to the target.
Databricks, Amazon Redshift, Snowflake Data Cloud
caseTransformation
String
Controls the case of letters in the target names.
Case transformation strategy options to handle the case of letters in generated target table (or object) names and column (or field) names are:
- Same as source. Use the same case as the source table (or object) names and column (or field) names. Enter value "none" to use this option.
- UPPERCASE. Use all uppercase.
- lowercase. Use all lowercase.
Google BigQuery, Snowflake Data Cloud
useConnectionDir
Boolean
Uses the parent directory specified in the connection properties.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
addOperationSequence
Boolean
Adds a metadata column that records a generated, ascending sequence number for each change operation that the job inserts into the target tables. The sequence number reflects the change stream position of the operation.
This field is applicable when you use Apply Mode.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
cycleId
Boolean
Set to true to add a metadata column that includes the cycle ID of each CDC cycle in each target table. A cycle ID is a number that's generated by the CDC engine for each successful CDC cycle. If you integrate the job with Data Integration taskflows, the job can pass the minimum and maximum cycle IDs in output fields to the taskflow so that the taskflow can determine the range of cycles that contain new CDC data. This capability is useful if data from multiple cycles accumulates before the previous taskflow run completes.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL
schemaDirectory
String
A custom directory for storing the schema file, if you want to store it in a directory other than the default. This property is optional.
For initial loads, the schema is stored in the data directory by default. For incremental load and combined initial and incremental load tasks, the default value is {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the data directory. Enclose placeholders in curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}.
The task writes the schema only to output data files in CSV format. Parquet and Avro data files contain their own embedded schema.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcDataDirectory
String
For incremental load and combined initial and incremental load tasks, the path to the subdirectory that contains the cdc-data data files.
You can build the path from a folder path, timestamp values, schema name, and table name. Enclose each placeholder in curly brackets { }. Placeholder values are not case sensitive. Use {TaskTargetDirectory} for a task-specific base directory on the target instead of the folder path specified in the connection properties.
Default value is {TaskTargetDirectory}/data/{TableName}/data.
If you use Amazon S3 with the Apache Iceberg Open Table format, this property does not apply. Files are saved under the warehouse base directory.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcCompletionDirectory
Boolean
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default value is {TaskTargetDirectory}/cycle/completed.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcContentsDirectory
String
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {TaskTargetDirectory}/cycle/contents.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcDataDirectoryPartitioning
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether to create a timestamp subdirectory for each CDC cycle under each data directory.
Options are:
- true. Create a timestamp subdirectory for each CDC cycle under the summary contents and completed subdirectories.
- false. Write individual data files to the same directory without a timestamp, unless you define an alternative directory structure.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcSummaryDirectoryPartitioning
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether to create a timestamp subdirectory for each CDC cycle under the summary contents and completed subdirectories.
Options are:
- true. Create a timestamp subdirectory for each CDC cycle under the summary contents and completed subdirectories.
- false. Do not create timestamp subdirectories under the summary contents and completed subdirectories.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
cdcContentsListFiles
Boolean
For incremental load and combined initial and incremental load tasks, indicates whether to list individual data files under the contents subdirectory.
Options are:
- true. List individual data files under the contents subdirectory.
- false. Do not list individual data files under the contents subdirectory.
If you set cdcSummaryDirectoryPartitioning to false, the default is true. All individual files are listed in the contents subdirectory unless you configure custom subdirectories by using placeholders, such as for timestamp or date.
If you set cdcDataDirectoryPartitioning to true, you can still set this property to true to list individual files and group them by CDC cycle.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Google Cloud Storage
The following table describes custom attributes that you can set for the target:
Field
Type
Description
Supported target types
forceRunOnOtherAgent
Boolean
Set to true to force the job to run on a Secure Agent other than the one it would run on by default.
All supported targets
forceRunOnSpecificAgent
String
Set this property to force an application or database ingestion and replication job to run on a specific agent in the group. This setting applies only to incremental load jobs and the incremental CDC portion of combined load jobs.
This capability is helpful when you want to manually assign CDC jobs to a specific agent specified by the agent ID.
For example, "forceRunOnSpecificAgent": "agent-007"
To find the ID, go to Runtime Environments in Administrator, expand the runtime environment for the Secure Agent group, click the agent, and copy the agent ID from the browser URL.
All applicable targets
Runtime attributes
The following table describes the run-time options that you can set for a task:
Field
Type
Description
Supported target types
schemaDriftOptions
String
For Microsoft SQL Server, Oracle, or PostgreSQL sources in a database ingestion and replication incremental load or combined initial and incremental load task, the schema drift options for DDL operations.
Tasks that have a Microsoft Azure Synapse Analytics target ignore the renameColumn option.
All supported targets
numberOfRowsInOutputFile
Integer
The maximum number of rows that the task writes to an output data file on a target. When set to null, no row limit is applied and each output file contains all rows for the corresponding table or partition.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Microsoft Azure Synapse Analytics, Flat File, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, Google Cloud Storage
fileExtensionBasedOnFileType
Boolean
Indicates whether you want the output data files to have the .dat extension.
- true. The output files have file-name extensions based on their file types.
- false. The output files have the .dat extension.
The default value is true.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
applyCycleChangeLimit
Integer
In an incremental load task, the number of records that must be processed before the job ends an apply cycle. When this record limit is reached, the job ends the apply cycle and writes the change data to the target.
Default is 10000.
Note:
Either the applyCycleChangeLimit parameter or applyCycleIntervaltime_unit parameter must have a non-zero value.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
applyCycleIntervalDays
Integer
In an incremental load task, the number of days that must elapse before the application ingestion and replication job or database ingestion and replication job ends an apply cycle.
You can specify this parameter with the applyCycleIntervalHours, applyCycleIntervalMins, and applyCycleIntervalSecs parameters or specify a subset of these parameters.
Default is 0.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
applyCycleIntervalHours
Integer
For targets in an incremental load task, the number of hours that must elapse before a job ends an apply cycle.
Default is 0.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
applyCycleIntervalMins
Integer
In an incremental load task, the number of minutes that must elapse before the application ingestion and replication job or database ingestion and replication job ends an apply cycle.
Default is 15.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
applyCycleIntervalSecs
Integer
For targets in an incremental load task, the number of seconds that must elapse before the application ingestion and replication job or database ingestion and replication job ends an apply cycle.
Default is 0.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
lowActivityFlushHours
Integer
The amount of time, in hours, that must elapse during a period of no change activity on the source before the application ingestion and replication job or database ingestion and replication job ends an apply cycle.
You can use this parameter in conjunction with the lowActivityFlushMins parameter or specify only one of these parameters. When this time limit is reached, the job ends the apply cycle and writes the change data to the target.
Default is 0. If you do not specify a value for lowActivityFlushHours or lowActivityFlushMins, the application ingestion and replication job or database ingestion and replication job ends apply cycles only after either the applyCycleChangeLimit or cycle interval time limit is reached.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
lowActivityFlushMins
Integer
The amount of time, in minutes, that must elapse during a period of no change activity on the source before the application ingestion and replication job or database ingestion and replication job ends an apply cycle.
You can use this parameter in conjunction with the lowActivityFlushHours parameter or specify only one of these parameters. When the overall time limit is reached, the ingestion and replication job ends the apply cycle and writes the change data to the target.
Default is 0. If you do not specify a value for lowActivityFlushHours or lowActivityFlushMins, the job ends apply cycles only after either the applyCycleChangeLimit or cycle interval time limit is reached.
Amazon S3, Microsoft Azure Data Lake Storage Gen2, Flat File, Google Cloud Storage
executeInTaskflow
Boolean
Indicates whether the task can be run within a taskflow.
Any load type with Amazon Redshift, Oracle, SQL Server, or Snowflake target.
Initial load jobs with Amazon S3, Azure SQL Database, Databricks, Google BigQuery, Google Cloud Storage, Kafka, Microsoft Azure Data Lake Storage Gen2, Microsoft Azure Synapse Analytics, Microsoft Fabric OneLake, Oracle Cloud Object Storage, or PostgreSQL target
enableAutoTuning
Boolean
Set this parameter to true to enable auto-tuning for a database ingestion and replication initial load job or the unload phase of a combined load job. Auto-tuning optimizes job performance on the source and target sides based on performance and system metrics collected automatically from your environment, such as network and database latency, row counts, table sizes, CPU core counts, and memory usage, along with application metrics such as JVM heap allocation and task capacity.
Default is false.
Db2 for i, Db2 for LUW, Db2 for z/OS, MySQL, Oracle, PostgreSQL, SAP HANA, SQL Server, or Teradata source type with any supported target type.
The following table describes checkpoint attributes that define when and how progress is saved during data ingestion and replication:
Field
Type
Description
checkpointAllRows
Boolean
Indicates whether the incremental load task job performs checkpoint processing for every message that is sent to the target. When this parameter is set to true, the checkpointEveryCommit, checkpointRowCount, and checkpointFrequencySecs parameters are ignored.
Default is true.
Kafka
checkpointEveryCommit
Boolean
For targets in an incremental load job, indicates whether the job performs checkpoint processing for every commit that occurs on the source.
Default is false. If you set this parameter to true, do not also specify checkpointAllRows.
Kafka
checkpointRowCount
Integer
For a Kafka target in an incremental load task, the maximum number of messages that the job sends to the target before adding a checkpoint.
If you set this option to 0, an ingestion and replication job does not perform checkpoint processing based on the number of messages. If you set this option to 1, the ingestion and replication job adds a checkpoint for each message.
Default is 0. If you set this parameter to a non-zero number, do not also specify checkpointAllRows.
Kafka
checkpointFrequencySecs
Integer
For a Kafka target in an incremental load task, the maximum number of seconds that must elapse before the ingestion and replication job adds a checkpoint. If you set this option to 0, the job does not perform checkpoint processing based on elapsed time.
Default is 0. If you set this parameter to a non-zero number, do not also specify checkpointAllRows.
Kafka
The following table describes custom attributes that you can set for a task:
Field
Type
Description
customProperties
String
Custom properties for the source, target, and task definition that Informatica provides to meet special requirements. Specify these properties only at the direction of Informatica Global Customer Support.
For example: "customProperties": { "readerInputIsPersisted": true
All
use.table.based.restart
Boolean
For application and database ingestion and replication jobs that use the Query-based CDC method, if the job is aborted or stopped for any reason, you can use this property to define the restart and recovery model.
Options are:
- true. Use table-based restart. For every cycle of the Query-based CDC interval, the DML data from individual source tables is applied to the corresponding target tables after the current CDC interval completes. The data of all the tables is committed together at the end of the CDC cycle. If the property is set to true, the job can automatically switch over to another Secure Agent in the Secure Agent group.
- false. Use cycle-based restart. The DML data from source tables is applied to the corresponding target tables only when the DML data of all candidate tables is available for a current CDC interval cycle. The data is kept in persistent cache while the job waits for the data for all tables to be fetched successfully. If the property is set to false, the job cannot automatically switch over to another Secure Agent in the Secure Agent group.
Microsoft Azure Synapse Analytics, Snowflake Data Cloud, Amazon Redshift, Google BigQuery, Databricks, Oracle, Microsoft SQL Server, PostgreSQL, MySQL, Teradata, Db2 for i, Netezza, Db2 for LUW, Db2 for z/OS
The following table describes schedule attributes that you can set for a task:
Field
Type
Description
schedule
String
The name of a predefined schedule in Administrator to run job instances for initial load tasks automatically after deployment.
All endpoints
Example: Create a Salesforce to Snowflake task
In this example, you create application ingestion and replication tasks that load Salesforce data to Snowflake using initial, incremental, and combined initial and incremental loads.
Send each request body in a POST request to the following resource:
POST https://qa-pod1-ing.rel.infaqa.com/dbmi/public/api/v2/task/create
Initial load
The following request body creates an initial load task named sf_to_snowflake_initial. The task uses the Salesforce Bulk API 2.0 to read the Custom_all__c object, splits the data into partitions of 10,000 records with up to five partitions read in parallel, and loads the data to the ADOBE_PB schema in Snowflake one time:
The following request body creates an incremental load task named sf_to_snowflake_incremental. The task checks the Period object for changes every five minutes, starting from the restart point 2026-10-12 10:25:56 AM. The task renames the target table to Period_O12C:
Optional target properties. In the example, the task doesn't add the last replicated time or the cycle ID to the target tables, and it doesn't change the case of the target table and column names.
runtimeOptions.schemaDriftOptions
Replicates modified and added fields and ignores dropped and renamed fields.
replicationRules.overrides
Repeats the renaming rule for the replication.
A successful request returns HTTP status code 201. The response is similar to the following example:
The following request body creates a combined load task named sf_to_snowflake_combined. The task loads the cli_sf__c object to the cli_sf__c3 table in Snowflake and then checks for changes every five minutes:
In this example, you create database ingestion and replication tasks that load Oracle data to Snowflake using initial, incremental, and combined initial and incremental loads.
Send each request body in a POST request to the following resource:
POST <baseUrl>/dbmi/public/api/v2/task/create
Initial load
The following request body creates a combined load task named sf_to_snowflake_combined. The task uses the Salesforce Bulk API 2.0 to load the cli_sf__c object once, excluding the Period object, and then checks for changes every five minutes. The task renames the target table to cli_sf__c3:
The following request body creates an incremental load task named oracle_to_snowflake_cdc_incremental. The task uses the query-based change data capture (CDC) method to read changes from the SRC_ALLNUM_8K table in the QUERYBASEDCDC schema every five minutes, starting from the latest change. The task uses the QUERY_BASED_COL timestamp column to identify changed rows and loads the data to the NUM_SPIPE2 table in the SNOW_PB schema in Snowflake. The task replicates added and modified columns and ignores renamed and dropped columns:
The following request body creates a combined load task named oracle_to_snowflake_combined. The task loads the COMPO8 table in the SP_UNLOAD schema once, and then uses the log-based CDC method to check for changes every five minutes, starting from the latest change. The task renames the target table to COMPO8_O9 and adds audit columns that describe each change, including the operation type, time, owner, transaction ID, and sequence:
Interval at which the task checks the source for changes. In this example, the task checks every 5 minutes.
source.readerInputIsPersisted
Indicates whether the task stores transaction data in a disk buffer. In this example, the task stores transaction data in a disk buffer.
target.addBeforeImages
Indicates whether the task adds the before images of changed rows to the target.
target.auditData
Settings for the audit columns. The value audit for applyMode applies an audit trail of every change to the target. The other properties add columns that record the operation type, time, owner, transaction ID, and sequence. The column names start with INFA_.
A successful request returns HTTP status code 201. The response is similar to the following example:
Alternatively, you can get the details from the task user interface as follows:
•To find the project ID where your task resides, go to Explore and select the project. The project ID is visible in the URL.
•To find the folder ID, select the folder in the project. The folder ID appears in the URL.
Sample request
Use the following sample as a reference to get the task ID:
GET https://pod-ing.ics.dev:11447/dbmi/public/api/v2/task/fetchId?projectId=1lHbX3urKB8dHJ071tXV8Fg&folderId=5GX14m3CuFAeVWqnsSX9mC&taskName=zendeskToSlfk
Sample response
A successful response returns a summary including the task name and task ID similar to the following example:
To paginate the results, you can include the following pagination attributes in your GET request:
GET /dbmi/public/api/v2/task/fetchId?orderBy={nameOfThecolumn}&pageNo={int}&pageSize={int}
You can include the following pagination attributes in the request:
Fields
Type
Description
orderBy
string
Column name to sort the results.
You can use one of the following column names to specify the sorting order:
- name
- createdTime
- lastUpdatedTime
- lastAccessedTime
pageNo
Integer
Page number to retrieve after pagination.
pageSize
Integer
Number of records per page.
Deploy the task
After you get the taskId, deploy the task using the deploy resource.
Use the following POST request to deploy the task:
POST /dbmi/public/api/v2/task/deploy/{taskId}
Include the following attribute in the request:
Field
Type
Description
taskId
integer
The ID associated with the task.
Sample response
A successful deployment request returns a 202 status code with a summary including the job ID and deployment status, indicating that deployment has started:
Deployment runs asynchronously. For tasks that include many tables, deployment might take longer. The API responds immediately with a "Deployment started" message. To monitor the status, query the Get Status Status API with the job ID. For more information, see Get Job Status API.
Failed responses
If the request fails, the API might return one of the following HTTP error codes:
HTTP status code
Error code
Error message
400
CDIR_VALIDATION. INVALID_VALUE
Configuration value {field} is invalid.
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions to perform this job.
404
CDIR_TASK_NOT_FOUND
Task{TaskId} was not found.
409
CDIR_JOB_ALREADY_DEPLOYED
Job {jobId} is already deployed.
500
CDIR_INTERNAL_ERROR
Unexpected error occurred during deployment.
503
CDIR_SERVICE_ UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when you deploy a task that is in an error state:
Include the following attribute in the request body:
Field
Type
Description
jobId
integer
The ID of the job that you want to start.
Sample body
Use the following request body to pass the job ID to the resource:
{ "jobId": "15" }
Each task corresponds to a single job ID. Therefore, when you retrieve the job ID for a given task, only one job ID is returned.
Sample response
A successful response returns a 202 status code indicating the job is accepted to start:
{ "status": "Success", "code": 202, "message": "Job with org id 2HimcQ9cXW5kLUCOUkXGi8 and job name zendeskToSlfk-UserA-test_8 has been accepted to start successfully." }
Note:
The start job API initiates the job asynchronously. The response confirms that the job has been accepted to start. To check if the job started successfully and monitor its progress, query the job status API. For more information, see Get Job Status API.
Failed responses
If the request fails, the API might return one of the following HTTP error codes:
HTTP status code
Error code
Error message
400
CDIR_VALIDATION_MISSING_FIELD
Required parameter {field} is missing.
400
CDIR_VALIDATION_ INVALID_VALUE
Value for {field} is invalid.
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed. Access token is missing or expired.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions for this request.
404
CDIR_JOB_NOT_FOUND
Job {jobId} was not found.
409
CDIR_JOB_ALREADY_RUNNING
Job {jobId} is already running.
409
CDIR_JOB_NOT_DEPLOYED
Deploy the job {jobId} before performing this operation.
409
CDIR_JOB_INVALID_STATE like failed/stopped/aborted
Job {jobId} cannot perform the requested action while in state {state}. Verify the job's status before retrying.
500
CDIR_INTERNAL_ERROR
An unexpected internal error occurred with this job.
503
CDIR_SERVICE_UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when you start a job that is in a stopped state:
{ "error": { "httpStatus": 409, "code": "CDIR_JOB_INVALID_STATE", "message": "Job {134609} cannot perform the requested action while in its current state. Verify the job's status before retrying.", "details": [ { "jobId": "134609" } ], "traceId": "464ca200-2f04-4bb3-bc90-365641aa65f7" } }
Stop a database or application ingestion and replication job
Use the stop action of the job resource to stop a database ingestion and replication job or application ingestion and replication job that has the status Up and Running, Running with Warning, or On Hold.
POST request
Use the following POST request to stop a job:
POST /dbmi/public/api/v2/job/stop
Include the following attribute in the request body:
Field
Type
Description
jobId
integer
The ID of the job that you want to stop.
Request body
Use the following request body to pass the job ID to the resource:
{ "jobId": "<jobId>" }
Success response
A successful response returns a status indicating the job has been accepted to stop.
{ "status": "Success", "code": 202, "message": "Job with org id 2HimcQ9cXW5kLUCOUkXGi8 and job name zendeskToSlfk-UserA-test_8 has been accepted to stop successfully." }
Note:
The stop job API processes requests asynchronously. To verify whether the job stopped successfully, query the Get Status Status API with the job ID. For more information, see Get Job Status API.
Failed responses
If the request fails, the API might return one of the following HTTP error codes:
HTTP status code
Error code
Error message
400
CDIR_VALIDATION_MISSING_FIELD
Required parameter {field} is missing.
400
CDIR_VALIDATION_INVALID_VALUE
Value for {field} is invalid.
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed. Access token is missing or expired.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions for this request.
404
CDIR_JOB_NOT_FOUND
Job {jobId} was not found.
409
CDIR_JOB_NOT_DEPLOYED
Deploy the job {jobId} before performing this operation.
409
CDIR_JOB_INVALID_STATE like failed, aborted
Job {jobId} cannot perform the requested action while in state {state}. Verify the job's status before retrying.
500
CDIR_INTERNAL_ERROR
An unexpected internal error occurred with this job.
503
CDIR_SERVICE_UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when you stop a job that is in a deployed state:
{ "error": { "httpStatus": 409, "code": "CDIR_JOB_INVALID_STATE", "message": "Job {135332} cannot perform the requested action while in an unexpected state. Verify the job's status before retrying.", "details": [ { "jobId": "135332" } ], "traceId": "2a4d019f-fcc4-4eb9-9934-649f6a2e0ad3" } }
Resume a database or application ingestion and replication task
Use the resume action of the job resource to resume a database ingestion and replication job or application ingestion and replication job that failed or was stopped or aborted because of a schema drift error.
POST request
Use the following POST request to resume a job:
POST /dbmi/public/api/v2/job/resume
Include the following required attributes in the request body:
Field
Type
Description
jobId
integer
The ID of the job that you want to resume.
parameters
Map<String, JSON>
Overrides schema drift options and controls how a job that is in a Stopped, Aborted, or Failed state resumes.
For an incremental load job, resume options are:
- replicate. Allows the job to replicate the DDL changes to the target.
- ignore. Does not replicate DDL changes that occur on the source database to the target.
- stop. Stops processing the source table on which the DDL change occurred.
Set the resume option based on what your source supports:
- REPLICATE. For incremental or combined load jobs that are in the Stopped, Aborted, or Failed state due to a schema drift error, allows the job to replicate the DDL change to the target. This override applies only to tables in an Error state caused by the configured Stop Table or Stop Job schema drift settings.
- IGNORE. For incremental or combined load jobs that are in the Stopped, Aborted, or Failed state due to a schema drift error, does not replicate DDL changes that occur on the source database to the target. This override applies only to tables in an Error state caused by the Stop Table or Stop Job schema drift settings.
- STOP_TABLE. For incremental or combined load jobs that are in the Stopped, Aborted, or Failed state due to a schema drift error, stops processing the source table on which the DDL change occurred. This override applies only to tables in an Error state caused by the configured Stop Table or Stop Job schema drift settings.
- RESYNC. For combined load jobs that have Failed or Running With Warnings state due to a schema drift error, resynchronizes the tables that are in error. You can't use this command if the job uses a source that allows selecting individual columns for replication or if the source does not support schema drift.
- RESYNC_RETAIN. For combined load jobs that are in the Stopped, Aborted, or Failed state and have a source that supports selecting individual columns for data replication, resynchronizes the same columns that have been processed for CDC, retaining the current structure of the source and target tables. No checks for changes to the source or target table definitions are performed. If source DDL changes affected the source table structure, those changes are not processed.
- RESYNC_REFRESH. For combined load jobs that are in the Stopped, Aborted, or Failed state and have a source that supports selecting individual columns for data replication, resynchronizes the target tables with the latest source table definitions, including any DDL changes. After the target tables are refreshed, the structure of source and target tables match.
Note:
If you are resynchronizing a subtask for a database ingestion and replication combined load job that has a Db2 for i, Db2 for LUW Log-based CDC, Oracle, PostgreSQL, SAP HANA, or SQL Server source, or an application ingestion and replication combined load job that has a NetSuite, Salesforce, or SAP Mass Ingestion source, use RESYNC_RETAIN or RESYNC_REFRESH option instead of the Resync option.
Request body
Use the following request body to pass attributes without resume options:
If you specify the RESYNC or RESYNC_RETAIN resume option, the API automatically applies a generic pattern "*.*" to resyncPatterns, so you do not need to specify a pattern explicitly. For IGNORE, REPLICATE, and STOP_TABLE resume options, the API uses the pattern you specify in the resumeOptions field. The API creates a SchemaChangeHandlingOption for the pattern with a list of SchemaDriftItems for schema operations such as DROP_COLUMN, ADD_COLUMN, RENAME_COLUMN, and MODIFY_COLUMN. Each operation is mapped to a corresponding SchemaDriftAction derived from the resume action.
In this request, all tables in the sales schema replicates the schema changes.
Success response
A successful response returns the Success status, which indicates the job has been accepted to resume.
{ "status": "Success", "code": 202, "message": "Job with org id 2HimcQ9cXW5kLUCOUkXGi8 and job name zendeskToSlfk-UserA-test_8 has been accepted to resume successfully." }
Note:
The resume job API processes requests asynchronously. To verify whether the job resumed successfully, query the Get Job Status API with the job ID. For more information, see Get Job Status API.
Failed responses
If the request fails, the API returns one of the following HTTP error codes:
HTTP status code
Error code
Error message
400
CDIR_VALIDATION_MISSING_FIELD
Required parameter {field} is missing.
400
CDIR_VALIDATION_ INVALID_VALUE
Value for {field} is invalid.
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed. Access token is missing or expired.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions for this request.
404
CDIR_JOB_NOT_FOUND
Job {jobId} was not found.
409
CDIR_JOB_ALREADY_RUNNING
Job {jobId} is already running.
409
CDIR_JOB_NOT_DEPLOYED
Deploy the job {jobId} before performing this operation.
409
CDIR_JOB_INVALID_STATE like deployed and completed
Job {jobId} cannot perform the requested action while in state {state}. Verify the job's status before retrying.
500
CDIR_INTERNAL_ERROR
An unexpected internal error occurred with this job.
503
CDIR_SERVICE_UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when you try to resume a job that is already in a running state:
Undeploy a database or application ingestion and replication job
Use the undeploy action of the job resource to undeploy a database ingestion and replication job or application ingestion and replication job. Before you undeploy a job, ensure that it is not running and the status is Aborted, Completed, Deployed, Failed, or Stopped .
POST request
Use the following POST request to undeploy a job.
POST /dbmi/public/api/v2/job/undeploy
Include the following attribute in the request body:
Field
Type
Description
jobId
integer
The ID of the job that you want to undeploy.
Request body
Use the following request body to pass the job ID to the resource:
{ "jobId": "<jobId>" }
Success response
A successful response returns a Success status which indicates that the job is accepted for undeployment.
{ "status": "Success", "code": 202, "message": "Job with org id 2HimcQ9cXW5kLUCOUkXGi8 and job name zendeskToSlfk-UserA-test_8 has been accepted to undeploy successfully." }
Note:
The undeploy job API processes requests asynchronously. To verify whether the job undeployed successfully, query the Get Job Status API with the job ID. For more information, see Get Job Status API.
Failed responses
If the request fails, the API returns one of the following HTTP error codes:
HTTP status code
Error code
Error message
400
CDIR_VALIDATION_MISSING_FIELD
Required parameter {field} is missing.
400
CDIR_VALIDATION_ INVALID_VALUE
Value for {field} is invalid.
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed. Access token is missing or expired.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions for this request.
404
CDIR_JOB_NOT_FOUND
Job {jobId} was not found.
409
CDIR_JOB_ALREADY_RUNNING
Job {jobId} is already running.
409
CDIR_JOB_NOT_DEPLOYED
Deploy the job {jobId} before performing this operation.
500
CDIR_INTERNAL_ERROR
An unexpected internal error occurred with this job.
503
CDIR_SERVICE_UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when you undeploy a job that is in an undeployed state:
{ "error": { "httpStatus": 409, "code": "CDIR_JOB_NOT_DEPLOYED", "message": "Deploy the job {138514} before performing this operation.", "details": [ { "jobId": "138514" } ], "traceId": "2fa22ed7-6d23-4f35-b209-881fe8ac5fad" } }
Get job status for a database or application ingestion and replication task
Use the status action of the job resource to retrieve the status of a database ingestion and replication or application ingestion and replication job.
GET request
Use the following GET request to fetch the status of a job.
GET /dbmi/public/api/v2/job/status
Include the following attribute in the request body:
Field
Type
Description
jobId
integer
The ID of the job for which you want to retrieve the status.
Request body
Use the following request body to pass the job ID to the resource:
{ "jobId": "<jobId>" }
Get response
If successful, the response includes the following information about the status of a job:
Field
Type
Description
jobId
Long
Unique identifier of the job.
assetName
String
Name of the task.
assetType
String
Type or category of the asset. Returns either APPMI_TASK or DBMI_TASK based on the task type you specified in the request.
startedBy
String
Email or identifier of the user who started the job.
orgId
String
Organization identifier to which the job belongs.
runtimeEnv
String
The name of the runtime environment in which the job ran.
status
String
The status of the job. A job can have one of the following status:
- Up and Running. The job is running.
- Running with Warning. The job is running with a warning. This state can also occur when one or more table-specific subtasks fail but some subtasks are still running.
- On Hold. The job is in a paused state while the DBMI agent is being updated.
- Stopping. The job is stopping in response to a Stop request.
- Stopped. The job was intentionally stopped.
- Failed. The job ended abnormally, the task deployment to the job failed, or one or more table-specific subtasks failed. Also, for an initial load job, the job was stopped.
- Deploying. The job is being deployed.
- Deployed. The job has been deployed.
- Aborting. The job is stopping immediately in response to an Abort request.
- Aborted. The job has been aborted.
- Undeploying. The job is being undeployed.
- Undeployed. The job has been undeployed.
- Completed. The job completed successfully.
durationInSeconds
Long
Duration of the job execution in seconds.
endTime
String (ISO 8601)
Timestamp when the job ended.
location
String
The project or project folder that contains the task definition.
startTime
String (ISO 8601)
Timestamp when the job started.
errorMessage
String or null
Error message if the job failed. Null if no error.
lastAction
String
Description of the last action performed on the job. For example, Job was started by a user.
jobConfig
Object (JobInfoExtraData)
Configuration details related to the task associated with the job.
The jobConfig parameter includes the following information:
Field
Type
Description
taskId
String
Unique identifier of the task associated with the job.
taskMode
String (Enum)
Represents the type of load operation performed. Options are:
- UNLOAD. Loads source data read at a single point in time to a target. After the data is loaded, the ingestion and replication job ends.
- CDC. Loads data changes continuously or until the ingestion and replication job is stopped or ends. The job loads the changes that have occurred since the last time it ran or from a specific start point.
- COMBINED. Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
srcConnId
String
Source connection identifier used in the task.
tgtConnId
String
Target connection identifier used in the task.
deployTime
String (ISO 8601)
Timestamp when the task was deployed.
schedulerJobName
String or null
Name of the scheduler job if applicable. Null if not scheduled.
schedulerId
String
Identifier for the scheduler. Returns empty string if not applicable.
agentId
String
Identifier of the agent handling the job deployment. Returns empty if not assigned.
isRunOnOtherAgentEnabled
Boolean
Flag that indicates if the job can run on other agents.
isServerless
Boolean or null
Indicates if the job runs in a serverless environment.
deployVersion
String
Version of the deployment. For example, 58.0.0-SNAPSHOT.
featureTags
List or null
List of feature tags associated with the job. Returns Null if none.
srcVendor
String or null
Vendor or source system type. For example, WORKDAY.
tgtVendor
String or null
Vendor or target system type. For example, SNOWFLAKE.
cdcGroupJobId
Long or null
CDC group job ID. Returns this parameter only for CDC and combined jobs.
cdcGroupInmdtStrgConId
String or null
CDC group intermediate storage connection ID. Returns this parameters only for CDC and combined jobs.
mapOfApplyJobDetails
Map<String, Object>
Map of apply jobs in a CDC staging job and jobs for migrating jobs to a CDC staging group.
Success response example
A successful response returns details about the job status similar to the following example.
Get statistics for a database or application ingestion and replication job
Use the metric action of the job resource to retrieve detailed statistics for a database ingestion and replication job or application ingestion and replication job.
POST request
Use the following POST request to retrieve detailed job statistics:
POST /dbmi/public/api/v2/job/metrics
Include the following required attributes in the request body:
Field
Type
Description
jobId
integer
The ID of the job for which you want to retrieve detailed statistics.
parameters
Map<String,json>
Map containing the metricsOptions and their values.
Include the following required metricsOptions attributes:
Field
Type
Description
stateFilter
string
Filter for the state of the objects for which you want to retrieve metrics. If you specify null, no filtering is applied.
sort
array
Defines the sorting order of the results. For example, to sort by "srcTable" in ascending ("asc") order, pass the following attribute:"
"sort": [ "srcTable", "asc" ],
search
string
Search string to filter objects by name or other searchable fields. Empty string means no filter applied.
offset
number
Maximum number of objects to return in one response or page size. For example, if you set the offset to 25, the request can returns metrics for up to 25 objects in one response.
limit
number
Number of objects to skip from the beginning for pagination. For example, 0 means start from the first object.
Request body
Use the following request body to pass the job ID and metrics options:
A successful response includes the following detailed statistics about the job and its tasks:
Parameter
Type
Description
jobId
Long
Unique identifier of the job.
assetId
String
Asset identifier related to the job.
jobName
String
Name of the job.
assetType
String
Type or category of the asset. The asset type can be "APPMI_TASK" or "DBMI_TASK" based on the task type you specified.
startedBy
String
Email or identifier of the user who started the job.
correlationId
String
Correlation identifier for tracking related jobs or processes.
orgId
String
Organization identifier to which the job belongs.
runtimeEnv
String
The name of the runtime environment in which the job ran.
status
String
The status of the job. Options are:
- Up and Running. The job is running.
- Running with Warning. The job is running with a warning. This state can also occur when one or more table-specific subtasks fail but some subtasks are still running.
- On Hold. The job is in a paused state while the Database Ingestion and Replication (DBMI) agent is being updated.
- Stopping. The job is stopping in response to a Stop request.
- Stopped. The job was intentionally stopped.
- Failed. The job ended abnormally, the task deployment to the job failed, or one or more table-specific subtasks failed. Also, for an initial load job, the job was stopped.
- Deploying. The job is being deployed.
- Deployed. The job has been deployed.
- Aborting. The job is stopping immediately in response to an Abort request.
- Aborted. The job has been aborted.
- Undeploying. The job is being undeployed.
- Undeployed. The job has been undeployed.
- Completed. The job completed successfully.
duration
Long
Duration of the job execution, in milliseconds or seconds.
endTime
String (ISO 8601 timestamp)
Timestamp when the job ended.
startTime
String (ISO 8601 timestamp)
Timestamp when the job started.
location
String
The project or folder that contains the task definition.
jobConfig
Object
Configuration details related to the job task.
metricsInfo
List
List of metrics for each task associated with the job.
counts
Object
Summary counts of subtasks by status, including aggregate unload totals for the job.
currentThroughput
Double
Current throughput of the job, in records per second.
The jobConfig parameter includes the following information:
Parameter
Type
Description
taskId
String
Unique identifier of the task associated with the job.
taskMode
String (Enum)
Represents the type of load operation performed.
Options are:
- UNLOAD. Loads source data read at a single point in time to a target. After the data is loaded, the ingestion and replication job ends.
- CDC. Loads data changes continuously or until the ingestion and replication job is stopped or ends. The job loads the changes that have occurred since the last time it ran or from a specific start point.
- COMBINED. Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
srcConnId
String
Source connection identifier used in the task.
tgtConnId
String
Target connection identifier used in the task.
deployTime
String (ISO 8601)
Timestamp when the task was deployed.
schedulerJobName
String or null
Name of the scheduler job if applicable. Null if not scheduled.
schedulerId
String
Identifier for the scheduler. Returns empty string if not applicable.
agentId
String
Identifier of the agent handling the job deployment. Returns empty if not assigned.
isRunOnOtherAgentEnabled
Boolean
Flag that indicates if the job can run on other agents.
isServerless
Boolean or null
Indicates if the job runs on a serverless environment.
deployVersion
String
Version of the deployment. For example, 58.0.0-SNAPSHOT.
featureTags
List or null
List of feature tags associated with the job. Returns Null if none.
srcVendor
String or null
Vendor or source system name. For example, ORACLE.
tgtVendor
String or null
Vendor or target system name. For example, SNOWFLAKE.
cdcGroupJobId
Long or null
The CDC group job ID. Returns this parameter for database ingestion and replication CDC staging tasks.
cdcGroupInmdtStrgConId
String or null
CDC group intermediate storage connection ID. Returns this parameter for database ingestion and replication CDC staging tasks.
mapOfApplyJobDetails
Map<String, Object>
Map of apply jobs in database ingestion and replication CDC staging jobs and for migrating jobs to a CDC staging group.
The metricsInfo parameter includes the following information:
Parameter
Type
Description
jobName
String
Name of the job.
runId
String
Run identifier for the task execution.
recordsRead
Long
Number of records read during the task.
recordsWritten
Long
Number of records written during the task.
captureProgress
Object or null
Optional progress capture object. Null if not applicable.
duration
Long
Duration of the task execution, usually in milliseconds.
subTasks
List
List of subtasks metrics related to this task.
datavalidation
List
Data validation results for the task.
For an initial load task, the subTasks parameter includes the following information:
Parameter
Type
Description
srcName
String
Name of the source object for the subtask.
tgtName
String
Name of the target object for the subtask.
state
String
Current state of the subtask. For example, COMPLETED.
agentId
String
Identifier of the agent that processed the subtask.
For a CDC task, the subTasks parameter includes the following information:
Parameter
Type
Description
srcName
String
Name of the source object for the subtask.
tgtName
String
Name of the target object for the subtask.
state
String
Current state of the subtask. For example, RUNNING.
inserts
Long
Number of insert operations applied for the subtask.
updates
Long
Number of update operations applied for the subtask.
deletes
Long
Number of delete operations applied for the subtask.
lobs
Long
Number of LOB (large object) values processed for the subtask.
unloadCount
Long
Number of records unloaded for the subtask during an initial load phase.
agentId
String
Identifier of the agent that processed the subtask.
The counts parameter includes the following information:
Parameter
Type
Description
total
Integer
Total number of subtasks or objects counted.
statusCounts
List
List of status and count pairs. For example, {"status": "COMPLETED", "counts": 1}.
totalUnload
Object
Aggregate record totals for the job.
totalUnload.totalRead
Long
Total number of records read across all subtasks.
totalUnload.totalWritten
Long
Total number of records written across all subtasks.
Sample response for initial load metrics
The following sample response shows the returned initial load metrics:
If the request fails, the API might return one of the following HTTP error codes:
HTTP status code
Error code
Error message
400
CDIR_VALIDATION_MISSING_FIELD
Required parameter {field} is missing.
400
CDIR_VALIDATION_ INVALID_VALUE
Value for {field} is invalid.
401
CDIR_AUTH_UNAUTHORIZED
Authentication failed. Access token is missing or expired.
403
CDIR_AUTHZ_FORBIDDEN
You do not have the required permissions for this request.
404
CDIR_JOB_NOT_FOUND
Job {jobId} was not found.
400
CDIR_VALIDATION_MISSING_METRICSOPTIONS
Required parameter metricsOptions is missing for job.
500
CDIR_INTERNAL_ERROR
An unexpected internal error occurred with this job.
503
CDIR_SERVICE_UNAVAILABLE
Service is temporarily unavailable.
The following example shows a sample error response when a request is forbidden:
{ "error": { "httpStatus": 403, "code": "CDIR_AUTHZ_FORBIDDEN", "message": "You do not have the required permissions for this request.", "details": [ { "jobId": "137275" } ], "traceId": "8d6e7acc-6d16-4299-93dc-72590df5293d" } }
Get database or application ingestion and replication task details
Use the details action of the task resource to retrieve detailed information about a database ingestion and replication task or application ingestion and replication task in the organization. This API provides detailed task payload information, which you can easily reuse when you create or update tasks.
GET request
Use the following GET request to retrieve task details:
GET /dbmi/public/api/v2/task/details?taskId={taskId}&projectId={projectId}&folderId={folderId}
Request parameters
You can include the task ID in the URL to retrieve details for a specific task. Alternatively, provide the project ID and folder ID in the URL to retrieve details for all tasks within a particular project and folder. If you omit the task ID, project ID, and folder ID, the API returns details for all tasks in the organization.
Field
Type
Description
projectId
String
ID of the project where the task resides.
folderId
String
ID of the folder where the task resides in the project.
taskId
integer
ID of the task.
You can also include the following optional pagination attributes in the request:
Field
Type
Description
orderBy
string
Column name to sort the results.
You can use one of the following column names to specify the sorting order:
- name
- createdTime
- lastUpdatedTime
- lastAccessedTime
pageNo
Integer
Page number to retrieve after pagination.
pageSize
Integer
Number of records per page.
Get request
Without pagination
Use the following GET request to retrieve task details based on task ID, project ID, and folder ID without pagination:
GET /dbmi/public/api/v2/task/details?taskId={taskId}&projectId={projectId}&folderId={folderId}
For example:
GET https://pod-ing.ics.dev:11447/dbmi/public/api/v2/task/details?taskId=2&folderId=5GX14m3CuFAeVWqnsSX9mC
With pagination
To paginate the results, you can include the following parameters in your GET request:
GET /dbmi/public/api/v2/task/details?orderBy={nameOftheColumn}&pageNo={int}&pageSize={int}
For example:
GET https://pod-ing.ics.dev:11447/dbmi/public/api/v2/task/details?orderBy=name&pageNo=1&pageSize=2
Successful response
A successful response returns HTTP status code 200 and includes the following header:
Content-Type: application/json HTTP/2 200 Created
Response fields
The response includes the task details with the following fields:
Field
Type
Description
totalCount
int
Total number of documents returned in the response.
Documents
Object List
List of document objects, each representing a task or asset.
Each document object includes the following fields:
Field
Type
Description
taskId
int
Identifier of the task, retrieved from the repoInfo.repoHandle.
name
string
Name of the document or task.
documentState
string
Current state of the document, such as VALID.
documentType
string
Type of the document. For example, APPMI_TASK or DBMI_TASK.
parentInfo
Object List
List of parent entities related to the document, such as Space, Project, or Folder.
description
string
Description of the document. This field can be empty.
owner
string
Identifier of the owner of the document.
createdBy
string
Identifier of the user who created the document.
createdTime
string (ISO 8601)
Timestamp indicating when the document was created.
accessedTime
string (ISO 8601)
Timestamp indicating when the document was last accessed.
updateTime
string (ISO 8601)
Timestamp indicating when the document was last updated.
dbmiTask
JSON Node
Detailed JSON object containing task-specific configuration and metadata.
nativeData
NativeData
Optional native data associated with the document. The value can be null.
contentType
string
Content type of the document, which is ignored during JSON serialization.
docRef
DocRef
Reference to the document in the repository. The value can be null.
repoInfo
RepositoryInfo
Repository-related information including the repository handle. The value can be null.
The dbmiTask object includes the following top-level sections:
Field
Type
Description
type
string
Type of the task object, for example dbmi.
general
Object
General task settings, such as name, description, location, runtime environment, and load type.
source
Object
Source connection and configuration details for the task, including connector-specific custom properties.
target
Object
Target connection and configuration details for the task, including connector-specific custom properties.
runtimeOptions
Object
Runtime behavior settings for the task, such as checkpointing, auto-tuning, and schema drift options.
replicationRules
Object
Source and target object selection rules and column transformations applied by the task.
The ParentInfo object includes the following fields:
Field
Type
Description
parentId
string
Identifier of the parent entity, such as a Space, Project, or Folder.
parentName
string
Name of the parent entity.
parentType
string
Type of the parent entity, for example, Space, Project, or Folder.
Sample response without pagination
The following example shows detailed information returned for a single task without pagination: