The sources that Application Ingestion and Replication support depend on whether the application ingestion and replication tasks transfer a point-in-time snapshot of data in a batch initial load operation or load incremental change data from a specific start point.
The following table lists the source types that Application Ingestion and Replication support along with the types of load operations supported for each source type:
Source type
Supported load operations
Adobe Analytics
Initial load, incremental load, and combined initial and incremental load
Google Analytics
Initial load, incremental load, and combined initial and incremental load
Marketo
Initial load, incremental load, and combined initial and incremental load
Microsoft Dynamics 365
Initial load, incremental load, and combined initial and incremental load
NetSuite
Initial load, incremental load, and combined initial and incremental load
Oracle Fusion Cloud Applications
REST: Initial load, incremental load, and combined initial and incremental load
BICC: Initial load, incremental load, and combined initial and incremental load
Salesforce
Initial load, incremental load, and combined initial and incremental load
Salesforce Marketing Cloud
Initial load
SAP ERP Central Component (SAP ECC)
Initial load, incremental load, and combined initial and incremental load depending on the connection type.
SAP ODP
SAP ECC: Initial load, incremental load, and combined initial and incremental load
SAP S4/HANA: Initial load, incremental load, and combined initial and incremental load
SAP Mass Ingestion
SAP ECC: Initial load, incremental load, and combined initial and incremental load for Oracle database. Certified on Snowflake target.
SAP S4/HANA: Initial load and incremental load for HANA database. Certified on Snowflake target.
ServiceNow
Initial load, incremental load, and combined initial and incremental load
Workday
Initial load, incremental load, and combined initial and incremental load
Zendesk
Initial load, incremental load, and combined initial and incremental load
To determine the connectors to use for the source types, see Connectors and Connections > Applications Ingestion and Replication connectors.
Guidelines for Google Analytics sources
Consider the following guidelines when you use Google Analytics sources:
•The first time an incremental load job runs to capture change data for a Google Analytics source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•In Application Ingestion and Replication ingestion initial load jobs, the Google Analytics report description is the input data. Each report contains a list of dimension and metrics that replicates to the target. There are 535 columns including dimensions and metrics. However, Google Analytics can fetch a maximum of 10 metrics and 9 dimensions in a single report. You can configure multiple reports to view all dimensions & metrics.
•Before you configure an application ingestion and replication task, you must create the report description document and save it in a location that is accessible by the Secure Agent. You should have read-permission for the file.
•Google Analytics fetches data only for days and not for a given interval that is represented as a timestamp.
•Consider the following guidelines while updating the reports description document for Application Ingestion and Replication:
- Don't change the order of the field while renaming it in the report. You must overwrite the new field name.
- While adding new fields to the report, add the fields at the end of fields' list of the report.
- To delete fields, remove the corresponding field names from the fields' list of the report.
Guidelines for Marketo sources
Consider the following guidelines when you use Marketo sources:
•The first time an incremental load job runs to capture change data for a Marketo source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•Incremental load jobs configured for Marketo sources can replicate only insert and update operations performed on the source objects.
•Application Ingestion and Replication does not replicate the relationships defined between Marketo objects. The target does not contain the relationships that exists between the objects on the source.
•By default, Marketo instances are limited to 500 MB of downloaded data per day. Additionally, bulk API downloads are limited to a window of 31 day period.
•Some Marketo objects do not allow APIs to directly retrieve the data stored in them. To retrieve the data from such objects, APIs require certain filters as input parameters. The filters that the APIs use are fields of other Marketo objects. Application Ingestion and Replication does not replicate the data stored in such Marketo objects that are dependent on other objects.
•Application ingestion and replication jobs do not propagate the data stored in custom objects that are not linked to any lead object. Additionally, application ingestion and replication jobs do not propagate the data stored in custom object fields with the display name ID.
•By default, Application Ingestion and Replication can capture change data for the following Marketo objects:
- Campaign
- Email Templates
- Folder
- Landing Pages
- Landing Page Templates
- Program
- Segmentations
- Smart Campaign
- Smart Lists
- Snippets
- Static Lists
Note: To capture change data for the Campaign, Email Templates, Folder, Landing Pages, Landing Page Templates, Segmentations, and Snippets objects, you must set the marketo.include.additional.cdc.objects custom property to TRUE. You can set the custom property on the Source tab of the application ingestion and replication task wizard.
Guidelines for Microsoft Dynamics 365 sources
Consider the following guidelines when you use Microsoft Dynamics 365 sources:
•The first time an incremental load job runs to capture change data for a Microsoft Dynamics 365 source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•Incremental load jobs for Microsoft Dynamics 365 sources don't capture the change data for objects without the modifiedon field.
•Incremental load jobs and combined initial and incremental load jobs for Microsoft Dynamics 365 sources don't detect and replicate the changes that are made to the source schema.
•Incremental load jobs and combined initial and incremental load jobs for Microsoft Dynamics 365 sources don't capture the DELETE data operation for a CDC cycle.
•If a source object contains locale ID and timezonecode values, Application Ingestion and Replication propagates the values to the target without converting them to the actual values that they represent.
•Application Ingestion and Replication does not propagate fields of the following data types:
- CUSTOMERTYPE
- OWNERTYPE
- PARTYLISTTYPE
- VIRTUALTYPE
- CALENDARRULESTYPE
- UNKNOWN
Guidelines for NetSuite sources
Consider the following guidelines when you use NetSuite sources:
•The first time an incremental load job runs to capture change data for a NetSuite source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•Incremental load jobs configured for NetSuite sources do not capture change data for the tables that do not contain a primary key.
•Incremental load jobs do not capture change data for the tables that do not contain at least one of the following timestamps:
- The timestamp of when its records were created.
- The timestamp of the last time its records were modified.
Note: For the tables that contain only the timestamp of when its records were created, incremental load jobs capture only the insert operations performed on the records.
•When an application ingestion and replication job retrieves a document or image from the source, the job propagates the unique identifier of the file cabinet that contains the document or image instead of propagating the binary content of the file.
•Application Ingestion and Replication does not capture the deletion of records from the source tables.
•Incremental load jobs and combined initial and incremental load jobs configured for NetSuite sources do not detect and replicate the deletion and renaming of source fields.
•If a NetSuite Mass Ingestion connection configured for a NetSuite.com data source is already used in an application ingestion and replication task and if the task is in any status other than Undeployed, you must not modify the connection to access a NetSuite2.com data source. Instead of modifying the existing connection, create a new connection for the NetSuite2.com data source and then configure a new application ingestion and replication task using the new connection.
•Application Ingestion and Replication does not propagate data stored in the following NetSuite tables:
- Account_activity
- Account_period_activity
- Case_types
- Case_origins
- CompanyAddressbook
- CompanyAddressbookEntityAddress
- Deleted_records
- MAP_wave_custbody17
- MAP_wave_custbody_cseg4
- MAP_wave_custbody_csegcs_multsel
Guidelines for Oracle Fusion Cloud sources
Consider the following guidelines when you use Oracle Fusion Cloud sources:
•Application Ingestion and Replication can replicate data of the following Oracle Fusion Cloud Applications Suite modules and applications:
Module
Application
Enterprise Resource Planning (ERP)
Cloud Financials
Procurement
Project Management
Risk Management and Compliance
Human Capital Management (HCM)
Human Resources
Industry
Higher Education
Public Sector
Digital Experience for Communications
CX for Consumer Goods
CX for Financial Services
Sales
Loyalty
Partner Relationship Management
Customer Data Management
Sales Automation
Subscription Management
Service
B2B Service
Supply Chain & Manufacturing (SCM)
AI Apps
Cloud Service Logistics
Fusion Cloud Inventory Management
Maintenance
Manufacturing
Order Management
Product Lifecycle Management
Supply Chain Collaboration
Supply Chain Planning
•Application Ingestion and Replication uses Business Intelligence Cloud Connector (BICC) or REST approach to transfer data from the source to the target.
•Incremental load jobs and combined initial load and incremental load jobs configured for Oracle Fusion Cloud sources do not capture and replicate the changes made to the source schema.
•Enable the Support Incremental option in the BICC console for the datastore to support the incremental load and combined initial and incremental load types. To enable the Support Incremental option, at least one of the columns in the datastore with TIMESTAMP or DATE data type must be marked as either Incremental Filter or Is Effective Start Date.
Note: If more than one column is marked as Incremental Filter or Is Effective Start Date in the table, it might result in inaccurate data extraction due to limitations in Oracle.
•By default, incremental load jobs capture change data only for the objects that contain the LASTUPDATEDATE or CREATIONDATE field. However, some objects in Oracle Fusion Cloud (REST) contain only the CREATIONDATE field instead of the LASTUPDATEDATE field. To capture the change data for such objects that contain only the CREATIONDATE field, set the oraclefusion.cdc.allow.creationdate.objects custom property to true on the Source tab of the application ingestion and replication task wizard. If you set the custom property to true for a deployed task, make sure that you undeploy the job, and then deploy the task again for the change to take effect.
If an object has only the CREATIONDATE field, only insertions are recorded, and updates and deletions are ignored as the LASTUPDATEDATE field isn't available.
Guidelines for Salesforce sources
Consider the following guidelines when you use Salesforce sources:
•The first time an incremental load job runs to capture change data for a Salesforce source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•By default, incremental load jobs capture change data only for the objects that contain the lastmodifieddate or systemModstamp field. However, some objects in Salesforce contain only CreatedDate field instead of lastmodifieddate and systemModstamp fields. To capture the change data for such objects that contain only the CreatedDate field, set the salesforce.cdc.allow.createddate.objects custom property to true on the Source tab of the application ingestion and replication task wizard. If you set the custom property to true for a deployed task, make sure that you undeploy the job and then deploy the task again for the change to take effect.
•If you want to replicate Data.com objects, your Salesforce account must be assigned with the appropriate Data.com license.
•Application Ingestion and Replication does not capture the relationships defined between Salesforce objects. The target does not contain the relationships between the source objects.
•If an object contains a compound field, Application Ingestion and Replication propagates the component fields that make up the compound field instead of propagating the compound field. For example, if the compound field geolocation is a combination of fields geolocation_longitude and geolocation_lattitude, Application Ingestion and Replication separately loads the data stored in the geolocation_longitude and geolocation_lattitude fields to the target instead of propagating the compound field geolocation.
•The masking configuration of source fields is retained on the target.
•All incremental load activities use only the standard REST API.
•The lookup fields are represented on the target by their unique alphanumeric identifiers.
•When a new field with a default value of the Function data type is added to an existing source object, Application Ingestion and Replication does not propagate the default value to any existing row on the target. However, when new rows are added to the object, the incremental jobs propagate the values stored in the field for the newly added rows.
•Application ingestion and replication jobs do not replicate the data stored in big objects that are not indexed on the Salesforce platform. Before you deploy an application ingestion and replication task for a Salesforce source, ensure that you index all the big objects which are selected by the object selection rules in the task.
•If a source field of Decimal data type contains a value whose scale exceeds the maximum limit defined for the field, application ingestion and replication jobs might fail to propagate such fields. However, you can trim the scale of the value to the defined limit while propagating the field. You can trim the scale of the value by setting the custom property salesforce.decimal.trim.scale to true.
Guidelines for Salesforce Marketing Cloud sources
Consider the following guidelines when you use Salesforce Marketing Cloud sources:
•Application Ingestion and Replication can ingest the data stored in the data extension objects of Salesforce Marketing Cloud.
•You can configure only initial load jobs for Salesforce Marketing Cloud sources.
•When an application ingestion and replication job replicates a data extension object on the target, the job prepends DATA_EXTENSION__ to the name of the object. For example, if an application ingestion and replication job replicates the data extension object Address, the name of the object on the target is DATA_EXTENSION__Address.
•An application ingestion and replication initial load job with Salesforce Marketing Cloud source and Microsoft Azure Synapse Analytics target might fail to deploy for some objects.
Guidelines for SAP ECC and SAP S4/HANA sources using the SAP ODP Extractor connector
Consider the following guidelines when you use SAP ECC or SAP S4/HANA sources with SAP ODP Extractor connector:
•When you use an SAP source for the first time, perform the following steps before you configure an application ingestion and replication task for the source:
1Verify that the appropriate SAP Notes are available in the SAP server. For more information, see the SAP Connector Guide.
2Download the SAP Java Connector (SAP JCo) library from the SAP website.
3Copy the native and JAR files from the downloaded SAP JCo library to the following directory:
4Copy the sap-adapter-common.jar file from <Secure_Agent>\downloads\package-ICSAgent_<version>\package\ICS\main\bin\rdtm\javalib\sap to the following directory:
•Before you configure an application ingestion and replication task for an SAP source, ensure that you release the data sources from which you want to transfer data. You can use the following functions to release the required data sources:
- BS_ANLY_DS_RELEASE_ODP
- RODPS_OS_EXPOSE
•The first time an incremental load job runs to capture change data for an SAP source, the job retrieves and loads the change records from the latest position in the data stream that contains source data. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•Incremental load jobs for SAP sources do not capture the change data for data sources that do not have a primary key. Additionally, the incremental load jobs do not capture change data for the following SAP data sources:
- The data sources for which SAP does not support delta.
- The data sources that are configured to provide change data in the form of additive images.
•When an application ingestion and replication task encounters NULL values in the SAP table source column, it does not replicate NULL with blank data to the target table.
•Application Ingestion and Replication does not detect and propagate schema changes in SAP sources.
•Application Ingestion and Replication does not propagate fields of the following obsolete SAP data types:
- D16S
- D34S
- PREC
- VARC
Additionally, application ingestion and replication jobs do not propagate fields of the RSTR data type.
Configuring SAP User Authorization
Configure the SAP user account to process SAP table data.
The following table describes the required authorization to read from SAP tables:
Read Object Name
Required Authorization
S_BTCH_JOB
DELE, LIST, PLAN, SHOW.
Set Job Operation to RELE.
S_PROGRAM
BTCSUBMIT, SUBMIT
S_RFC
SYST, SDTX, SDIFRUNTIME, /INFADI/TBLRDR, RFC1
S_TABU_DIS / S_TABU_NUM
Provide SAP table name from which you want to read data.
Installing SAP ODP Connector Transport Files
Install the SAP ODP Extractor transport files on the SAP machines that you want to access. Before you install the transports on your production system, install and test the transports in a development system.
Verify that you install the latest transport files to extract data from the SAP ODP object.
Install the following data file and cofile to read data from the SAP ODP object:
Data and Cofile Names
Transport Request
Functionality
- K900426.IN7
- R900426.IN7
IN7K900426
Install the transports only when you want to read from an SAP ODP object that supports hierarchy.
You can use the SAP ODP Extractor Connector without installing the SAP ODP Extractor transport files for objects that do not support hierarchy.
Installing Transport Files
Install the latest transport files from a Secure Agent directory to read from a Unicode SAP system. The transport files are for SAP version ERP 6.0 EHP7 system or later.
1 Find the transport files in the following directory on the Secure Agent machine:
2Copy the cofile transport file to the Cofile directory in the SAP transport management directory on each SAP machine that you want to access.
The cofile transport file uses the following naming convention: <number>.<sap system>.
3Copy the data transport file to the Data directory in the SAP transport management directory on each SAP machine that you want to access.
The data transport file uses the following naming convention: <number>.<sap-system>.
4To import the transports to SAP, in the STMS, click Extras > Other Requests > Add and add the transport request to the system queue.
5In the Add Transport Request to Import Queue dialog box, enter the request number for the cofile transport.
The request number inverts the order of the renamed cofile as follows: <sap-system><number>.
6In the Request area of the import queue, select the transport request number that you added, and click Import.
7If you are upgrading from a previous version of the Informatica Transports, select the Overwrite Originals option.
Guidelines for SAP sources using the SAP Mass Ingestion connector
The SAP Mass Ingestion connector supports the following databases:
•SAP ECC (Oracle database)
•SAP S/4 HANA (Trigger based)
Source preparation
To use underlying Oracle database in application ingestion and replication tasks, first prepare the source database and review the usage considerations
•Define the following system environment variables on the Linux or Windows system where the Secure Agent runs:
- ORACLE_HOME environment variable. Points to the Oracle client installation directory, one level up from the bin directory on Windows or lib directory on Linux. This environment variable is not required. However, if you do not define it, you'll need to specify the full Oracle client installation path when you define other environment variables or Secure Agent properties that specify a path to a subdirectory.
- TNS_ADMIN environment variable. If you specify a TNS name in the Database Connect String property of the Oracle Database Ingestion connection properties, use this environment variable to point to the directory location of the tsnnames.ora file when the file is not in the default $ORACLE_HOME/network/admin directory. The tnsnames.ora file, along with the Oracle Call Interface (OCI), is used to communicate with the Oracle source database.
Note: In Administrator, you can set the ociPath property for the Database Ingestion agent service (DBMI agent) to point to the OCI library that contains the oci.dll or libcIntsh.so file. The OCI library is used by application ingestion and replication CDC tasks to connect to Oracle. By default, Oracle uses the ociPath value of $ORACLE_HOME/lib on Linux or %ORACLE_HOME%\bin on Windows.
•Make sure the Application Ingestion and Replication user has the Oracle privileges that are required for the application ingestion and replication load type to be performed.
•Application ingestion and replication jobs that use log-based CDC require read access to Oracle online and archive redo logs to read incremental change data. If the redo logs are remote from the on-premises system where the Secure Agent runs, make sure that read access to the logs is provided, for example, by using Oracle Automatic Storage Management (ASM), mounting the logs to a network file system (NFS).
•If you use Oracle 11.2.04, set the Oracle COMPATIBLE initialization parameter to 11.2.04 to ensure that Oracle has all of the most current redo logs fixes for that release.
•Ensure that the Oracle Database Client or Instant Client is installed and configured on the Secure Agent server for the Secure Agent to communicate with Oracle. If you do not already have an Oracle client installed, you can download a client and access installation information from the Oracle web site, or ask your Oracle DBA to download and configure an Oracle client.
•For incremental load or combined initial and incremental load operations that use log-based CDC, perform the following prerequisite tasks in Oracle:
- Enable ARCHIVELOG mode for the Oracle database. If the database is not in an Amazon RDS environment, issue the following SQL statements:
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE ARCHIVELOG; ALTER DATABASE OPEN; SHUTDOWN IMMEDIATE; STARTUP;
- Enable Oracle minimal global supplemental logging on the source database.
Usage considerations for SAP ECC (Oracle database)
•For incremental load operations with Oracle sources, Application Ingestion and Replication provides alternative capture methods for capturing change data from the source and applying the data to a target. Application Ingestion and Replication reads data changes from Oracle redo logs. This method requires users to have extended privileges.
•Because Application Ingestion and Replication expects each source table row to be unique, Informatica recommends that each source table have a primary key. Application Ingestion and Replication does not honor unique indexes in place of a primary key. If no primary key is specified, Application Ingestion and Replication treats all columns as if they are part of the primary key.
•If Oracle source CHAR or VARCHAR columns contain nulls, the application ingestion and replication job does not delimit the null values with double-quotation (") marks or any other delimiter when writing data to a Amazon S3, Flat File, Microsoft Azure Data Lake, or Microsoft Azure Synapse Analytics target.
•Application Ingestion and Replication can process data across a RESETLOGS boundary. To avoid the source and targets becoming out of sync, Informatica recommends that you stop capture processing before performing a RESETLOGS and then restart capture processing after the RESETLOGS event. Otherwise, the capture process might send data to the target that is subsequently reverted by the RESETLOGS event, causing the source and target to become out of sync.
•Alternative strategies for accessing the Oracle redo logs are available.
•If an application ingestion and replication incremental load or combined initial and incremental load task contains an Oracle source table name or one or more column names that are longer than 30 characters, Oracle suppresses supplemental logging for the entire table, including primary keys and foreign keys. As a result, most operations on the table fail. This problem is caused by an Oracle restriction. In this situation, exclude the table from capture processing or rename the long table and column names to names of 30 characters or less.
•Do not use custom data type mapping rules to map an Oracle source RAW column to a target CHAR or VARCHAR column. Otherwise, the deployment of the application ingestion and replication task might fail.
•Application Ingestion and Replication does not support invisible columns in Oracle source columns, regardless of the target type. For these columns, application ingestion and replication incremental load jobs and combined initial and incremental load jobs propagate nulls to the corresponding target columns.
•Application ingestion and replication combined initial and incremental load jobs that have Oracle sources can read changes from copies of the archive redo logs. You must set the Reader Mode property to ARCHIVECOPY in the Oracle Database Ingestion connection properties and also set the source custom property pwx.cdcreader.oracle.reader.additional with the dir and file parameters. The dir parameter points to the name of the base directory that the CDC log reader scans for the archive log copies, and the file parameter specifies a mask that is used to filter the log copies.
Installing transport files
Install the latest transport files from a Secure Agent directory to read from a Unicode SAP system. The transport files are for SAP version ECC 5.0 or later.
1 Find the transport files in the following directory on the Secure Agent machine:
2Copy the cofile transport file to the Cofile directory in the SAP transport management directory on each SAP machine that you want to access.
The cofile transport file uses the following naming convention: K<number>.EP6.
3Copy the data transport file to the Data directory in the SAP transport management directory on each SAP machine that you want to access.
The data transport file uses the following naming convention: R<number>.EP6.
4To import the transports to SAP, in the STMS, click Extras > Other Requests > Add and add the transport request to the system queue.
5In the Add Transport Request to Import Queue dialog box, enter the request number for the cofile transport.
The request number inverts the order of the renamed cofile as follows: EP6K<number>.
For example, for a cofile transport file renamed as K900215.EP6, enter the request number as EP6K900215.
6In the Request area of the import queue, select the transport request number that you added, and click Import.
7If you are upgrading from a previous version of the Informatica Transports, select the Overwrite Originals option.
Oracle privileges
To deploy and run an application ingestion and replication task that has an Oracle source, the source connection must specify a Application Ingestion and Replication user who has the privileges required for the ingestion load type.
Privileges for incremental load processing with log-based CDC
Note: If the Oracle logs are managed by ASM, the user must have SYSASM or SYSDBA authority.
For an application ingestion and replication task that performs an incremental load or combined initial and incremental load using the log-based CDC method, ensure that the Application Ingestion and Replication user (cmid_user) has been granted the following privileges:
GRANT CREATE SESSION TO <cmid_user>;
GRANT SELECT ON table TO <cmid_user>; -- For each source table created by user GRANT EXECUTE ON DBMS_FLASHBACK TO <cmid_user>;
-- The following grant is required for combined initial and incremental loads only. Do not -- use ANY TABLE unless your security policy allows it. GRANT FLASHBACK ON table|ANY TABLE TO <cmid_user>;
-- Include the following grant only if you want to Execute the CDC script for enabling -- supplemental logging from the user interface. If you manually enable supplemental -- logging, this grant is not needed. GRANT ALTER table|ANY TABLE TO <cmid_user>;
GRANT SELECT ON DBA_CONSTRAINTS TO <cmid_user>; GRANT SELECT ON DBA_CONS_COLUMNS TO <cmid_user>; GRANT SELECT ON DBA_INDEXES TO <cmid_user>; GRANT SELECT ON DBA_LOG_GROUPS TO <cmid_user>; GRANT SELECT ON DBA_LOG_GROUP_COLUMNS TO <cmid_user>; GRANT SELECT ON DBA_OBJECTS TO <cmid_user>; GRANT SELECT ON DBA_OBJECT_TABLES TO <cmid_user>; GRANT SELECT ON DBA_TABLES TO <cmid_user>; GRANT SELECT ON DBA_TABLESPACES TO <cmid_user>; GRANT SELECT ON DBA_USERS TO <cmid_user>;
GRANT SELECT ON "PUBLIC".V$ARCHIVED_LOG TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$CONTAINERS TO <cmid_user>; -- For Oracle multitenant environments GRANT SELECT ON "PUBLIC".V$DATABASE TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$DATABASE_INCARNATION TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$ENCRYPTION_WALLET TO <cmid_user>; -- For Oracle TDE access GRANT SELECT ON "PUBLIC".V$LOG TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$LOGFILE TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$PARAMETER TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$PDBS TO <cmid_user>; -- For Oracle multitenant environments GRANT SELECT ON "PUBLIC".V$SPPARAMETER TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$STANDBY_LOG TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$THREAD TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$TRANSACTION TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$TRANSPORTABLE_PLATFORM TO <cmid_user>; GRANT SELECT ON "PUBLIC".V$VERSION TO <cmid_user>;
GRANT SELECT ON SYS.ATTRCOL$ TO <cmid_user>; GRANT SELECT ON SYS.CCOL$ TO <cmid_user>; GRANT SELECT ON SYS.CDEF$ TO <cmid_user>; GRANT SELECT ON SYS.COL$ TO <cmid_user>; GRANT SELECT ON SYS.COLTYPE$ TO <cmid_user>; GRANT SELECT ON SYS.IDNSEQ$ TO <cmid_user>; GRANT SELECT ON SYS.IND$ TO <cmid_user>; GRANT SELECT ON SYS.INDPART$ TO <cmid_user>; GRANT SELECT ON SYS.OBJ$ TO <cmid_user>; GRANT SELECT ON SYS.PARTOBJ$ TO <cmid_user>; GRANT SELECT ON SYS.RECYCLEBIN$ TO <cmid_user>; GRANT SELECT ON SYS.TAB$ TO <cmid_user>; GRANT SELECT ON SYS.TABCOMPART$ TO <cmid_user>; GRANT SELECT ON SYS.TABPART$ TO <cmid_user>; GRANT SELECT ON SYS.TABSUBPART$ TO <cmid_user>;
-- Also ensure that you have access to the following ALL_* views: ALL_CONSTRAINTS ALL_CONS_COLUMNS ALL_ENCRYPTED_COLUMNS ALL_INDEXES ALL_IND_COLUMNS ALL_OBJECTS ALL_TABLES ALL_TAB_COLS ALL_TAB_PARTITIONS ALL_USERS
Privileges for incremental load processing with query-based CDC
For an application ingestion and replication task that performs an incremental load or combined initial and incremental load using the query-based CDC method, ensure that the user has the following privileges at minimum:
GRANT CREATE SESSION TO <cmid_user>;
GRANT SELECT ON DBA_INDEXES TO <cmid_user>; GRANT SELECT ON DBA_OBJECT_TABLES TO <cmid_user>; GRANT SELECT ON DBA_OBJECTS TO cmid_user; GRANT SELECT ON DBA_TABLES TO <cmid_user>; GRANT SELECT ON DBA_USERS TO <cmid_user>; GRANT SELECT ON DBA_VIEWS TO <cmid_user>; -- Only if you unload data from views
GRANT SELECT ANY TABLE TO <cmid_user>; -or- GRANT SELECT ON table TO <cmid_user>; -- For each source table created by user
GRANT SELECT ON ALL_CONSTRAINTS TO <cmid_user>; GRANT SELECT ON ALL_CONS_COLUMNS TO <cmid_user>; GRANT SELECT ON ALL_ENCRYPTED_COLUMNS TO <cmid_user>; GRANT SELECT ON ALL_IND_COLUMNS TO <cmid_user>; GRANT SELECT ON ALL_INDEXES TO <cmid_user>; GRANT SELECT ON ALL_OBJECTS TO <cmid_user>; GRANT SELECT ON ALL_TAB_COLS TO <cmid_user>; GRANT SELECT ON ALL_USERS TO <cmid_user>;
GRANT SELECT ON "PUBLIC"."V$DATABASE" TO cmid_user; GRANT SELECT ON "PUBLIC"."V$CONTAINERS" TO cmid_user; GRANT SELECT ON SYS.ATTRCOL$ TO <cmid_user>; GRANT SELECT ON SYS.CCOL$ TO <cmid_user>; GRANT SELECT ON SYS.CDEF$ TO <cmid_user>; GRANT SELECT ON SYS.COL$ TO <cmid_user>; GRANT SELECT ON SYS.COLTYPE$ TO <cmid_user>; GRANT SELECT ON SYS.IND$ TO <cmid_user>; GRANT SELECT ON SYS.IDNSEQ$ TO cmid_user; GRANT SELECT ON SYS.OBJ$ TO <cmid_user>; GRANT SELECT ON SYS.RECYCLEBIN$ TO <cmid_user>; GRANT SELECT ON SYS.TAB$ TO <cmid_user>;
Privileges for initial load processing
For an application ingestion and replication task that performs an initial load, ensure that the user has the following privileges at minimum:
GRANT CREATE SESSION TO <cmid_user>;
GRANT SELECT ON DBA_INDEXES TO <cmid_user>; GRANT SELECT ON DBA_OBJECT_TABLES TO <cmid_user>; GRANT SELECT ON DBA_OBJECTS TO cmid_user; GRANT SELECT ON DBA_TABLES TO <cmid_user>; GRANT SELECT ON DBA_USERS TO <cmid_user>; GRANT SELECT ON DBA_VIEWS TO <cmid_user>; -- Only if you unload data from views
GRANT SELECT ANY TABLE TO <cmid_user>; -or- GRANT SELECT ON table TO <cmid_user>; -- For each source table created by user
GRANT SELECT ON ALL_CONSTRAINTS TO <cmid_user>; GRANT SELECT ON ALL_CONS_COLUMNS TO <cmid_user>; GRANT SELECT ON ALL_ENCRYPTED_COLUMNS TO <cmid_user>; GRANT SELECT ON ALL_IND_COLUMNS TO <cmid_user>; GRANT SELECT ON ALL_INDEXES TO <cmid_user>; GRANT SELECT ON ALL_OBJECTS TO <cmid_user>; GRANT SELECT ON ALL_TAB_COLS TO <cmid_user>; GRANT SELECT ON ALL_USERS TO <cmid_user>;
GRANT SELECT ON "PUBLIC"."V$DATABASE" TO cmid_user; GRANT SELECT ON "PUBLIC"."V$CONTAINERS" TO cmid_user; GRANT SELECT ON SYS.ATTRCOL$ TO <cmid_user>; GRANT SELECT ON SYS.CCOL$ TO <cmid_user>; GRANT SELECT ON SYS.CDEF$ TO <cmid_user>; GRANT SELECT ON SYS.COL$ TO <cmid_user>; GRANT SELECT ON SYS.COLTYPE$ TO <cmid_user>; GRANT SELECT ON SYS.IND$ TO <cmid_user>; GRANT SELECT ON SYS.IDNSEQ$ TO cmid_user; GRANT SELECT ON SYS.OBJ$ TO <cmid_user>; GRANT SELECT ON SYS.RECYCLEBIN$ TO <cmid_user>; GRANT SELECT ON SYS.TAB$ TO <cmid_user>;n
Oracle privileges for Amazon RDS for Oracle sources
If you have an Amazon RDS for Oracle source, you must grant certain privileges to the Application Ingestion and Replication user.
Important: You must log in to Amazon RDS under the master username to run GRANT statements and procedures.
Grant the SELECT privilege, at minimum, on objects and system tables that are required for CDC processing to the Application Ingestion and Replication user (cmid_user). Some additional grants are required in certain situations.
Use the following GRANT statements:
GRANT SELECT ON "PUBLIC"."V$ARCHIVED_LOG" TO "cmid_user";
GRANT SELECT ON "PUBLIC"."V$DATABASE" TO "cmid_user"; GRANT SELECT ON "PUBLIC"."V$LOG" TO "cmid_user"; GRANT SELECT ON "PUBLIC"."V$LOGFILE" TO "cmid_user"; GRANT SELECT ON "PUBLIC"."V$TRANSPORTABLE_PLATFORM" TO "cmid_user"; GRANT SELECT ON "PUBLIC"."V$THREAD" TO "cmid_user"; GRANT SELECT ON "PUBLIC"."V$DATABASE_INCARNATION" TO "cmid_user"; GRANT SELECT ON "PUBLIC"."V$TRANSACTION" TO "cmid_user";
GRANT SELECT ON "SYS"."DBA_CONS_COLUMNS" TO "cmid_user"; GRANT SELECT ON "SYS"."DBA_CONSTRAINTS" TO "cmid_user"; GRANT SELECT ON DBA_INDEXES TO "cmid_user"; GRANT SELECT ON "SYS"."DBA_LOG_GROUP_COLUMNS" TO "cmid_user"; GRANT SELECT ON "SYS"."DBA_TABLESPACES" TO "cmid_user";
GRANT SELECT ON "SYS"."OBJ$" TO "cmid_user"; GRANT SELECT ON "SYS"."TAB$" TO "cmid_user"; GRANT SELECT ON "SYS"."IND$" TO "cmid_user"; GRANT SELECT ON "SYS"."COL$" TO "cmid_user";
GRANT SELECT ON "SYS"."PARTOBJ$" TO "cmid_user"; GRANT SELECT ON "SYS"."TABPART$" TO "cmid_user"; GRANT SELECT ON "SYS"."TABCOMPART$" TO "cmid_user"; GRANT SELECT ON "SYS"."TABSUBPART$" TO "cmid_user"; COMMIT;
/* For combined load jobs:*/ GRANT EXECUTE ON DBMS_FLASHBACK TO "cmid_user";
/*To provide read access to the Amazon RDS online and archived redo logs:*/ GRANT READ ON DIRECTORY ONLINELOG_DIR TO "cmid_user"; GRANT READ ON DIRECTORY ARCHIVELOG_DIR TO "cmid_user";
Additionally, log in as the master user and run the following Amazon RDS procedures to grant the SELECT privilege on some more objects:
Configuring BFILE access to Oracle redo logs in the Oracle file system
If you store redo logs in the local Oracle server file system and want to access the logs by using Oracle directory objects with BFILEs, perform the following configuration tasks:
Complete the following usual Oracle source preparation tasks, which are not specific to BFILE access:
•Define the ORACLE_HOME environment variable on the Linux or Windows system where the Secure Agent runs for Application Ingestion and Replication to use the Oracle Call Interface (OCI) to communicate with the Oracle source database.
•Make sure the Application Ingestion and Replication user has the Oracle privileges that are required for the application ingestion and replication incremental load processing. For more information, see Oracle privileges.
•Enable ARCHIVELOG mode for the Oracle database.
•Define the archive log destination.
Note: For BFILE access, use a specific archive log destination directory. Do not use the Oracle USE_DB_RECOVERY_FILE_DEST keyword to store archive logs by date in a Fast Recovery Area (FRA).
•Enable Oracle minimal global supplemental logging on the source database.
•If your Oracle source tables have primary keys, ensure that supplemental logging is enabled for all primary key columns. For source tables that do not have primary keys, ensure that supplemental logging is enabled for all columns from which change data will be captured.
Note: When you create an application ingestion and replication task, you have the option of generating a script that implements supplemental logging for all columns or only primary key columns for the selected source tables.
•Ensure that the Oracle MAX_STRING_SIZE initialization parameter is not set to EXTENDED. If it is set to EXTENDED, Application Ingestion and Replication will not be able to replicate inserts and updates for tables containing columns defined with large (extended size) VARCHAR2, NVARCHAR2, or RAW columns.
Additionally, for BFILE access, perform the following steps:
1Query the Oracle database for the online and archived redo log locations in the Oracle server file system. You can use the following example queries:
To get location of the online redo logs:
select * from v$logfile;
To get the archive log destination:
select dest_id, dest_name, destination, status from V$ARCHIVE_DEST;
2Create the ONLINELOG_DIR and ARCHIVELOG_DIR directory objects that point to the locations of log files from step 1. An Oracle directory object specifies a logical alias name for a physical directory on the Oracle server file system under which the log files to be accessed are located. For example:
CREATE DIRECTORY ONLINELOG_DIR AS '/u01/oracle/data'; CREATE DIRECTORY ARCHIVELOG_DIR AS '/u01/oracle/archivedata';
Note: If you plan to set the reader mode to ARCHIVEONLY in the Oracle Database Ingestion connection to read changes only from archive logs, you do not need to create an ONLINELOG_DIR directory or directory object.
The Oracle database does not verify that the directories you specify exist. Make sure you specify valid directories that exist in the Oracle file system.
3To verify that the directory objects were created with the correct file system paths for the redo logs, issue a select statement such as:
4Grant read and write access on the ONLINELOG_DIR and ARCHIVELOG_DIR directory objects to the Application Ingestion and Replication user who is specified in the Oracle Database Ingestion connection properties. For example:
grant read on directory "ARCHIVELOG_DIR" to "cmid_user"; grant read on directory "ONLINELOG_DIR" to "cmid_user";
5In the Oracle Database Ingestion connection properties, select the BFILE Access check box.
Oracle archive log retention considerations
Application ingestion and replication incremental and combined initial and incremental load jobs must be able to access transaction data in Oracle online and archive redo logs. If the logs are not available, application ingestion and replication jobs end with an error.
Typically, the Oracle DBA sets the archive log retention period based on your oganization's particular business needs and Oracle environment. Make sure that the source archive logs are retained for the longest period for which you expect change capture to be stopped or latent, plus about 1 hour, so that the logs will be available for restart processing.
To determine if the current log retention policy in your environment is sufficient to accommodate application ingestion and replication change capture processing, consider the following factors:
•How long are Oracle transactions typically open on a source?
•What is the longest period of time that change capture is allowed to be down or latent, accounting for weekends and holidays?
•What is the replication latency from source to target?
•Do you run application ingestion and replication jobs based on a schedule? If yes, what type of schedule?
•Is the pwx.cdcreader.oracle.option.additional ageOutPeriod=minutes custom property set on the Source page of task wizard?
Note: This property specifies the age at which outstanding UOWs without change records of CDC interest are removed from the calculation of the next restart point. You can use the property to prevent CDC failures that might occur if you shut down and then restart capture processing while the transaction is outstanding and the redo log in which the UOW started is not available.
•What is the redo generation rate?
•Do you ship copies of archive logs to a secondary system?
If the archive logs are not available when you need to restart capture processing in the logs, you can ask your DBA to restore them and to modify the retention period if necessary. Otherwise, perform another initial load to re-materialize the target and then start incremental change data processing again. However, in this case, you might lose some changes.
Guidelines for ServiceNow sources
Consider the following guidelines when you use ServiceNow sources:
•The first time an incremental load job runs to capture change data for a ServiceNow source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•The data stored in some ServiceNow objects can be retrieved only if the API client is configured with the maint and nobody roles. Application Ingestion and Replication does not retrieve data from such source objects that require the ServiceNow Mass Ingestion connection to be configured with the maint and nobody roles.
•You must have Admin privileges to access table APIs or aggregate APIs.
•Incremental load jobs configured for ServiceNow sources do not capture change data for the objects that do not contain the sys_updated_on field.
•Incremental load jobs and combined initial and incremental load jobs configured for ServiceNow sources do not detect and replicate the renaming of source fields.
•Application Ingestion and Replication retains the masking configuration of source fields on the target. For example, if a masked field on the source contains the value 1234****, the corresponding field on the target also contains the value 1234****.
•In the Secure Agent configuration properties, set the metadata_manager.jvm.ai.servicenow.enable.cdc.using.inherited.columns and task_container.jvm.ai.servicenow.enable.cdc.using.inherited.columns custom properties to true to include the child tables inheriting CDC columns from the parent table in the task.
• Set the ai.servicenow.enable.additional.datatypes custom property to true, to enable the following ServiceNow data types:
- Audio
- Basic Image
- Encrypted Text
- FX Currency
- Image
- Journal List
- List
- Name-Value Pairs
- Reference
- Video
- Wiki
Note: For the Reference data type, when the ai.servicenow.enable.additional.datatypes custom property is set to true, the columnName_value, columnName_display_value, and columnName_link attributes are available in the target. If the ai.servicenow.enable.additional.datatypes custom property is set to false, only the columnName attribute is available in the target.
Guidelines for Workday sources
Consider the following guidelines when you use Workday sources:
Application Ingestion and Replication provides you the option to extract Workday data through the following web services:
•Workday Web Services: Provides access to Workday data through SOAP APIs.
•Workday Report-as-a-Service (RaaS): Provides access to data in the custom objects and fields through custom reports.
The option to select the required web service appears on the Source tab of the application ingestion and replication task wizard.
Guidelines for Workday Web Services
•You can use Workday Web Services to ingest the data of Workday Human Capital Management (HCM). On the Source tab of the application ingestion and replication task wizard, you can select the specific HCM services that you want to replicate on your target.
•Application Ingestion and Replication replicates the data of only the source operations whose name start with Get_.
•Application ingestion and replication jobs retrieve the source data in an XML structure and then writes the data to the target as a single object in JSON or XML format. When you configure an application ingestion and replication job, on the Source tab of the task wizard, you can specify the format of the target data. On the target, each record contains the following fields:
- WID: Stores the unique identifier or primary key of the record.
- Data: Stores the content of the record in JSON or XML format.
•Application Ingestion and Replication retains the hierarchical structure of source data on the target.
•Application Ingestion and Replication does not propagate the data stored in custom objects.
•When an application ingestion and replication job ingests an operation to the target, the operation is renamed in the following format: <Service_Name>__<Operation_Name>
•Workday might not allow initial load jobs to replicate the data of the Get_Budgest_Pools operation.
•Initial load jobs configured for a Workday source and a Microsoft Azure Data Lake Storage Gen2 target do not generate schema files for the following operations if they do not contain any record:
•The first time an incremental load job runs to capture change data for a Workday source, the job retrieves and loads only the change records that are created after the date and time specified in the application ingestion and replication task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•Incremental load jobs can automatically capture the insert and update operations performed on the source. The jobs perform upsert operations on the target to replicate the insert and update operations performed on the source data. The application ingestion and replication task wizard does not provide the schema drift options for Workday sources because the upsert operations automatically replicate all changes made to the source schema.
•Application Ingestion and Replication can capture change data for the following Workday services and operations:
Service
Operations
Human_Resources
Get_Job_Profiles
Get_Organizations
Get_Workers
Recruiting
Get_Evergreen_Requisitions
Get_Job_Requisitions
Get_Organizations
Get_Positions
Staffing
Get_Organizations
Get_Positions
Get_Workers
Guidelines for Workday RaaS
•You can use Workday RaaS only in initial load tasks.
•On the Source tab of the application ingestion and replication task wizard, you can choose to read data from one or more custom reports. If you choose to extract multiple reports, create a CSV file on the Secure Agent host and list the name of the reports in the first column of the file.
•Application ingestion and replication jobs configured to use Workday RaaS cannot detect the changes made to the source schema.
Guidelines for Zendesk sources
Consider the following guidelines when you use Zendesk sources:
•The first time an incremental load job runs to capture change data for a Zendesk source, the job retrieves and loads only the change records that are created after the date and time specified in the associated ingestion task. You can specify the date and time when you configure the source in the application ingestion and replication task wizard. However, when you resume a stopped or aborted job, the job begins propagating change data from where it last left off.
•Incremental load jobs and combined initial and incremental load job configured for Zendesk sources do not detect and replicate the changes that are made to the source schema.
•If a source record contains multiple custom fields, application ingestion and replication jobs store the data of all the custom fields as a JSON object in a single column of the target table.
•Application ingestion and replication jobs can retrieve data from the custom fields that are present in the following Zendesk objects:
- Organizations
- Requests
- Tickets
- Users
•Initial load jobs propagate null values for the data stored in the following Zendesk fields:
- dns results field in the Support Addresses object.
- raw request and raw response fields in the Target Failures object.
- reason code field in the Satisfactory Ratings object.
- URL field in the Sharing Agreements object.
•Initial load jobs do not propagate the data stored in the following Zendesk objects:
- Attachments
- Channel Framework
- Dynamic Content Item Variants
- End users
- Incremental Skill-based Routing
- NPS® Invitations
- NPS® Recipients
- NPS® Responses
- OAuth Tokens for Grant Types
- Push Notification Devices
- Search
- Side Conversations
- Side Conversation Attachment
- Side Conversation Events
- Skill-based Routing
- Ticket Comments
- Ticket Import
- User Identities
- User Passwords
•Application Ingestion and Replication can capture change data for the following Zendesk standard objects:
- NPS Recipients
- Organizations
- Side Conversation Events
- Tickets
- Tickets Metric Events
- Users
•Incremental load jobs can capture the deletion of records that are stored in the following objects:
- Organizations
- Tickets
- Users
•Application Ingestion and Replication does not replicate the hierarchical structure of the source fields of the type Object. On the target table, all fields are at the same hierarchical level. When an application ingestion and replication job replicates an Object field with multiple hierarchical levels, the job creates the corresponding columns at the same hierarchical level.