Introduction to IDMC Cloud Data Integration Job Log Capture recipe

The recipe enhances the IDMC system visibility to enable effective monitoring, troubleshooting, and overall management of users' IDMC environments. It facilitates custom analytics and charge back requirements by enabling processes to systematically store internal IDMC log data to a user's cloud storage.

The IDMC CDI Job Log Capture recipe includes a taskflow designed to periodically extract CDI job log information from an IDMC organization and write it to a cloud storage location.

The taskflow is designed to run periodically, with the default schedule set to run once a day and extract CDI job log information from the previous day. For example, if the taskflow runs at 2 AM on April 11, it extracts log data from midnight April 9 to midnight April 10. With proper configuration, it can retrieve data from any IDMC organization or sub-organization, not necessarily limited to the organization where the recipe is installed.

The CDI job log information retrieved by the recipe is internally captured by the Operation Insights data collector service that runs within a Secure Agent. Therefore, there can be up to a 2 hour delay between when a job completes and when it can be captured by this log recipe. Also, verify that the OI Data Collector is running on all Secure Agents from which you are capturing log information.

The target location and type, such as AWS, Azure, Google, or Oracle, are determined by the connection object assigned to the taskflow’s mapping task, along with the target load path provided in the in_tgtPath input parameter.

This recipe is designed to run on Linux because it employs shell scripts to manage staging files. It utilizes both Data Integration and Application Integration, and it is not compatible with a serverless Application Integration server.