When you configure the Informatica Intelligent Cloud Services catalog source, you define the settings for the metadata extraction capability and other optional capabilities.
The metadata extraction capability extracts source metadata from external source systems. You can also configure other capabilities that the catalog source includes.
You can save the catalog source configuration at any point after you enter the connection information. After you save the catalog source, you can choose to run the catalog source job. To run the job once, click Run. To run metadata extraction and other capabilities on a recurring schedule, configure schedules on the Schedule tab.
Configure metadata extraction
When you configure the Informatica Intelligent Cloud Services catalog source, you choose a runtime environment, define filters, and enter configuration parameters for metadata extraction.
Before you configure metadata extraction, configure runtime environments in the IDMC Administrator.
1In the Connection and Runtime area, choose a serverless runtime environment or the Secure Agent group where you want to run catalog source jobs.
Note:
Serverless runtime environment options are available if the catalog source works with a serverless runtime environment.
2Choose to retain, delete, or deprecate objects that are deleted from the source system in the catalog with the Metadata Change Option.
- Retain. Retains objects that are deleted from the source system in the catalog. If you update or add a filter, the catalog retains objects extracted from the previous job and extracts additional objects that match the current filter. Objects deleted from the source system are not deleted from the catalog. Enrichments added on deleted objects and relationships are retained.
- Delete. Deletes metadata from the catalog based on objects deleted from the source system and changes you make to the filter. Enrichments added on deleted objects and relationships are also permanently lost. Objects renamed in the source system are removed and recreated in the catalog.
- Deprecate. When objects are deleted from the source system, the catalog changes the lifecycle of the imported objects to Obsolete. The lifecycle of objects also changes to Obsolete when you add exclusion filters or reduce the scope so that objects extracted in earlier jobs are no longer included. For example, the first job includes the sales schema. The second job also includes the sales schema, but excludes the sales.transactions table. In this case, the second job marks the sales.transactions object as Obsolete, even though the first job includes it. This does not impact enrichments applied to deprecated objects and relationships. Objects renamed in the source system are removed and recreated in the catalog. When you run the catalog source job again for other capabilities such as data classification, relationship discovery, or glossary association, the job doesn't consider obsolete objects. Obsolete objects remain in the catalog until they are purged when you run a Purge Obsolete Objects job on the Explore page.
Note:
You can also change the configured metadata change option when you run a catalog source.
3In the Filters area, define one or more filter conditions to apply for metadata extraction:
aSelect Yes.
bFrom the Include/Exclude list, choose to include or exclude metadata based on the filter parameters.
cFrom the Object type list, select Application Integration, CDIR, Data Integration, MDM SaaS, or Reference 360.
dFor the Application Integration object type, select Process or Guide as the filter criteria.
eFor the Data Integration object type, select All types as the filter criteria.
fFor the MDM SaaS object type, select Hierarchy, Relationship, or Business Entity as the filter criteria.
gFor the Reference 360 object type, select Code List or Hierarchy as the filter criteria.
hFor the CDIR object type, select Application or Database as the filter criteria.
iClick Select.
jIn the Select Values dialog box, enter a value to specify the object location.
Each pattern can contain wildcards. Use a question mark to represent a single character. For example, A? matches A1, Ab. Use an asterisk to represent multiple characters. For example, A* matches A, Ab, ABC. If an object name contains an asterisk, use double quotes. When you enter values for filters, enclose them in double quotes if you include spaces before or after the string value.
Examples:
▪ To include or exclude all Application Integration processes with names that start with ‘Process_’ located in the ‘Project1’ project or in any folder in the ‘Project1’ project, select Application Integration as the object type, select Process as the filter criteria, click Select, and enter Project1/*/Process_* in the values field.
▪ To include or exclude all Application Integration guides with names that start with 'Guide_' from the 'Folder1' folder located in the 'Project1' project, select Application Integration as the object type, select Guide as the filter criteria, click Select, and enter Project1/Folder1/Guide_* in the values field.
▪ To include or exclude all Data Integration mapping tasks in the 'Project1' project, select Data Integration as the object type, select All Types as the filter criteria, click Select, and enter Project1 in the values field.
▪ To include or exclude all Data Integration mapping tasks with names that start with 'Task_' located in the 'Project1' project or in any folder in the ‘Project1’ project, select Data Integration as the object type, select All Types as the filter criteria, click Select, and enter Project1/*/Task_* in the values field.
▪ To include or exclude all Data Integration mapping tasks with names that start with 'Task_' from the 'Folder1' folder located in the 'Project1' project, select Data Integration as the object type, select All Types as the filter criteria, click Select, and enter Project1/Folder1/Task_* in the values field.
▪ To include or exclude Reference 360 code lists with names that start with 'Enterprise' located in reference data sets with names that start with 'Reference' and end with 'Set', select Reference 360 as the object type, select Code List as the filter criteria, click Select, and enter Reference*Set/Enterprise* in the values field.
▪ To include or exclude Reference 360 code lists with names that start with 'CountryCodes' followed by one character located in the 'Country' reference data set, select Reference 360 as the object type, select Code List as the filter criteria, click Select, and enter Country/CountryCodes? in the values field.
▪ To include or exclude all Reference 360 hierarchies with names that start with the letter ‘H’, select Reference 360 as the object type, select Hierarchy as the filter criteria, click Select, and enter H* in the values field.
▪ To include or exclude all Reference 360 hierarchies with names that start with 'HierarchyNo' followed by one character, select Reference 360 as the object type, select Hierarchy as the filter criteria, click Select, and enter HierarchyNo? in the values field.
▪ To include or exclude the 'FullHierarchyName' Reference 360 hierarchy, select Reference 360 as the object type, select Hierarchy as the filter criteria, click Select, and enter FullHierarchyName in the values field.
▪ To include or exclude all MDM SaaS business entities with names that start with 'Objectname' located in the 'Project1' project or in any folder in the 'Project1' project, select MDM SaaS as the object type, select Business Entity as the filter criteria, click Select, and enter Project1/*/Objectname* in the values field.
▪ To include or exclude all MDM SaaS hierarchies with names that start with ‘Hier_’ from the ‘Folder1’ folder located in the ‘Project1’ project, select the MDM SaaS object type, select Hierarchy as the filter criteria, click Select, and enter Project1/Folder1/Hier_* in the values field.
▪ To include or exclude all MDM SaaS relationships with names that start with 'Rel_' followed by a single character, located in the 'Project1' project or in any folder in the 'Project1' project, select the MDM SaaS object type, select Relationship as the filter criteria, click Select, and enter Project1/*/Rel_? in the values field.
▪ To include or exclude all CDIR application mapping tasks in the ‘Project1’ project, select CDIR as the object type, select Application as the filter criteria, click Select, and enter Project1 in the values field.
▪ To include or exclude all CDIR application mapping tasks with names that start with ‘Task_’ located in the ‘Project1’ project or in any folder in the ‘Project1’ project, select CDIR as the object type, select Application as the filter criteria, click Select, and enter Project1/*/Task_* in the values field.
▪ To include or exclude all CDIR application mapping tasks with names that start with ‘Task_’ from the ‘Folder1’ folder, located in the ‘Project1’ project, select CDIR as the object type, select Application as the filter criteria, click Select, and enter Project1/Folder1/Task_* in the values field.
▪ To include or exclude all CDIR application mapping tasks with names that start with ‘Objectname’ located in the ‘Project1’ project or in any folder in the ‘Project1' project, select CDIR as the object type, select Application as the filter criteria, click Select, and enter Project1/*/Objectname* in the values field.
▪ To include or exclude all CDIR database mapping tasks in ‘Project1’, select CDIR as the object type, select Database as the filter criteria, click Select, and enter Project1 in the values field.
▪ To include or exclude all CDIR database mapping tasks with names that start with ‘Task_’ located in the ‘Project1’ project or in any folder in the ‘Project1’ project, select CDIR as the object type, select Database as the filter criteria, click Select, and enter Project1/*/Task_* in the values field.
▪ To include or exclude all CDIR database mapping tasks with names that start with ‘Task_’ from the ‘Folder1’ folder located in the ‘Project1’ project, select CDIR as the object type, select Database as the filter criteria, click Select, and enter Project1/Folder1/Task_* in the values field.
▪ To include or exclude all CDIR database mapping tasks with names that start with ‘Objectname’ located in the ‘Project1’ project or in any folder in the ‘Project1’ project, select CDIR as the object type, select Database as the filter criteria, click Select, and enter Project1/*/Objectname* in the values field.
To include or exclude multiple objects, click the Add icon to add filters with the OR condition.
4Optionally, in the Configuration Parameters area, enter properties to override default context values and job parameters.
The following table describes the property that you enter for Catalog Source Configuration Options:
Parameter
Description
Parameter file path replacements
Before you run the job, define values for the mapping parameters in the parameter file that is accessible by the Secure Agent. If the Secure Agent for Data Governance and Catalog and Data Integration run on different machines, then copy the parameter files to the Data Governance and Catalog Secure Agent machine.
- In Path, specify the path to the parameter files for the mapping tasks. For example, /cdi/agent/parameterfiles/path
- In Replacement, specify the path to the Data Integration parameter file that you copied to the Data Governance and Catalog Secure Agent machine. For example, /cdgc/agent/parameterfiles/path
The following table describes the advanced Catalog Source Configuration Options:
Note:
The advanced Catalog Source Configuration Options appears when you click Show Advanced.
Property
Description
Data Integration Input Fetch Mode
Determines the source of the Data Integration mapping task XML files that the catalog source uses to extract metadata. Select one of the following options:
- Files. Extracts metadata only from files in the directory specified in Data Integration Input Files Folder.
- API. Extracts metadata through the Informatica Intelligent Cloud Services API.
- Both. Extracts metadata through the API and from the files, and matches objects that both methods find.
Data Integration Input Files Folder
Specify the complete directory on the Secure Agent machine that contains the Data Integration input files. The directory must contain the mapping task XML files and parameter file.
This field appears only when you set the Data Integration Input Fetch Mode field to Files or Both.
Maximum Taskflow File Age in Days
Specify the age, in days, of the taskflow files that the catalog source filters. Default is -1, which includes all files.
If you enter an invalid value or a negative value other than -1, the catalog source uses the default value and includes all files. If you enter 0, the job fails.
This field appears only when you set the Data Integration Input Fetch Mode field to Files or Both.
The following table describes the property that you can enter for additional settings:
Note:
The Additional Settings section appears when you click Show Advanced.
Property
Description
Expert Parameters
Enter additional configuration options to be passed at runtime. Required if you need to troubleshoot the catalog source job.
Caution:
Use expert parameters when it is recommended by Informatica Global Customer Support.
5Configure additional capabilities for the catalog source by clicking on the tabs.
Configure data classification
Enable the data classification capability to identify and organize data into relevant categories based on the functional meaning of the data.
1Click the Data Classification tab.
2Select Enable Data Classification.
3Choose one or both of the following options:
- Generated Data Classifications. CLAIRE automatically generates data classifications for the data elements.
- Data Classification Rules. Choose from predefined or custom data classifications.
1Click Add Data Classification. The Select Data Classifications dialog box appears.
2Select the data classifications that you want to use.
3Click OK.
Configure glossary association
Enable the glossary association capability to associate glossary terms with technical assets, or to get recommendations for glossary terms that you can manually associate with technical assets in Data Governance and Catalog.
Metadata Command Center considers all published business terms in the glossary while making recommendations to associate your technical assets.
1Click the Glossary Association tab.
2Select Enable Glossary Association.
3Select Enable auto-acceptance to automatically accept glossary association recommendations.
4Specify the Confidence Score Threshold for Auto-Acceptance to set a threshold limit based on which the glossary association capability automatically accepts the recommended glossary terms.
Note:
Specify a percentage from 80 to 100. If the score is higher than the specified limit, the glossary association capability automatically assigns a matching glossary term to the data element.
5Select Enable Below-threshold Recommendations to receive glossary association recommendations below the auto-acceptance threshold. If you enable auto-acceptance, you can enable below-threshold recommendations to receive glossary recommendations below the auto-acceptance threshold.
6Specify the Confidence Score Threshold for Recommendations to set a threshold based on which the glossary association capability makes recommendations
If you enable auto-acceptance, specify a percentage from 80 to the selected auto-acceptance threshold. You can accept or reject the recommended glossary terms that fall within this range in Data Governance and Catalog.
If you disable auto-acceptance, specify a percentage from 80 to 100 inclusive.
7Choose to automatically assign business names and descriptions to technical assets. You can then choose to retain existing assignments and only assign business names and descriptions to assets that don't have assignments, or allow overwrite of existing assignments.
By default, existing assignments are retained.
8Optional. Choose to ignore specific parts of data elements when making recommendations. Select Yes and enter prefix and suffix keyword values as needed.
Click Select to enter a keyword. You can enter multiple unique prefix and suffix keywords. Keyword values are case insensitive.
9Optional. Choose specific top-level business glossary assets to associate with technical assets. Selecting a top-level asset selects its child assets as well. Select Top-level Glossary Assets and specify the assets on the Select Assets page.
10Optional. Choose to use abbreviations and synonym definitions from lookup tables for accurate glossary association. Select Yes to enable, and then click Select to upload a lookup table.
11Click Next.
The Associations page appears.
Configure automation
Enable the lineage discovery and autonomous enrichment capabilities to build complete lineage using CLAIRE and to receive enrichments propagated from catalog sources connected through lineage.
Enable Lineage-based Propagation on the Autonomous Enrichment tab of the Configure page before you configure this catalog source to receive enrichments. For more information, see Administration.
With the lineage-based propagation settings on this tab, you configure how this catalog source receives enrichments propagated from connected catalog sources. Enrichments propagate only to assets that don't already have enrichments.
1Click the Automation tab.
2In the Lineage Discovery section, select Enable Lineage Discovery.
3In the Filters area, define one or more filter conditions to apply for lineage discovery.
To define filters, you can choose to select catalog source types, asset groups, or enter a catalog source name or search from a list of catalog sources.
aSelect Yes to view filter options.
bFrom the Include/Exclude list, choose to include or exclude catalog sources for lineage discovery based on the filter parameters.
cFrom the filter type list, select catalog source type, catalog source name, or asset group.
dIn the filter value field, select the required catalog source types, or click the Search button and select catalog sources or asset groups.
Filters can contain the asterisk wildcard to represent multiple characters or empty text.
The filter options appear.
Examples:
▪ To include or exclude all Oracle catalog sources, select Catalog Source Type as the filter type and select Oracle in the filter value field.
▪ To include or exclude the 'Oracle_Retail' catalog source, select Catalog Source Name as the filter type and search for the catalog source or enter Oracle_Retail in the filter value field.
▪ To include or exclude all catalog sources with names that start with 'Oracle', select Catalog Source Name as the filter type and search for the catalog source or enter Oracle* in the filter value field.
▪ To include or exclude all catalog sources with names that end with 'Retail', select Catalog Source Name as the filter type and search for the catalog source or enter *Retail in the filter value field.
▪ To include or exclude all catalog sources with names that contain 'Ret', select Catalog Source Name as the filter type and search for the catalog source or enter *Ret* in the filter value field.
▪ To include or exclude all catalog sources that are part of the 'Financial Group' asset group, select Asset Group as the filter type and search Financial Group in the filter value field.
Note:
You can't add more than one include or exclude filter for the same filter type.
eOptionally, to define an additional filter with an AND condition, click the Add icon.
4In the Autonomous Enrichment section, select Enable Lineage-based Propagation.
The lineage-based propagation settings appear.
5Select the enrichment types to receive from connected catalog sources.
You can select Classification, Description, Business Name, and Glossary Term.
6Choose how to apply received enrichments.
- Select Auto-accepted to apply received enrichments automatically.
- Select Recommended to list them as suggestions that you can curate in Data Governance and Catalog.
7Choose whether to receive enrichments only from certified assets.
Assets are certified in Data Governance and Catalog. This option is selected by default. If you clear it, this catalog source also receives enrichments from assets that are not certified, which might be less reliable.
8Choose where this catalog source receives enrichments from.
- Select Catalog Source to receive enrichments from a specific catalog source.
- Select Nearest Neighbor to receive enrichments from the closest connected catalog source based on lineage.
9If you select Catalog Source, select a catalog source from the list.
Ensure that the catalog source you select is within 15 lineage hops. Enrichments don't propagate from a catalog source beyond that distance.
10If you select Nearest Neighbor, configure the lineage settings:
aSet Lineage Direction to search upstream, downstream, or in both directions.
Upstream catalog sources feed data into this catalog source. Downstream catalog sources receive data from this catalog source. Select both directions to search upstream and downstream at the same time.
bSet Number of Hops to define how far along lineage to search.
The search stops at the first connected catalog source that has enrichments, up to a maximum of 15 hops.
11Click Save.
When you save, the catalog source begins receiving enrichments created from that point onward. Saving doesn't retrieve enrichments created prior to this date and time. To receive such enrichments, run the catalog source job manually and include the Autonomous Enrichment capability.