Source Metadata and Data Profile Filter
In a resource that contains multiple tables and views, you can choose to extract source metadata from specific tables and views, and extract profile metadata and data from specific tables and views.
You can configure the filter field if one of the following conditions is true:
- •You want to extract metadata only for a few tables and views from a list of all of the tables and views in a resource.
- •You want to include a few tables and views from the next resource run.
- •You want to exclude a few tables and views from the resource run.
- •You want to exclude a few tables and views from the next resource run.
In a resource, you can use the following filters to include and exclude tables and views:
- •Source Metadata Filter. In the Metadata and Load Settings tab in the Resource wizard, enter the tables and views that you want to include or exclude in a resource run. The scanner extracts the source metadata based on the tables and views in the Source Metadata Filter field.
- •Data Profile Filter. In the Metadata and Load Settings tab in the Resource wizard, enter the tables and views that you want to include or exclude in a profile run. The profile scanner extracts profile metadata based on the tables and views in the Data Profile Filter field.
To include tables and views, you can enter full names of tables and views, or use the % wildcard character with the name or pattern. Use semicolons (;) to separate the table names and view names. If the pattern contains a special character, such as a whitespace, enclose the pattern within single quotes. For example, HR_id; 'HR Emp%'; HR_job%.
Note: The Data Profile Filter supports Java regular expressions such as % , ^ , $ , . , | , and ( a | b ), and the Source Metadata Filter supports the % expression. The filter constraint applies to Azure Microsoft SQL Server, Azure Microsoft SQL Data Warehouse, Hive, and all relational database sources.
You can choose one or more of the following patterns to identify the tables and views:
- • <name>%. Includes tables and views that start with <name>.
For example, A% includes all the tables and views that start with the character A.
- •%<name>. Includes tables and views that end with <name>.
For example, %id includes all the tables and views that end with the text 'id'.
- •%<name>%. Includes tables and views that contain <name>.
For example, %DB% includes all tables and views that contain the keyword "DB".
- •<name>. Includes tables and views that match <name>.
For example, HR_id filter includes any table or view in the resource that matches HR_id.
To exclude tables and views from the scan, prefix the term NOT before the name or pattern. For example, NOT %ID; NOT 'ID%'. Use semicolons (;) to separate the table names and view names.
Troubleshooting
- What happens if I include a table in the Source Metadata Filter that depends on an external object?
- When a filtered table or view depends on an external object such as a database, the resource scanner extracts the metadata of the external object even when the external object does not meet the filter conditions because the filter does not apply to the dependent external object.
- How do I know which tables and views were excluded in the profile scan?
- By default, the log files do not list all the tables and views that were excluded in a scan. To view the list of excluded tables and views, enable the Debug log level and run the scan. The log files list all the excluded tables and views in the scan.