Qlik Sense is a business intelligence and visual analytics platform that supports a range of use cases including centrally deployed analytic applications, dashboards, and embedded analytics within a scalable framework.
Objects extracted
Metadata Command Center extracts the following objects from a Qlik Sense source system:
•Application
•Application Model
•Column
•Condition
•Connection
•Connection Schema
•Database Schema
•Dimension
•Dimension Attribute
•Document
•Expression
•Field
•File
•File Directory
•Folder
•Hub
•Measure
•Measure Attribute
•QVD
•QVD Model
•Source Column
•Source Table
•Story
•Stream
•System Field
•Table
Prerequisites for configuring the Qlik Sense catalog source
Before you create a catalog source, ensure that you have the information required to connect to the source system.
Perform the following tasks:
•Copy the Qlik Sense certificate to the same machine where the Secure Agent is running. You can provide the path to this certificate on the Registration page in Metadata Command Center.
•To access assets in the Qlik Sense source system, perform one of the following steps:
- Assign a predefined Audit Admin role to the Qlik Sense user.
- Create a custom role that has read access to apps, app objects, streams, and data connections and assign it to the Qlik Sense user.
On the Registration page in Metadata Command Center, enter values for the following connection properties:
Property
Description
Engine API URL
The Qlik Sense server URL to connect to the Qlik API Engine.
User directory
User directory of the Qlik Sense server.
User
Qlik Sense user name to authenticate to the Qlik Sense server.
Certificate
Location of the client certificate required for authentication to the Qlik Sense server.
Certificate Password
Password of the Qlik Sense client certificate. Specify the password if the Qlik Sense client certificate is generated with a password.
Configuration parameters for metadata extraction
Expand the Catalog Source Configuration Options in the Metadata Extraction tab on the Configuration page. Configure the following additional parameters to extract metadata from a Qlik Sense source system:
Property
Description
Incremental Import
Choose whether you want to extract the metadata that has changed since the previous run or extract complete metadata. Select one of the following options:
- True. Extracts only the changes to the metadata since the last metadata extraction job.
- False. Extracts the complete metadata.
Log Folder
The path to the Qlik Sense log folder.
Use this option if you are extracting metadata from a source system that contains dynamic information such as subroutines, loops, and variable definitions. The log files in the log folder are used to extract complete lineage.
If the log folder on the Qlik Sense machine is accessible to the Secure Agent, you can specify the direct path to the folder. If the folder is not accessible, you can copy the files to a log folder accessible to the Secure Agent.
To enter multiple paths, use the -cluster.log.folder miscellaneous option.
QVD Folder
The path to the QVD folder stored in the QVD server.
Use this option if you are extracting metadata from a source system that contains parameterized connections.
To enter multiple paths, use the cluster.qvd.folder miscellaneous option.
Worker Threads
Optional. The number of worker threads to process metadata asynchronously.
You can enter a positive integer value.
If you don’t enter a value, Metadata Command Center computes and assigns a value between one and six based on the Java Virtual Machine (JVM) architecture and the number of available CPU cores on the Secure Agent machine.
Miscellaneous Options
You can specify the following additional options to pass at runtime:
-database.type ORACLE -log.notavailable -file.path [qlik server file path]=[agent file path] -m 4G -customXMILocation [path to xmi files]
- -m. Specify the memory size required to run the metadata extraction job. For example, enter -m 4G. The default memory size is 1GB.
- -database.type. Specify the list of connection database types as comma-separated value pairs. For example, enter -database.type ORACLE
If databases are accessed via generic ODBC connections, then specify the exact database type in order to properly parse the database-specific SQL syntax for lineage.
- -log.notavailable. Specify this option if have not entered any value for the Log Folder property. If you are extracting metadata from a source system that contains dynamic metadata such as subroutines, loops, and variable definitions, then the Qlik document execution log files are required because the dynamic metadata cannot be directly extracted from the Qlik scripts. In such cases, some critical metadata for lineage are missing.
This option lets you extract metadata even if the log folder path is not available.
- -cluster.log.folder. If you need to enter more than one log folder path, specify this option to enter the paths.
For example,
-cluster.log.folder d:\cluster1\
-cluster.log.folder d:\cluster2\
- -cluster.qvd.folder If you need to enter more than one QVD folder path, specify this option to enter the paths.
For example, you can specify multiple paths for multiple cluster configurations:
-cluster.qvd.folder d:\cluster1\
-cluster.qvd.folder c:\cluster2\
- -file.path. A Qlik document contains statements, such as INCLUDE, STORE, or LOAD, which operate with the file path. If the original file path is not accessible, then use this option to replace a portion of the original file path with a new one by specifying multiple file path options.
For example, -file.path [qlik server file path]=[agent file path].
The catalog source applies multiple file path options in the order in which they are specified.
- -directory. A Qlik document DIRECTORY statement is used to set the directory path for subsequent LOAD statements.
If this directory is inaccessible, then use a DIRECTORY statement to redirect it to another directory. Copy the DIRECTORY statement from a Qlik document execution log, add =, and specify the path to another directory. For example, if folder c:\folder1 is redirected to folder d:\folder2, then enter -directory "c:\folder1=d:\folder2".
When the path after the DIRECTORY statement is empty, such as -directory "[]=d:\folder2", then all DIRECTORY statements are redirected to the specified directory.
- -customXMILocation. Specify this option if you want to load the XMI files that are generated when you run a metadata extraction job. Specify the location where the XMI files are stored. For example:
- -connection.map. Specify this option if you want to map a source path to a destination path. You can use this option when different paths point to the same object. For example:
Here, directory C:\data is referred to by multiple network drives like M: and N: on Windows.
- -websocket.timeout Specify the time in seconds that the import bridge must wait for a websocket response. Default is 30.
When the Qlikshare files are on a Windows machine and the Secure Agent runs on a Linux machine, you need to copy the QVD and log files from Windows to the Linux machine. To view the complete lineage in Data Governance and Catalog and for the connection assignments to work, specify the following options to pass at runtime:
- -file.path "<location where QVD and log files are available on the Windows machine>=<location where QVD and log files are available on the Linux machine>"
- -directory "<all directories>=<location where QVD and log files are available on the Linux machine>"
- -connection.map "<location where QVD and log files are available on the Linux machine>=<location where QVD and log files are available on the Windows machine>"
If the source system references another source system, create a connection assignment in Metadata Command Center to view data lineage with endpoints. To create a connection assignment, create a connection based on the referenced source system, and then assign the connection to the catalog source.
Note: You can view the lineage with reference objects without creating a connection assignment. After connection assignment, you can view the actual objects.
You can assign the following source systems as endpoint catalog sources: