Catalog API
Use the Catalog API to extract data from the Data Integration Hub catalog, including topic metadata and metadata about the publications and subscriptions that are associated with each topic.
You can extract metadata pertaining to topics, publications, and subscriptions for which you have both View privileges and Read access permissions.
To extract data from the catalog, use the following REST URL:
http://localhost:18080/dih-console/api/v1/catalog/topics
Data Integration Hub Catalog API Response
When you use the Data Integration Hub Catalog API to extract data from the Data Integration Hub catalog, the API returns a JSON string that contains metadata about all the topics for which you have the required privileges.
The string includes the following data for each topic in the response:
- topicName
- Name of the topic.
- topicDesc
- Textual description of the topic.
- topicType
- Type of the topic: Delta or Full.
- topicTables
- For each table in the topic, an entry with the table name and detailed information about each of the table fields.
- publications
- For each publication that is associated with the topic, the following data is provided:
- publicationName
- Name of the publication.
- publicationDesc
- Textual description of the publication.
- applicationName
- Application from which the publication publishes data or files.
- publicationSourceType
- Type of publication source.
- publicationConnectionName
- For relational database publications and for HDFS publications: name of the connection from where the publication workflow reads the data or the files to be published.
- publicationDBType
- For relational database publications: type of database.
- subscriptions
- For each subscription that is associated with the topic, the following data is provided:
- subscriptionName
- Name of the subscription.
- subscriptionDesc
- Textual description of the subscription.
- applicationName
- Application that consumes data or files from the topic.
- subscriptionTargetType
- Type of subscription target.
- subscriptionConnectionName
- For relational database subscriptions and for HDFS subscriptions: name of the connection to where the subscription workflow writes the data or the files that the application consumes.
- subscriptionDBType
- For relational database subscriptions: type of database.
Sample Data Integration Hub Catalog API Response
The following example shows a response to a request to extract data from the Data Integration Hub catalog:
{
"responseCode": "SUCCESS",
"catalogTopics": [
{
"topicName": "FileTopic",
"topicDesc": null,
"topicType": "Delta"
"topicTables": [
{
"tableName": "Orders"
"tableFields":[
{
"name":"_Name_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_Type_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_ParentId_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
}
}
],
"publications": [
{
"publicationName": "FilePub",
"publicationDesc": null,
"applicationName": "FileApp",
"publicationSourceType": "FLAT_FILE",
"publicationConnectionName": null,
"publicationDBType": null
}
],
"subscriptions": [
{
"subscriptionName": "FileSub",
"subscriptionDesc": null,
"applicationName": "FileApp",
"subscriptionTargetType": "FLAT_FILE",
"subscriptionConnectionName": null,
"subscriptionDBType": null
}
]
},
{
"topicName": "OrderTopic",
"topicDesc": null,
"topicType": "Delta",
"topicTables": [
{
"tableName": "OrderTable"
"tableFields":[
{
"name":"_Name_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_Type_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_ParentId_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_StartDate_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_EndDate_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
}
},
{
"tableName": "CustomerTable"
"tableFields":[
{
"name":"_Name_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_Type_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_ParentId_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_ExpectedRevenue_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
},
{
"name":"_IsActive_"
"fieldType":"STRING"
"nullable":false
"scale":-1
"precision":255
"length":255
"primaryKey":false
"filterAccelerator":false
"encrypted":true
}
},
{
},
],
"publications": [
{
"publicationName": "OrdersPublication",
"publicationDesc": null,
"applicationName": "OrderPublications",
"publicationSourceType": "RDBMS",
"publicationConnectionName": "OrderSource",
"publicationDBType": "SQLSERVER"
}
],
"subscriptions": [
{
"subscriptionName": "OrdersSubscription",
"subscriptionDesc": null,
"applicationName": "OrderSubscriptions",
"subscriptionTargetType": "FLAT_FILE",
"subscriptionConnectionName": null,
"subscriptionDBType": null
},
{
"subscriptionName": "OrderSubs",
"subscriptionDesc": null,
"applicationName": "OrderSubscriptions",
"subscriptionTargetType": "RDBMS",
"subscriptionConnectionName": "OrderTarget",
"subscriptionDBType": "SQLSERVER"
}
]
}
]
}
]
}