Service Components
The service components of the Data Integration Service include modules that manage requests from client tools. They also include managers that manage application deployment, caches, and job optimizations.
The service components run within the Data Integration Service process. The Data Integration Service process must run on a node with the service role. A node with the service role can run application services.
Mapping Service Module
The Mapping Service Module manages requests to preview data and run mappings.
The following table lists the requests that the Mapping Service Module manages from the different client tools:
Request | Client Tools |
|---|
Preview source or transformation data based on mapping logic. | Developer tool Analyst tool |
Run a mapping. | Developer tool |
Run a mapping in a deployed application. | Command line |
Preview an SQL data service. | Developer tool |
Preview a web service operation mapping. | Developer tool |
Sample third-party client tools include SQL SQuirreL Client, DBClient, and MySQL ODBC Client.
When you preview or run a mapping, the client tool sends the request and the mapping to the Data Integration Service. The Mapping Service Module sends the mapping to the LDTM for optimization and compilation. The LDTM passes the compiled mapping to a DTM instance, which generates the preview data or runs the mapping.
When you preview data contained in an SQL data service in the Developer tool, the Developer tool sends the request to the Data Integration Service. The Mapping Service Module sends the SQL statement to the LDTM for optimization and compilation. The LDTM passes the compiled SQL statement to a DTM instance, which runs the SQL statement and generates the preview data.
When you preview a web service operation mapping in the Developer tool, the Developer tool sends the request to the Data Integration Service. The Mapping Service Module sends the operation mapping to the LDTM for optimization and compilation. The LDTM passes the compiled operation mapping to a DTM instance, which runs the operation mapping and generates the preview data.
Profiling Service Module
The Profiling Service Module manages requests to run profiles and generate scorecards.
When you run a profile in the Analyst tool or the Developer tool, the application sends the request to the Data Integration Service. The Profiling Service Module converts the profile into one or more mappings. The Profiling Service Module sends the mappings to the LDTM for optimization and compilation. The LDTM passes the compiled mappings to DTM instances that get the profiling rules and run the profile.
When you run a scorecard in the Analyst tool or the Developer tool, the application sends the request to the Data Integration Service. The Profiling Service Module converts the scorecard into one or more mappings. The Profiling Service Module sends the mappings to the LDTM for optimization and compilation. The LDTM passes the compiled mappings to DTM instances that generate a scorecard for the profile.
To create and run profiles and scorecards, you must associate the Data Integration Service with a profiling warehouse. The Profiling Service Module stores profiling data and metadata in the profiling warehouse.
SQL Service Module
The SQL Service Module manages SQL queries sent to an SQL data service from a third-party client tool.
When the Data Integration Service receives an SQL query from a third-party client tool, the SQL Service Module sends the SQL statement to the LDTM for optimization and compilation. The LDTM passes the compiled SQL statement to a DTM instance to run the SQL query against the virtual tables in the SQL data service.
If you do not cache the data when you deploy an SQL data service, a DTM instance is started to run the SQL data service. Every time the third-party client tool sends an SQL query to the virtual database, the DTM instance reads data from the source tables instead of cache tables.
Web Service Module
The Web Service Module manages web service operation requests sent to a web service from a web service client.
When the Data Integration Service receives requests from a web service client, the Web Service Module sends the web service operation mapping to the LDTM for optimization and compilation. The LDTM passes the compiled mapping to a DTM instance that runs the operation mapping. The Web Service Module sends the operation mapping response to the web service client.
Workflow Orchestration Service Module
The Workflow Orchestration Service Module manages requests to run workflows.
When you start a workflow instance in a deployed application, the Data Integration Service receives the request. The Workflow Orchestration Service Module runs and manages the workflow instance. The Workflow Orchestration Service Module runs workflow objects in the order that the objects are connected. The Workflow Orchestration Service Module evaluates expressions in conditional sequence flows to determine whether to run the next task. If the expression evaluates to true or if the sequence flow does not include a condition, the Workflow Orchestration Service Module starts and passes input data to the connected task. The task uses the input data to complete a single unit of work.
When a Mapping task runs a mapping, it sends the mapping to the LDTM for optimization and compilation. The LDTM passes the compiled mapping to a DTM instance to run the mapping.
When a task finishes processing a unit of work, the task passes output data back to the Workflow Orchestration Service Module. The Workflow Orchestration Service Module uses this data to evaluate expressions in conditional sequence flows or uses this data as input for the remaining tasks in the workflow.
Data Object Cache Manager
The Data Object Cache Manager caches data in an application.
When you enable data object caching, the Data Object Cache Manager can cache logical data objects and virtual tables in a database. The Data Object Cache Manager initially caches the data when you enable the application. Optimal performance for the cache depends on the speed and performance of the database.
By default, the Data Object Cache Manager manages the data object cache in the data object cache database. The Data Object Cache Manager creates the cache tables and refreshes the cache. It creates one table for each cached logical data object or virtual table in an application. Objects within an application share cache tables, but objects in different applications do not. If one data object is used in multiple applications, the Data Object Cache Manager creates a separate cache table for each instance of the data object.
Result Set Cache Manager
The Result Set Cache Manager manages cached results for SQL data service queries and web service requests. A result set cache is the result of a DTM instance that runs an SQL query against an SQL data service or a web service request against a web service operation.
When you enable result set caching, the Result Set Cache Manager creates in-memory caches to temporarily store the results of a DTM instance. If the Result Set Cache Manager requires more space than allocated, it stores the data in cache files. The Result Set Cache Manager caches the results for a specified time period. When an external client makes the same request before the cache expires, the Result Set Cache Manager returns the cached results. If a cache does not exist or has expired, the Data Integration Service starts a DTM instance to process the request and then it stores the cached the results.
When the Result Set Cache Manager stores the results by user, the Data Integration Service only returns cached results to the user that ran the SQL query or sent the web service request. The Result Set Cache Manager stores the result set cache for SQL data services by user. The Result Set Cache Manager stores the result set cache for web services by user when the web service uses WS-Security. The Result Set Cache Manager stores the cache by the user name that is provided in the username token of the web service request.
Deployment Manager
The Deployment Manager is the component in Data Integration Service that manages applications. When you deploy an application, the Deployment Manager manages the interaction between the Data Integration Service and the Model Repository Service.
The Deployment Manager starts and stops an application. The Deployment Manager validates the mappings, workflows, web services, and SQL data services in the application and their dependent objects when you deploy the application.
After validation, the Deployment Manager stores application run-time metadata in the Model repository. Run-time metadata includes information to run the mappings, workflows, web services, and SQL data services in the application.
The Deployment Manager creates a separate set of run-time metadata in the Model repository for each application. When the Data Integration Service runs application objects, the Deployment Manager retrieves the run-time metadata and makes it available to the DTM.
Logical Data Transformation Manager
The logical Data Transformation Manager (LDTM) optimizes and compiles jobs.
The LDTM can perform the following optimizations:
- Filter data to reduce the number of rows to be processed.
The LDTM applies optimization methods to filter data and reduce the number of rows to be processed. For example, the LDTM can use early selection optimization to move a filter closer to the source. It can use pushdown optimization to push transformation logic to a database. It can use the cost-based optimization method to change the join processing order. When you develop a mapping, you can choose an optimizer level that determines which optimization methods the LDTM can apply to the mapping.
- Determine the partitioning strategy to maximize parallel processing.
- If you have the partitioning option, the Data Integration Service can maximize parallelism for mappings and profiles. The LDTM dynamically determines the optimal number of partitions for each pipeline stage and the best way to redistribute data across each partition point.
- Determine the data movement mode to optimize processing of ASCII characters.
The LDTM determines whether to use the ASCII or Unicode data movement mode for mappings that read from a flat file or relational source. The LDTM determines the data movement mode based on the character sets that the mapping processes. When a mapping processes all ASCII data, the LDTM selects the ASCII mode. In ASCII mode, the Data Integration Service uses use one byte to store each character, which can optimize mapping performance. In Unicode mode, the service uses two bytes for each character.
After optimizing a mapping, the LDTM compiles the optimized mapping and makes it available to the execution Data Transformation Manager (DTM) to run.