Cluster Workflow Components
The cluster workflow creates an ephemeral cluster that includes a Create Cluster task, at least one Mapping task, and a Delete Cluster task.
The following image shows a sample cluster workflow:
Create Cluster Task
The Create Cluster task contains all the settings that Amazon EMR or Azure HDInsight requires to create a cluster with a master node and worker nodes. It also contains a reference to a Hadoop connection.
Create one Create Cluster task for a cluster workflow.
When you create a cluster workflow, you drag a Create Cluster task into the workflow editor, and then configure task properties.
Cloud Provisioning Configuration
The cloud provisioning configuration associates the Create Cluster task with the Hadoop connection associated with the workflow.
Configure the cloud provisioning configuration with information about the cloud platform account. To configure the cloud provisioning configuration, use the Administrator tool.
The Create Cluster task must include a reference to the cloud provisioning configuration.
For more information, see the Big Data Management Administrator Guide.
Hadoop Connection
Create a dedicated Hadoop connection to use with the cluster.
The Hadoop connection saves property values for the Data Integration Service to use for cluster workflow operations. When you run a cluster workflow, the Data Integration Service creates temporary Hadoop connections based on these values.
Mapping and Other Workflow Tasks
After the Create Cluster task, add other workflow tasks to run on the cluster.
Add one or more Mapping tasks to the workflow. You can also include Command and other workflow tasks.
Before you add mappings to Mapping tasks, you prepare the mappings. Configure the Hadoop Connection property in each mapping to designate whether to run the mapping on the cluster that the workflow creates, or on another cluster.
Delete Cluster Task
The Delete Cluster task terminates the cluster, and deletes the cluster and other resources that the cluster workflow creates.
To create an ephemeral cluster, add a Delete Cluster task to the workflow. When you do not include it, the cluster continues to run until you terminate it. You can use infacmd ccps deleteCluster to terminate the cluster at any time.