Big Data Management Administrator Guide > Connections > Cloud Provisioning Configuration
  

Cloud Provisioning Configuration

The cloud provisioning configuration establishes a relationship between the Create Cluster task and the Hadoop connection that the workflow uses to run mapping tasks. The Create Cluster task must include a reference to the cloud provisioning configuration. In turn, the cloud provisioning configuration points to the Hadoop connection that you create for use by the cluster workflow.
The properties to populate depend on the Hadoop distribution you choose to build a cluster on. Choose one of the following connection types:

AWS Cloud Provisioning Configuration Properties

The properties in the AWS cloud provisioning configuration enable the Data Integration Service to contact and create resources on the AWS cloud platform.

General Properties

The following table describes cloud provisioning configuration general properties:
Property
Description
Name
Name of the cloud provisioning configuration.
ID
ID of the cloud provisioning configuration. Default: Same as the cloud provisioning configuration name.
Description.
Optional. Description of the cloud provisioning configuration.
AWS Access Key ID
Optional. ID of the AWS access key, which AWS uses to control REST or HTTP query protocol requests to AWS service APIs.
If you do not specify a value, Informatica attempts to follow the Default Credential Provider Chain.
AWS Secret Access Key
Secret component of the AWS access key.
Required if you specify the AWS Access Key ID.
Region
Region in which to create the cluster. This must be the region in which the VPC is running.
Use AWS region values. For a list of acceptable values, see AWS documentation.
Note: The region where you want to create the cluster can be different from the region in which the Informatica domain is installed.

Permissions

The following table describes cloud provisioning configuration permissions properties:
Property
Description
EMR Role
Name of the service role for the EMR cluster that you create. The role must have sufficient permissions to create a cluster, access S3 resources, and run jobs on the cluster.
When the AWS administrator creates this role, they select the “EMR” role. This contains the default AmazonElasticMapReduceRole policy. You can edit the services in this policy.
EC2 Instance Profile
Name of the EC2 instance profile role that controls permissions on processes that run on the cluster.
When the AWS administrator creates this role, they select the “EMR Role for EC2” role. This includes S3 access by default.
Auto Scaling Role
Required if you configure auto-scaling for the EMR cluster.
This role is created when the AWS administrator configures auto-scaling on any cluster in the VPC.
Default: When you leave this field blank, it is equivalent to setting the Auto Scaling role to “Proceed without role” when the AWS administrator creates a cluster in the AWS console.

EC2 Configuration

The following table describes cloud provisioning configuration EC2 configuration properties:
Property
Description
EC2 Key Pair
EC2 key pair to enable communication with the EMR cluster master node.
Optional. This credential enables you to log into the cluster. Configure this property if you intend the cluster to be non-ephemeral.
EC2 Subnet
ID of the subnet on the VPC in which to create the cluster.
Use the subnet ID of the EC2 instance where the cluster runs.
Master Security Group
Optional. ID of the security group for the cluster master node. Acts as a virtual firewall to control inbound and outbound traffic to cluster nodes.
Security groups are created when the AWS administrator creates and configures a cluster in a VPC. In the AWS console, the property is equivalent to ElasticMapReduce-master.
You can use existing security groups, or the AWS administrator might create dedicated security groups for the ephemeral cluster.
If you do not specify a value, the cluster applies the default security group for the VPC.
Additional Master Security Groups
Optional. IDs of additional security groups to attach to the cluster master node. Use a comma-separated list of security group IDs.
Core and Task Security Group
Optional. ID of the security group for the cluster core and task nodes. When the AWS administrator creates and configures a cluster In the AWS console, the property is equivalent to the ElasticMapReduce-slave security group
If you do not specify a value, the cluster applies the default security group for the VPC.
Additional Core and Task Security Groups
Optional. IDs of additional security groups to attach to cluster core and task nodes. Use a comma-separated list of security group IDs.
Service Access Security Group
EMR managed security group for service access. Required when you provision an EMR cluster in a private subnet.

Azure Cloud Provisioning Configuration Properties

The properties in the Azure cloud provisioning configuration enable the Data Integration Service to contact and create resources on the Azure cloud platform.

Authentication Details

The following table describes authentication properties to configure:
Property
Description
Name
Name of the cloud provisioning configuration.
ID
ID of the cloud provisioning configuration. Default: Same as the cloud provisioning configuration name.
Description
Optional. Description of the cloud provisioning configuration.
Subscription ID
ID of the Azure account to use in the cluster creation process.
Tenant ID
A GUID string associated with the Azure Active Directory.
Client ID
A GUID string that is the same as the Application ID associated with the Service Principal. The Service Principal must be assigned to a role that has permission to create resources in the subscription that you identified in the Subscription ID property.
Client Secret
An octet string that provides a key associated with the client ID.

Storage Account Details

Choose to configure access to one of the following storage types:
The following table describes the information you need to configure Azure Data Lake Storage (ADLS) with the HDInsight cluster:
Property
Description
Azure Data Lake Store Name
Name of the ADLS storage to access. The ADLS storage and the cluster to create must reside in the same region.
Data Lake Service Principal Client ID
A credential that enables programmatic access to ADLS storage. Enables the Informatica domain to communicate with ADLS and run commands and mappings on the HDInsight cluster.
The service principal is an Azure user that meets the following requirements:
  • - Permissions to access required directories in ADLS storage.
  • - Certificate-based authentication for ADLS storage.
  • - Key-based authentication for ADLS storage.
Data Lake Service Principal Certificate Contents
The Base64 encoded text of the public certificate used with the service principal.
Leave this property blank when you create the cloud provisioning configuration. After you save the cloud provisioning configuration, log in to the VM where the Informatica domain is installed and run infacmd ccps updateADLSCertificate to populate this property.
Data Lake Service Principal Certificate Password
Private key for the service principal. This private key must be associated with the service principal certificate.
Data Lake Service Principal Client Secret
An octet string that provides a key associated with the service principal.
Data Lake Service Principal OAUTH Token Endpoint
Endpoint for OAUTH token based authentication.
The following table describes the information you need to configure Azure General Storage, also known as blob storage, with the HDInsight cluster:
Property
Description
Azure Storage Account Name
Name of the storage account to access. Get the value from the Storage Accounts node in the Azure web console. The storage and the cluster to create must reside in the same region.
Azure Storage Account Key
A key to authenticate access to the storage account. To get the value from the Azure web console, select the storage account, then Access Keys. The console displays the account keys.

Cluster Deployment Details

The following table describes the cluster deployment properties that you configure:
Property
Description
Resource Group
Resource group in which to create the cluster. A resource group is a logical set of Azure resources.
Virtual Network Resource Group
Optional. Resource group to which the virtual network belongs.
If you do not specify a resource group, the Data Integration Service assumes that the virtual network is a member of the same resource group as the cluster.
Virtual Network
Name of the virtual network or vnet where you want to create the cluster. Specify a vnet that resides in the resource group that you specified in the Virtual Network Resource Group property.
The vnet must be in the same region as the region in which to create the cluster.
Subnet Name
Subnet in which to create the cluster. The subnet must be a part of the vnet that you designated in the previous property.
Each vnet can have one or more subnets. The Azure administrator can choose an existing subnet or create one for the cluster.

External Hive Metastore Details

You can specify the properties to enable the cluster to connect to a Hive metastore database that is external to the cluster.
You can use an external relational database like MySQL or Amazon RDS as the Hive metastore database. The external database must be on the same cloud platform as the cluster to create.
If you do not specify an existing external database in this dialog box, the cluster creates its own database on the cluster. This database is terminated when the cluster is terminated.
The following table describes the Hive metastore database properties that you configure:
Property
Description
Database Name
Name of the Hive metastore database.
Database Server Name
Server on which the database resides.
Database User Name
User name of the account for the domain to use to access the database.
Database Password
Password for the user account.