Advanced Clusters > Advanced configurations > Microsoft Azure properties
  

Microsoft Azure properties

Create an advanced configuration to configure properties for an advanced cluster. The properties describe where you want to start the cluster on your cloud platform and the infrastructure that you want to use.
The basic properties describe the advanced configuration and define the cloud platform to host the advanced cluster. To configure the cluster, configure the platform, advanced, and runtime properties.

Basic configuration

The following table describes the basic properties:
Property
Description
Name
Name of the advanced configuration.
Description
Description of the advanced configuration.
Runtime Environment
Runtime environment to associate with the advanced configuration. The runtime environment can contain only one Secure Agent. A runtime environment cannot be associated with more than one configuration.
Cloud Platform
Cloud platform that hosts the cluster.
Select Microsoft Azure.
Private Cluster
Creates an advanced cluster in which cluster resources have only private IP addresses.
When you choose to create a private cluster, you must specify the VNet and subnet in the advanced properties.

Platform configuration

The following table describes the platform properties:
Property
Description
Region
Region in which to create the cluster. Use the drop-down menu to view the regions that you can use.
Master Instance Type
Instance type to host the master node. Use the drop-down menu to view the instance types that you can use.
The list of available instance types is filtered based on the minimum number of resources that the cluster requires.
Worker Instance Type
Instance type to host the worker nodes. Use the drop-down menu to view the instance types that you can use.
The instance types that you can use depend on your Azure account.
For information to verify that the instance type that you select from the drop-down menu is supported on your account, refer to the Microsoft Azure documentation.
Number of Worker Nodes
Number of worker nodes in the cluster. Specify the minimum and maximum number of worker nodes.
Enable Spot Instances
Indicates whether to use Spot Instances for worker nodes.
Spot Instance Price Ratio
Maximum percentage of On-Demand Instance price to pay for Spot Instances. Specify an integer value between 1 and 100.
Required if you enable Spot Instances. If you do not enable Spot Instances, this property is ignored.
Enable High Availability
Indicates whether the cluster is highly available. You can enable high availability only if the region has availability zones 1, 2, and 3. One master node is created in each availability zone.
Availability Zones
List of availability zones where cluster nodes are created. The list of availability zones is populated automatically based on the region.
If the region has availability zones 1, 2, and 3, worker nodes are created across the zones.
Azure Disk Size
Size of the Azure disk to attach to a worker node for temporary storage during data processing. The disk size scales between the minimum and maximum based on job requirements. The range must be between 80 GB and 16 TB.
By default, the minimum and maximum disk sizes are 100 GB.
Note: When the disk size scales down, the jobs that are currently running on the cluster might take longer to complete.
Cluster Shutdown
Cluster shutdown method. You can select one of the following cluster shutdown methods:
  • - Smart shutdown. The Secure Agent stops the cluster when no job is expected during the defined idle timeout, based on historical data.
  • - Idle timeout. The Secure Agent stops the cluster after the amount of idle time that you define.
Mapping Task Timeout
Amount of time to wait for a mapping task to complete before it is terminated. By default, a mapping task does not have a timeout.
If you specify a timeout, a value of at least 10 minutes is recommended. The timeout begins when the mapping task is submitted to the Secure Agent.
Resource Group (Storage)
Storage resource group that holds the staging and log storage accounts.
The resource group can be a maximum of 90 characters.
If you specify an initialization script path, the storage account that holds the init script must be part of the same resource group.
Staging Location
Location on Azure Data Lake Storage Gen2 to store staging data that is generated when you run jobs.
Use the format: abfs(s)://<file system>@<storage account>.dfs.core.windows.net/<folder path>
If encryption is enabled, specify the ABFSS protocol. Otherwise, specify the ABFS protocol.
Log Location
Location on Azure Data Lake Storage Gen2 to store logs that are generated when you run a job.
Use the format: abfs(s)://<file system>@<storage account>.dfs.core.windows.net/<folder path>
If encryption is enabled, specify the ABFSS protocol. Otherwise, specify the ABFS protocol.

Advanced configuration

The following table describes the advanced properties:
Property
Description
Resource Group (Cluster)
Cluster resource group that holds cluster resources. If you do not specify a resource group, the agent creates a resource group to populate with cluster resources.
The resource group can be a maximum of 90 characters.
Service Principal Client ID
Service principal that the agent uses to manage Azure resources.
Key Vault
Key vault that stores the service principal credentials.
Secret Name
Name of the secret that stores the service principal credentials.
VNet
Azure VNet in which to create the cluster. Use the format: resourceGroup/VNet. The VNet must be in the specified region.
If you choose not to create a private cluster, you don't need to specify a VNet. In this case, the agent creates a VNet on your Azure account based on the region that you select.
Optional if you use custom network security groups.
Subnet
Required when a VNet is specified. Subnet in which to create cluster nodes.
Optional if you use custom network security groups.
IP Address Range
CIDR block that specifies the IP address range that the cluster can use. The IP address range cannot overlap with the IP addresses of the subnets.
For example: 10.0.0.0/24
Required if you use an existing VNet. The IP address range is used as the CIDR for Kubernetes service IPs and is required due to the tight integration between Kubernetes networking and Azure's VNet infrastructure.
Optional if you use custom network security groups.
Initialization Script Path
Location on Azure Data Lake Storage Gen2 that stores the initialization script to run on each cluster node when the node is created.
Use the format: abfs(s)://<file system>@<storage account>.dfs.core.windows.net/<folder path>/file.sh
The script must be a bash script and it can reference other init scripts in the same folder.
Master Security Group ID
Security group that defines the inbound and outbound security rules for master nodes in the cluster. The Secure Agent attaches this security group to all master nodes in the cluster.
Use the format: <resource group name>/<NSG name>
The master security group can be a maximum of 155 characters.
Note: If the advanced configuration includes the cluster resource group, and the NSG (network security group) belongs to the cluster resource group, you can use the network security group name as the value.
This security group replaces the default master security group created by Data Integration. For more information, see the How-To article "Create user defined security groups in Azure".
When you specify a master security group, the worker security group is required.
Worker Security Group ID
Security group that defines the inbound and outbound security rules for worker nodes in the cluster. The Secure Agent attaches this security group to all worker nodes in the cluster..
Use the format: <resource group name>/<NSG name>
The worker security group can be a maximum of 155 characters.
Note: If the advanced configuration includes the cluster resource group, and the NSG (network security group) belongs to the cluster resource group, you can use the network security group name as the value.
This security group replaces the default worker security group created by Data Integration. For more information, see the How-To article "Create user defined security groups in Azure".
When you specify a worker security group, the master security group is required.
Azure Tags
Tags on Microsoft Azure to apply to cluster nodes. Each tag has a key and a value.
You can list a maximum of 30 tags. The Secure Agent also assigns default tags to cloud resources. The default tags do not contribute to the limit of 30 tags.
Note: Issues can occur when you override default tags. For more information, see Default tags for cloud resources.
Tags cannot include UTF-8 characters \u241e and \u241f that correspond to record and unit separators represented by ASCII control characters 30 and 31.

Runtime configuration

The following table describes the runtime properties:
Property
Description
Encrypt Data
Indicates whether temporary data on the cluster is encrypted.
Note: Encrypting temporary data might slow down job performance.
Runtime Properties
Custom properties to customize the cluster and the jobs that run on the cluster.

Validating the configuration

You can validate the information needed to create or update an advanced configuration before you save the configuration properties.
The validation process performs the following validations:
When you use managed identity as a Secure Agent credential, you need to add the key ccs.azure.k8s.prevalidation.agent.clientid to the runtime property in the advanced configuration.

Spot Instances

You can configure an advanced cluster to use Spot Instances to host worker nodes.
Spot Instances are spare compute capacity that cloud providers offer at a lower price than On-Demand Instances. This can result in significant cost savings when performing internal tests and debugging in development or QA environments. The performance of On-Demand and Spot Instances of the same instance type is similar.
Note: Spot Instances are not always available, and your cloud provider can interrupt running Spot Instances to reclaim the capacity. Therefore, you shouldn't use Spot Instances on strict SLA-bound jobs.
Spot Instances are most beneficial when the frequency of interruptions is under 5%. Use the Spot Instance advisor on AWS to see a list of instances with different levels of interruptions.
The following chart shows the potential savings between On-Demand and Spot Instances. The chart also shows the differences in savings with different levels of frequency of interruptions:
The bar chart shows the costs for On-Demand and Spot instances when the frequency of interruption is less than 5% and the costs when the frequency of interruption is greater than 20%.
In the chart, you can see that when the frequency of interruption is below 5%, Spot Instances can save you nearly 50% on the total cost compared to On-Demand Instances. However, when the frequency of interruption exceeds 20%, your savings drops to 36%.
When you use Spot Instances, you set a Spot Instance price ratio. The Spot Instance price ratio is the maximum price you will pay for Spot Instances as a percentage of the On-Demand Instance price. For example, if On-Demand Instances cost $0.68 an hour and you set the Spot Instance price ratio to 50, you will pay the current Spot Instance price as long as the price is $0.34 an hour or less.
The Secure Agent always creates a number of On-Demand worker nodes equal to the minimum number of worker nodes that you configure. When you enable Spot Instances and the cluster scales up, the agent tries to create additional worker nodes on Spot Instances up to the maximum number of worker nodes. If Spot Instances are not available or cost more than the maximum price you set, the cluster uses On-Demand Instances for the worker nodes.
For example, if you set the minimum number of worker nodes to 5 and the maximum to 8, the agent creates 5 nodes on On-Demand Instances and tries to create 3 nodes on Spot Instances. If you set the maximum number of worker nodes equal to the minimum, the cluster uses only On-Demand Instances.
If your cloud provider interrupts a Spot node that is running an advanced job, the agent uses On-Demand nodes to complete the job.

High availability

An advanced cluster can become highly available to eliminate a single point of failure when the master node goes down. If you enable high availability and one master node goes down, other master nodes will be available and jobs on the cluster can continue running.
When a cluster is highly available, watch out for job failures in the following scenarios:

Accessing a new staging location

If you plan to use a new staging location, the Secure Agent must be able to access the location before you update the location in the advanced configuration.
To use the new staging location, complete the following tasks:
  1. 1Update the permissions of the managed identity that is assigned to the Secure Agent machine.
  2. 2Edit the staging location in the advanced configuration.

Propagating tags to cloud resources

The Secure Agent propagates tags to cloud resources based on the Azure tags that you specify in an advanced configuration.
The agent propagates tags to the following resources:
If your enterprise follows a tagging policy, make sure to manually assign tags to other cloud resources.
Note: The Secure Agent propagates tags only to cloud resources that the agent creates. For example, if you create a VNet and specify the VNet in an advanced configuration, the agent does not propagate Azure tags to the VNet.

Default tags for cloud resources

In addition to the cloud platform tags that you specify in an advanced configuration, the Secure Agent assigns several default tags to cluster resources. Do not override the default tags.
The following table describes tags that the agent assigns to cluster resources:
Cloud platform tag
Description
infa:ccs:hostname
The host name of the Secure Agent machine that started the cluster.
If the Secure Agent machine stops unexpectedly and the Secure Agent restarts on a different machine, the host name is the original Secure Agent machine.
infa:k8scluster:configname
Name of the advanced configuration that is used to create the cluster.
infa:k8scluster:workdir
Staging directory that the cluster uses.
InfaInternalInitDone
Used internally.
KubernetesCluster
Identifies an advanced cluster.
Some default tags do not have a namespace and can conflict with the user-defined tags that you specify in an advanced configuration, such as KubernetesCluster. If you specify a user-defined tag with the same name, you might override the tag and issues can occur on the advanced cluster.

Data encryption

Encryption protects the data that is used to process jobs. You can use encryption to protect data at rest, temporary data, and data in transit.
Encryption is available for the following types of data:
Data at rest
By default, Azure encrypts staging data and log files. For more information, refer to the Microsoft Azure documentation.
For information about encrypting source and target data, see the help for the appropriate connector in the Data Integration help.
Temporary data
Temporary data includes cache data and shuffle data that cluster nodes generate.
To encrypt temporary data, enable encryption in the advanced configuration. If you enable encryption, temporary data is encrypted using the HMAC-SHA1 algorithm by default. To use a different algorithm, contact Informatica Global Customer Support.
Data in transit
By default, Azure uses the Transport Layer Security (TLS) protocol to encrypt data in transit to and from cloud storage, including staging data and log files.
When encryption is enabled, you can specify the ABFSS protocol when you configure the staging and log locations in an advanced configuration. If encryption is not enabled, you must use the ABFS protocol.