Step 10. Create user defined security groups (optional)
You can create your own security groups if you don't want to use the default security groups.
Data Integration creates a network security group (NSG) by default when you create an advanced cluster on Azure. If your organization prefers to manage your own NSGs at the VNet level, you can create your own network security groups for advanced clusters on Azure.
Default network security groups for advanced clusters
Data Integration generates two network security groups (NSGs) for advanced clusters on Azure, one for master nodes and one for worker nodes. Understanding the security rules in these default NSG helps you define your own network security groups.
NSG for master nodes
Before you create your own custom NSG, it is helpful to understand the inbound and outbound rules.
The following image shows the default master node NSG:
Inbound rules
The following table describes the inbound rules for the NSG:
Rule
Description
SSH access
This rule has the IP address of the Secure Agent machine as the source. By default, SSH access is through port 22.
Apache Livy server access
This rule has the IP address of the Secure Agent machine as the source. By default, the Livy server access rule uses TCP port 31447. Data preview uses this rule.
Kubernetes API Server access
The Secure Agent uses this rule to access the Kubernetes API server to perform tasks such as deploying and monitoring Kubernetes applications and monitoring cluster resources.
Any Kubernetes client that is external to the advanced cluster also needs this rule to use the advanced cluster.
Other default inbound rules
The following default inbound rules also apply:
- Intra-VNet communication. Allows worker nodes to communicate with master nodes.
- Inbound traffic from the load balancer for distributing Kubernetes requests to the master node.
Outbound rules
Outbound rules allow outbound traffic to any nodes in the same VNet and to the internet. Data Integration needs access to various Azure services to support certain deployments.
Instead of using outbound rules to restrict outbound traffic to the internet, you can define firewall policies to validate outbound traffic. You can associate the subnet in which the advanced cluster is configured with a route table that routes all traffic to a firewall.
Using firewall policies is more flexible because the destination can be a domain, subdomain, or wildcard characters in the domain name. This allows you to create application rules for internet services with public IP addresses or a range of Azure services such as *.windows.net, *.azure.net, *.microsoft.com, and *.azure.com.
When both NSG rules and firewall policies exist, Data Integration considers both.
NSG for worker nodes
Use the rules from the default network security group for worker nodes to help you create your own custom NSG.
The following image shows the default worker node NSG, which uses public IP addresses:
Inbound rules
The following table describes the inbound rules for the NSG:
Rule
Description
SSH access
You need this rule only for troubleshooting. It isn't used by Data Integration.
For example, you can use this rule to pull logs from worker nodes. Configure this rule the same way as the NSG for master nodes.
Allow incoming traffic from TCP ports 10250, 10257, and 10259.
Outbound rules
The outbound rules for worker nodes are the same as for master nodes. Worker nodes access the same internet locations as master nodes plus additional locations such as external data sources.
Example using private clusters
The following image shows an example of NSGs for a worker node that is deployed in a private cluster, with more restrictive permissions:
User defined security groups in an advanced cluster on Azure
If you don't want to use the default network security groups, you can create your own.
To create your own network security groups on Azure, perform the following tasks:
1Configure a user-defined NSG.
2Ensure inbound rules apply for node recovery.
3Update permissions for agent and cluster roles.
Configure a user-defined NSG
You can replace the default NSGs generated by Data Integration with your own pre-existing NSGs for master and worker nodes in the advanced cluster.
To override the default NSGs with your own, perform the following steps:
1Open Administrator.
2Select Advanced Clusters.
3Select your cluster from the list.
4Select the Advanced Configuration tab for your cluster.
5Configure the following properties:
- Master Security Group ID
- Worker Security Group ID
Tip: If the advanced configuration includes the cluster resource group, and the NSG belongs to the cluster resource group, you can use <NSG name> as the value.
When you configure a user-defined NSG, consider the following rules and guidelines:
•The NSG must be in the same region as the cluster.
•The NSG can be in a different resource group from the cluster resource group.
•The master and worker nodes can share the same NSG, but this isn't recommended. The NSG for worker nodes typically allows more inbound traffic than it needs. For master nodes, the goal is to restrict inbound traffic to secure the node.
Ensure inbound rules apply for node recovery
When you create inbound security rules for a NSG, ensure that these rules are compatible with other Secure Agent machines.
When you use your own NSG, Data Integration can't modify the rules within the NSG. If the Secure Agent machine encounters an error and you need to restore the Secure Agent on another machine, ensure that all inbound security rules in the NSG for master nodes apply to the new Secure Agent machine.
To make the NSG resilient to changes to the Secure Agent machine, place the Secure Agent machine in a subnet within the same VNet as the cluster. You can specify the subnet's CIDR address as the source for these rules.
Update permissions for agent and cluster roles
Some permissions are no longer necessary for the agent and cluster roles if you use your own network security groups.
If you use your own NSGs in Azure, you can remove the following permissions from both the agent and cluster roles:
If the resource group that holds the NSGs differs from the cluster resource group, make sure that the NSG resource group allows the agent and cluster roles to read the security group and assign it to the cluster nodes. For example, the resource group QA_US_WEST holds some NSGs. The agent role in Data Integration needs an advanced cluster in a different resource group: YX-RESOURCE-GROUP.
To allow the agent role to access the resource group, create a custom role named k8s-cluster-resource-read under QA_US_WEST with the following role definition:
Assign the custom role to the agent’s managed identity and the cluster service principal in QA_US_WEST.
When you update permissions for agent and cluster roles, consider the following guidelines:
•Assign the agent's managed identity to the agent role at the Secure Agent machine level.
•Assign the cluster service principal to the cluster role at the cluster resource group level or subscription level if no cluster resource group is defined.
Troubleshoot cluster pre-validation failures
Knowing which conditions can cause pre-validation of clusters to fail can help you avoid problems later.
Cluster pre-validation fails when any of these conditions occur:
•The resource group can’t be identified for the NSG. For example, the NSG is configured without a resource group, and the advanced configuration doesn’t have the cluster resource group defined.
•The NSG is in a different region from the cluster.
•The NSG doesn’t exist in the resource group.
•The NSG for either the master or worker node isn't defined.
•Read and join permissions are missing on the NSG resource group in the agent or cluster role.
Cluster creation fails quickly in most of these error cases, but errors due to missing permissions might not appear until after a cluster node is created.
