Step 3. Create the VPC and subnets (optional)
If you create your own VPC and subnets to host an advanced cluster, prepare the VPC and subnets according to cluster requirements.
Complete the following tasks:
- •Create subnets that support enough IP addresses to assist an elastic load balancer and the nodes in the advanced cluster.
- •Verify the routing configuration to make sure that the VPC and subnets can route requests in the cluster.
- •Accept inbound traffic on the Secure Agent machine so that the Spark driver can communicate with the Secure Agent.
Create subnets with enough IP addresses
Create subnets that support enough IP addresses to assist an elastic load balancer and the nodes in the advanced cluster.
For each subnet, calculate the number of required IP addresses according to the following guidelines:
- 1Add eight IP addresses to make sure that the elastic load balancer can scale properly.
- 2Add one IP address for the master node. If you want to use a cluster that is highly available, add 3 IP addresses instead.
- 3Add IP addresses equal to the maximum number of worker nodes.
For example, if the advanced cluster can have a maximum of 10 worker nodes, each subnet must support at least 19 IP addresses.
Verify the routing configuration
Verify that the VPC and subnets can route requests in an advanced cluster.
To make sure that the VPC and subnets can route requests, verify the following items on AWS:
- •The VPC contains all necessary networking components, including a route table, an internet gateway, and a network ACL.
- •DNS hostnames and DNS resolution are enabled.
- •The route table allows any EC2 instance to use the internet gateway that is attached to the VPC.
For more information, refer to the AWS documentation.
Accept inbound traffic
Accept inbound traffic on the Secure Agent machine so that the Spark driver can communicate with the Secure Agent.
Complete the following tasks:
- 1Add an inbound rule to the AWS security group that is attached to the Secure Agent machine.
- 2Specify the port 0-65535 to accept inbound traffic.
- 3Specify the VPC in CIDR notation.