Runtime Environments > Elastic runtime environments > Create an elastic runtime environment

Create an elastic runtime environment

Create an elastic runtime environment in Administrator.

1In Administrator, open the Runtime Environments page.

2Select New Runtime Environment.

3On the General page, enter the name of the elastic runtime environment.

4Select Elastic Runtime Environment as the environment type.

5Optionally, enter a description.

6Click Next.

7On the Enable Components page, select the services to enable.

8Select the connectors to enable for the elastic runtime environment.

For a list of connectors that an elastic runtime environment can use, see Connectors in an elastic runtime environment.

9Select additional services to enable, such as Secret Manager.

Note:

If you don't enable services and connectors when you create the elastic runtime environment, you can enable them later. For more information, see Working with elastic runtime environments. For instructions about setting up Secret Manager, see Organization Administration.

10Click Create.

11On the Environment Configuration tab, configure the elastic runtime environment properties.

Environment configuration

Configure the elastic runtime environment properties to determine how IDMC runs the Kubernetes cluster and processes data in the elastic runtime environment.

Platform

The following table describes the platform properties:

Property	Description
Region	Region in which to create the elastic runtime environment. Note: You can't update the region after you deploy the elastic runtime environment.
Worker Instance Type	Instance type to host the worker nodes. To verify that the instance type is supported in your AWS setup, refer to the AWS documentation. Note: The worker nodes for the Secure Agent services use the same instance type as the master node.
Worker Instance Profile	Instance profile to attach to worker nodes. The instance profile is the value at the end of the ARN.
Enable High Availability	Enables high availability for the elastic runtime environment. When high availability is enabled, the elastic runtime environment maintains an odd number of master nodes. Because you manually create one original master node as part of the setup, the environment creates an additional, even number of master nodes based on the number of subnets that you provide. For example, if you provide two subnets, the environment will have a total of three master nodes. Note: Worker nodes are always highly available and are created across multiple availability zones and subnets regardless of whether high availability is enabled. For more information about high availability, refer to the Kubernetes documentation.
Subnets	Subnet IDs from different regions when high availability is enabled. Enter the subnet ID for at least two subnets. For example: subnet-01234567890abcdef A master node is created in each subnet as long as the total number of master nodes is an odd number. Each master node in the environment, including the original master node, must be in a different region. The created master nodes use the same configuration as the original master node but have different IP addresses.
EBS Volume Size	Size of the EBS volume to attach to a worker node for temporary storage during data processing.
AMI ID	Amazon Machine Image ID. For example: ami-01234567890abcdef

Note:

An elastic runtime environment can become highly available to eliminate a single point of failure when the master node goes down. If you enable high availability and one master node goes down, other master nodes will be available and jobs can continue running.

When an elastic runtime environment is highly available, watch out for job failures in the following scenarios:

•If all master nodes go down, jobs will fail.
•If too many master nodes go down, the Kubernetes API server becomes unavailable. The threshold for the number of failures is (n+1)/2 where n is the number of master nodes. For example, if the cluster has 3 master nodes and 2 master nodes go down, the Kubernetes API server becomes unavailable and jobs fail on the cluster.

Job auto-scaling policy

An elastic runtime environment uses the following sets of worker nodes:

•Worker nodes that run jobs. The job auto-scaling policy determines the number of worker nodes that are created to run jobs in the organization.
•Worker nodes that run the Secure Agent services. This set of worker nodes scales automatically to accommodate high job volume.

Note:

Worker nodes that run jobs and worker nodes that run the Secure Agent services aren't tracked separately. The elastic runtime environment interface doesn't show these as separate, individually configurable node counts. Instead, it displays a single, combined worker node count.

If no jobs are submitted to the elastic runtime environment by the time it reaches the worker node idle timeout, the number of worker nodes is calculated using the following equation:

(min worker nodes in job auto-scaling policy) + (worker nodes for Secure Agent services)

Even if the minimum number of worker nodes in the job auto-scaling policy is set to zero, the environment has at least one worker node that runs the Secure Agent services.

The following table describes the properties in the job auto-scaling policy:

Property	Description
Min Worker Nodes	Minimum number of worker nodes.
Max Worker Nodes	Maximum number of worker nodes.
Worker Node Idle Timeout	Amount of idle time before a worker node times out.

System storage

The following table describes the system storage properties:

Property	Description
Type	Type of system storage. Select EFS.
File System ID	File system ID for the EFS file system.
Access Point ID	Access point ID for the EFS file system. The ID is the alphanumeric value at the end of the ARN.
Source File Path	Relative path under the access point where source files physically reside. Exclude the leading forward slash. Note: The source file path can't be empty.

Data storage

The following table describes the data storage properties:

Property	Description
Type	Type of data storage. Select EFS.
File System ID	File system ID for the EFS file system.
Access Point ID	Access point ID for the EFS file system. The ID is the alphanumeric value at the end of the ARN.
Source File Path	Relative path under the access point where source files physically reside. Exclude the leading forward slash. For example: data Note: The source file path can't be empty.
Projected File Path	Path that includes the file system mount point and the access point where files are projected in the runtime environment. For flat files, use the first part of the directory that you enter in the flat file connection. Exclude the leading forward slash. For example, a flat file connection accesses files at /mnt/shared_data/california/sales/data where /mnt/shared_data is the file system mount point, /california/sales is the access point, and data is the source file path. For the projected file path, enter mnt/shared_data/california/sales.

Note:

For more information about how data storage paths map to flat file connections, see Upload files to data storage.

Advanced configuration

The following table describes the advanced properties:

Property	Description
Cloud Ecosystem Tags	Tags for cloud resources. Add at least one tag. The NAME and Name tags are reserved for system use. Note: You can't update the tags after you deploy the elastic runtime environment.
Runtime Properties	Properties to customize the elastic runtime environment and the jobs that run in the environment.