After you configure your host environment, you can set up and deploy your elastic runtime environment.
Create an elastic runtime environment
The installation wizard guides you through to creating your elastic runtime environment. Note that creating the environment is just one step within the entire deployment procedure.
Deploy an elastic runtime environment
Step-by-step process to deploy your elastic runtime environment.
Creating an elastic runtime environment
You can create as many elastic runtime environments as necessary, each with a different configuration.
1In Administrator, select Runtime Environments.
2Click New Runtime Environment.
3Enter the name of your elastic runtime environment.
4Select Elastic Runtime Environment as the environment type.
5Optionally, enter a description for your environment.
6Click Next.
7Select the services and connectors to enable for this runtime environment.
Note: You can use only the Data Integration service for an elastic runtime environment.
8Click Create.
9Enter your environment settings in the Environment Configuration tab. For more information, see Environment configuration.
10Optionally, configure your services in the Service Configuration tab. For more information, see Service configuration.
Services are designed to work well in their default configurations. You'll need to configure your services only if you're an advanced technical user with specific needs, or you've been directed to change service settings by Informatica Global Customer Support.
Service and connector assignment for elastic runtime environments
Your organization can enable and disable specific Informatica Intelligent Cloud Services and connectors that your organization is licensed to use for an elastic runtime environment.
You can perform the following actions:
Enable or disable Informatica Intelligent Cloud Services for an elastic runtime environment.
You can enable services when you want the elastic runtime environment to run the tasks, processes, and product features associated with the service. By default, when you create an elastic runtime environment, it contains no services or connectors. You then enable your licensed services and connectors as required.
Note: You can use only the Data Integration service for an elastic runtime environment.
Disable services when you do not want the elastic runtime environment to run the tasks, processes, or product features associated with the service. Any task, process, or product feature that uses the elastic runtime environment no longer runs.
For more information about the Secure Agent services that each service requires, see Secure Agent services.
Enable or disable connectors for an elastic runtime environment.
You can enable specific connectors when you want the elastic runtime environment to be able to communicate with cloud and on-premise applications, platforms, databases, and flat files.
Disable connectors when you do not want the elastic runtime environment to download the packages associated with the connectors. When you disable a connector, any connection that uses the elastic runtime environment no longer runs.
Enable or disable services and connectors for an elastic runtime environment on the Runtime Environments page:
Service assignment guidelines
Use the following guidelines when you enable and disable services for an elastic runtime environment:
•Before you disable a service, verify that no connection, task, or process that is using the elastic runtime environment requires the service.
If a connection, task, or process uses the elastic runtime environment and you disable a required service, the task or process can't run. For example, the connection for a mapping source uses runtime environment RuntimeEnv1. If you disable Data Integration Server on RuntimeEnv1, the mapping task fails at run time.
•Before you disable a service, verify that no feature that uses the elastic runtime environment requires the service.
If a feature uses the elastic runtime environment and you disable a required service, the feature can't be used.
•When you create a connection, select a runtime environment in which the required services are enabled.
•If you configure your organization to store connection properties locally, the Data Integration service must be enabled for the elastic runtime environment.
Connectors in an elastic runtime environment
When you create a connection, you can specify an elastic runtime environment.
You can use an elastic runtime environment configured in AWS for the following connectors:
•Amazon Athena
•Aurora
•Amazon DynamoDB V2
•Amazon Redshift V2
•Amazon S3 V2
•Box
•Concur V2
•Coupa V2
•Cvent
•Databricks Delta
•Db2 Warehouse on Cloud
•Elasticsearch
•Google Big Query V2
•Google Cloud Storage V2
•Jira
•Marketo V3
•Microsoft Azure Blob Storage V3
•Microsoft Azure Cosmos DB SQL API
•Azure Data Lake Store Gen2
•Microsoft Azure SQL Data Warehouse V3
•CDM Folders V2
•Microsoft Fabric Data Warehouse
•Microsoft Fabric Lakehouse
•Microsoft OneLake
•MongoDB
•MongoDB V2
•OData
•Odata Consumer
•OData V2 Protocol Reader
•PostgreSQL
•Redis
•REST V2
•Salesforce
•Salesforce Marketing Cloud
•ServiceNow
•Shopify
•Snowflake Cloud Data Warehouse V2
•Stripe
•SuccessFactors OData
•SuccessFactors SOAP
•SurveyMonkey
•WS Consumer
•Workday V2
•Xactly
•Xero
•Zendesk V2
Environment configuration
Configure specific details about your elastic runtime environment on the Environment Configuration tab.
The environment configuration deals with the infrastructure where the elastic runtime environment containers will be deployed. It also defines how these containers can take inputs (for example, associated EBS storage types and values) and write outputs to locations such as logs.
Configure information in the Platform Configuration and Advanced Configuration tabs as necessary. You must configure at least your platform.
Note: If you make changes to a cluster that's already deployed, the changes won't take effect until the cluster is redeployed.
Configure the platform
The following table describes the Platform Configuration fields:
Field
Description
Region
Region in which to create the cluster. Use the drop-down menu to view the regions that you can use.
Worker Instance Type
Instance type to host the worker nodes. Use the drop-down menu to view the instance types that you can use in your region.
To verify that the instance type that you select from the drop-down menu is supported in the selected availability zones and your AWS account, refer to the AWS documentation.
Worker Instance Profile
Instance profile to be attached to the worker nodes.
The name must consist of alphanumeric characters with no spaces. You can also include any of the following characters: _+=,.@-
Number of Worker Nodes (Min, Max)
Number of worker nodes in the cluster. Specify the minimum and maximum number of worker nodes.
Worker Node Idle Timeout
Amount of idle time before a worker node times out.
Enable High Availability
Select this check box to indicate that the cluster is highly available.
When you enable high availability, the Subnets IDs field is displayed.
For more information about high availability, refer to the Kubernetes documentation.
Subnets IDs
When high availability is enabled, enter two subnet names from different availability zones.
This field doesn't appear if high availability isn't enabled.
Note: Be sure to enter the subnet name and not the subnet ID.
EBS Volume Size
Size of the EBS volume to attach to a worker node for temporary storage during data processing.
Application and Secure Agent service logs are stored in System Storage. The following table describes the System Storage fields:
Field
Description
Type
Only EFS disks are supported.
File System
File system is the file system ID of the EFS disk.
Source Mount
File system path to be mounted in the elastic runtime environment.
Note: Exclude the initial forward slash. For example, if the path is /sysdisk, use sysdisk.
Access Point
ID of the Amazon EFS file system access point. The access point ensures isolation for tenants in a multi-tenant EFS file system.
To find the access point ID, use the alphanumeric value at the end of the ARN.
After an access point is set up, you can configure the file system policy to allow access only to the access point for the IAM role.
Data storage is where you store your environment-specific configuration input files for tasks that execute in the elastic runtime environment. The following table describes the Data Storage fields:
Description
Type
Only EFS disks are supported.
File System
File system is the file system ID of the EFS disk.
Source Mount
File system path to be mounted in the elastic runtime environment.
Note: Exclude the initial forward slash. For example, if the path is /datadisk, use datadisk.
Access Point
ID of the Amazon EFS file system access point. The access point ensures isolation for tenants in a multi-tenant EFS file system.
To find the access point ID, use the alphanumeric value at the end of the ARN.
After an access point is set up, you can configure the file system policy to allow access only to the access point for the IAM role.
Advanced Configuration tab
You can further customize your elastic runtime environments on the Advanced Configuration tab. The following table describes the advanced configuration properties:
Configuration
Description
Initialization Script Path
Ignore this property. An elastic runtime environment doesn't start using initialization scripts.
Cloud Ecosystem Tags
Define tags to describe the cloud ecosystem.
These are hyperscalar ecosystem tags that you can use to track the AWS compute and storage used by the elastic runtime environment infrastructure.
You need to define at least one tag, with any value.
Note: You can't create tags with the names "NAME" or "Name" because these labels are reserved for system use.
Runtime Properties
Configure properties to customize the cluster and the jobs that run on the cluster.
At least one tag is required, but the value can be empty.
Service configuration
Use the Service Configuration tab to modify the default values of the service properties or apply a custom configuration to an elastic runtime environment.
Note: Don't change any of the service properties unless you are an advanced technical user or you have been directed by Informatica Global Customer Support. Incorrect settings can affect the operation of your system. You can use the reset icon to reset all values to their initial system defaults.
Service Configuration
The list displays all the properties for the services in use in your elastic runtime environment.
Click the edit icon to change the value of any property or flag a property as sensitive. Click the save icon to save the changes.
Note: Your entries aren't validated, so check your values carefully before saving.
If Informatica Global Customer Support directs you to, you can add new service properties on the Custom Configuration tab. Add a new row for each property.
Deploying an elastic runtime environment
Complete the following tasks to deploy an elastic runtime environment. After the environment is deployed, you can use it in tasks, mappings, and connections.
Step 1. Clean up cluster (optional)
If you need to reinstall the elastic runtime environment after receiving an error, clean up Kubernetes resources in the AWS environment before reinstalling the elastic runtime environment.
On the jump host, run the following commands:
cd /tmp/src/ ./uninstall.sh rm $HOME/.kube/config
Note: The script doesn't remove any AWS resources that were provisioned during the last install.
Step 2. Download the installer on the jump host
Perform the following steps:
1Copy the private key from your local machine to the jump host by running the following commands:
Open the config.txt file by running the following command:
vi config.txt
The following table describes the values to update in the config.txt file:
Variable
Description
IDS_HOST
Update the IDS_HOST domain to the POD for your organization. For example: dm-us.informaticacloud.com. To locate your domain, locate your POD in POD Availability and Networking and copy the property from the Login URL field.
USER
IICS user name
PASS
IICS password
runtimeEnvironmentId
Elastic runtime environment ID.
To find the ID, navigate to the Runtime Environments page in Administrator, copy the elastic runtime environment ID from the URL.
For example, in the URL https://usw1.dmr-us.informaticacloud.com/cloudUI/products/administer/main/elastic-agent/KUBERNETES/0141GU25000000000002/overview, the elastic runtime environment ID is 0141GU25000000000002.
SECURITY_GROUP_NAME
Name of the the security group that allows access to the elastic runtime environment. For more information, see Step 1. Create AWS resources.
If you didn't create a security group and the cluster installer role has the necessary permissions, the cluster installer automatically creates a security group and the inbound and outbound rules and then continues the installation.
If you created a security group and the cluster installer role has the necessary permissions to modify the security group, the cluster installer adds any required inbound rules that are missing. If the cluster installer role doesn't have the necessary permissions to modify the security group, cluster installation stops. To continue the installation, edit the security group manually and add the required inbound rules For more information about the required inbound rules, see Step 1. Create AWS resources.
VPC_NAME
Name of the VPC that you created.
For reference, the config.txt file includes the following information:
# Distributed Configuration for Cluster Installer # ================================================= # Customize the following variables to match your environment. # # IDS_HOST # The host address or URL for the IDS service. export IDS_HOST="<POD URL like dm-us.informaticacloud.com>" # USER # Informatica Intelligent Cloud Services user name for the organization that you want to log in to. export USER="<IICS user name>" # PASS # Informatica Intelligent Cloud Services password. export PASS="<IICS password>" # runtimeEnvironmentId # A unique identifier for your runtime environment. export runtimeEnvironmentId="<elastic runtime environment ID from Administrator>" # PROXY_USER PROXY_HOST PROXY_PORT # Specify these values if your organization uses HTTP proxy for outbound communication. # You will be prompted to enter Proxy password by the cluster installation script if PROXY_USER is specified. export PROXY_USER= export PROXY_HOST= export PROXY_PORT= # # majorVersion # Major version number for your release. Update as needed. export majorVersion=202507
# Following variables are provided so that you can customize cluster creation as per your organization policies. # # KEY_PAIR_NAME and KUBE_CONFIG_SECRET_NAME # These values identify the key pair and its secret name used to access your cluster. # Name of the Key Pair that is used to login to the nodes in the cluster KEY_PAIR_NAME="idmc-elastic-rte-key-pair" KUBE_CONFIG_SECRET_NAME="idmc-elastic-rte-kube-config" # # ORG_ADMIN_CREDS_SECRET_NAME # The name of the secret that stores organization administrator credentials. ORG_ADMIN_CREDS_SECRET_NAME="idmc-elastic-rte-org-creds" # # SECURITY_GROUP_NAME # The security group name defined for access within your environment. SECURITY_GROUP_NAME="<security group name like sg_ert>" # # NODE_NAME_PREFIX # Prefix for naming the nodes in your cluster. NODE_NAME_PREFIX="idmc-elastic-rte" # # AGENT_APP_LAUNCH_TEMPLATE_NAME and AGENT_APP_ASG_NAME # Launch template and auto scaling group names for the agent application. AGENT_APP_LAUNCH_TEMPLATE_NAME="idmc-elastic-rte-agent-app-launch-tmpl" AGENT_APP_ASG_NAME="idmc-elastic-rte-agent-app-asg" # # JOB_NODE_LAUNCH_TEMPLATE_NAME and JOB_NODE_ASG_NAME # Launch template and auto scaling group names for job nodes. JOB_NODE_LAUNCH_TEMPLATE_NAME="idmc-elastic-rte-job-node-launch-tmpl" JOB_NODE_ASG_NAME="idmc-elastic-rte-jon-node-asg" # # CONTROL_PLANE_ELB_NAME # The load balancer name for the control plane in high-availability setups. # Used only when High Availability (HA) mode is enabled. CONTROL_PLANE_ELB_NAME="idmc-elastic-rte-control-plane-elb" # # efsNameTag # Tag for EFS (Elastic File System) shared storage to added upon creation export efsNameTag="idmc-elastic-rte-efs-name" # # IS_RUNNING_ON_MASTER # Set to false if this script is not executed on the master node; otherwise, leave as true. IS_RUNNING_ON_MASTER=true # # VPC_NAME # Provide the VPC name if the script is not running directly on the master node. VPC_NAME="<VPC name like vpc_ert>" # # resourceCreationLogFile # Log file name where created resources will be recorded. export resourceCreationLogFile="resource_creation_log.txt"
Step 4. Run the cluster installation script
Run the following command to run the cluster installation script:
nohup ./create_cluster_nodes_ha.sh &
You can monitor the installation process with the following command:
tail -f nohup.out
Note: If you have a proxy enabled, exclude nohup.out from the command to run the cluster installation script:
./create_cluster_nodes_ha.sh
Step 5. Mount the EFS on the master node
Mount the EFS onto the master node and grant access to the access point for both system storage and data storage. After you do this, you can use NFS and you don't need to install EFS.
To install the EFS utility on the master node:
1Run the following commands:
cd /tmp sudo yum install -y cargo rust openssl-devel sudo yum install -y git rpm-build make git clone https://github.com/aws/efs-utils cd efs-utils make rpm sudo yum install -y build/amazon-efs-utils*rpm
2Run the following commands:
sudo mkdir /opt/efs/ sudo chown cloud-user:cloud-user /opt/efs/ sudo mount -t efs -o tls,accesspoint=fsap-0e85eb677fecc5a3c fs-0136a7b011fd040f7:/ /opt/efs cd /opt/efs/
Step 6. Verify that the elastic runtime environment is running
To verify that the elastic runtime environment is running in your AWS environment, go to the Runtime Environments page in Administrator and drill down on the Secure Agent services in the elastic runtime environment. Verify that the Data Integration Server is running and that one or more instances are running.
Note: It might take a few minutes for the Data Integration Server to start up the instances.
