A fully-managed cluster provides a serverless infrastructure that intelligently scales based on your workload and offers the lowest total cost of ownership for your organization.
The Secure Agent manages the entire Kubernetes lifecycle, including cluster startup, shutdown, auto-scaling, and upgrade. The agent manages the compute infrastructure and can create the advanced cluster using Spot Instances to further reduce costs for your organization.
A fully-managed cluster includes the following capabilities:
•The cluster scales based on the size of the workload and the resource boundaries that you specify. Jobs consume fewer resources during smaller workloads, and the cluster accommodates bursts in the processing load.
•The cluster consumes resources only while you're running jobs. The Secure Agent determines when to stop the cluster based on the cluster shutdown method that you select in the advanced configuration.
•CLAIRE®, Informatica's AI engine, uses machine learning to auto-tune the jobs that run on the cluster to achieve optimal job performance.
•A secondary tuning process on the cluster analyzes the data size in a mapping and the cluster capacity to further auto-tune the job.
•The cluster allows you to configure permissions to set access limits on the Secure Agent to your environment.
•High availability, recovery, and resilience ensure that jobs can continue running smoothly during interruptions.
•The data remains in your cloud environment.
Setting up cluster resources for a fully-managed cluster
In a fully-managed cluster, you set up some cluster resources like storage locations and roles, and Informatica creates the rest.
The following table lists the cluster resources that you can set up:
Cluster resource
Required or optional
Secure Agent machine, such as an EC2 instance or a Linux virtual machine where the Secure Agent is intalled
Required
Storage locations, such as locations on S3 or ADLS Gen2 for staging and log files, which include a storage account resource group on Microsoft Azure
Required
Cluster resources related to access permissions, such as IAM roles for cluster management, managed identities, service principals, and secrets in the Key Vault
Required
VPC and subnets, or VNet and subnets on Microsoft Azure
Optional
Security groups to attach to cluster nodes
Optional
Informatica creates and manages all other resources, including load balancers, Auto Scaling groups or Virtual Machine Scale Sets, and volumes and disks to attach to cluster nodes.
Creating a fully-managed cluster
When a developer runs a job, the Secure Agent uses the advanced configuration that’s associated with the job’s runtime environment to create a fully-managed cluster.
The agent performs the following tasks:
1Creates a cluster configuration that includes configuration information about the cluster. The configuration is stored using YAML files that the Secure Agent populates.
2Provisions the necessary resources to create a cluster.
Note: Informatica uses a secure pathway to fetch job-related container images for cluster nodes from the Informatica-specific JFrog repository. For clusters on Google Cloud, it also accesses the public internet to fetch files that are required to create the logical cluster layer on cluster nodes.
Submitting jobs to a fully-managed cluster
When the fully-managed cluster is running, the Secure Agent submits jobs to run on the cluster.
To submit a job to the cluster, the Secure Agent generates an execution plan that divides the data logic in the mapping into multiple Spark tasks. The cluster launches Spark drivers and Spark executors to process the Spark tasks simultaneously.
As developers run additional jobs, the cluster provisions and deprovisions resources to adapt to the size and number of jobs. For example, the cluster can provision additional cluster nodes and cluster storage during processing bursts.
Each job generates a session log, a Spark driver log, Spark executor logs, and an agent job log.
Stopping a fully-managed cluster
The Secure Agent stops a fully-managed cluster based on the cluster shutdown method that you select in the advanced configuration.
The Secure Agent can either wait to shut down the cluster after an idle timeout, or the agent can perform a smart shutdown that is based on historical data.
The Secure Agent also stops the cluster in the following situations:
•The cluster fails to start or fails to stop.
•The agent cannot reach the Kubernetes API server within a certain amount of time.
After the Secure Agent stops the cluster, the agent verifies that all cluster resources are deleted, except for some Informatica binaries that remain in the staging location in the infa_rpm.tar file. The binaries are required in order to run jobs on the cluster, and the file is reused the next time that the agent starts the cluster.
The agent deletes the infa_rpm.tar file in the following situations:
•You clear the runtime environment in the advanced configuration.
•You associate the advanced configuration with a different runtime environment.
•The agent process on the Secure Agent machine shuts down.
The agent restarts the cluster when a developer runs another job in the same runtime environment.