Upload flat files to the EFS for data storage to use them as sources and targets in mappings and tasks. By defining access points and data storage configurations in Administrator, you can organize files, isolate data for different teams, and change physical file locations without breaking existing flat file connections.
The following table lists the path layers that an elastic runtime environment uses to process flat files:
Path layer
Origin
Example
EFS access point
Configured in AWS.
/california/sales
Source file path
Subfolder structure under the access point where files physically reside. Configured in the data storage properties in Administrator. Excludes the leading forward slash.
data
Physical EFS location
Comes from the following formula:
Access point + Source file path
/california/sales/data
File system mount point for the EFS
Local directory where the EFS is attached so that you can interact with it.
/mnt/shared_data
Projected file path
Path inside Kubernetes Pods where files are projected for processing. Comes from the following formula:
Mount point + Access point
Configured in the data storage properties in Administrator. Excludes the leading forward slash.
mnt/shared_data/california/sales
Pod container directory
Comes from the following formula:
/ + Projected file path
/mnt/shared_data/california/sales
Flat file connection directory
Configured in the flat file connection.
/mnt/shared_data/california/sales/data
For production workloads, upload flat files to the EFS for data storage using a dedicated staging server or your organization's standard file transfer process. If you mount the EFS on a staging node, you can create a file system mount point like /mnt/efs to mount the EFS instance, and then upload flat files under the mount point.
Informatica doesn't recommend mounting the EFS for data storage on the master node. Unlike system storage, the cluster installer doesn't create a mount directory for data storage on the master node. The master node rotates during release upgrades and environment configuration changes, which unmounts local connections.
For more information about flat file connections, see the help for the appropriate connector.
Uploading flat files for mappings and tasks
To use a flat file as a source in a mapping or task, upload the flat file to the EFS for data storage. Then, configure the data storage properties in Administrator.
1Upload your flat files to the EFS for data storage using SMB, NFS, or an external file transport mechanism.
You can organize flat files using any directory structure. For example, you can upload them to /data under the access point.
2In Administrator, edit the elastic runtime environment.
3In the Data Storage table, add a row and enter the following details:
- Type: Select EFS.
- File System ID: Enter the EFS file system ID.
- Access Point ID: Enter the access point ID.
- Source File Path: Enter the relative path under the access point with no leading forward slash. For example: data
- Projected File Path: Enter the file system mount point and access point, excluding the leading forward slash. For example, if the file system mount point is /mnt/shared_data and the access point is /california, enter mnt/shared_data/california.
Example: Multi-team access control
Enable secure, isolated, multi-team access to a shared EFS file system by defining team-specific EFS access points, flat file connections, and data storage properties in the elastic runtime environment.
An organization has two sales teams, California Sales and Oregon Sales. Both teams share the same EFS file system but must be strictly isolated to their own directories:
•California Sales data: /california/sales/data
•Oregon Sales data: /oregon/sales/data
To enforce boundary isolation so that the teams access only their own files, complete the following tasks:
1In AWS, create access points on the EFS with the following root directories:
- California access point: /california/sales
- Oregon access point: /oregon/sales
In the AWS Management Console, the root directories appear under the Path column for each access point.
2In IDMC, create flat file connections with the following directories for each regional team:
- California directory: /mnt/shared_data/california/sales/data
The elastic runtime environment performs the following actions:
•Reads and writes files at /california/sales/data and /oregon/sales/data on the EFS.
•Processes files at /mnt/shared_data/california/sales and /mnt/shared_data/oregon/sales on the Kubernetes Pods.
Because each EFS access point enforces its own permission sets, California Sales can access flat files only under /california/sales/data, and Oregon Sales can access flat files only under /oregon/sales/data.
Example: Migrate data without modifying connections or tasks
Migrate data to a new secure EFS access point and process the data in an elastic runtime environment without modifying existing connections or tasks.
An organization has an existing flat file connection that reads data from /mnt/legacy/sales/orders. The organization wants to migrate the data to a new secured access point, /ca_region, without changing existing connections or tasks.
To configure the elastic runtime environment to read data from the new location, complete the following tasks:
1In Administrator, edit the elastic runtime environment.
2Add the following row to the Data Storage table:
- Type: Select EFS.
- File System ID: Enter the EFS file system ID.
- Access Point ID: Enter the access point ID for /ca_region.
- Source File Path: orders
- Projected File Path: mnt/legacy/sales
The elastic runtime environment performs the following actions:
•Reads and writes files at /ca_region/orders on the EFS.
•Processes files at /mnt/legacy/sales on the Kubernetes Pods.
Because the elastic runtime environment projects flat files from /ca_region/orders to /mnt/legacy/sales, it allows existing connections and tasks to continue running without modification.