Amazon SageMaker Lakehouse Connector > Mappings for Amazon SageMaker Lakehouse > Sources for Amazon SageMaker Lakehouse
  

Sources for Amazon SageMaker Lakehouse

When you configure a mapping in advanced mode to use an Amazon SageMaker Lakehouse source, you can configure the source properties.
Specify the name and description of the Amazon SageMaker Lakehouse source. Configure the source and advanced source properties for the Amazon SageMaker Lakehouse object.
The following table describes the Amazon SageMaker Lakehouse source properties that you can configure in a Source transformation:
Property
Description
Connection
Name of the source connection.
You can select an existing connection, create a new connection, or define parameter values for the source connection property.
If you want to overwrite the source connection properties at run time, select the Allow parameter to be overridden at run time option.
Source Type
Type of the Amazon SageMaker Lakehouse source object.
You can choose from the following source types:
  • - Single Object. Select to specify a single Amazon SageMaker Lakehouse object.
  • - Parameter. Select to specify a parameter name. You can configure the source object in a mapping task associated with a mapping that uses this Source transformation.
Parameter
A parameter file where you define values that you want to update without the need to edit the task.
Select an existing parameter for the source object or click New Parameter to define a new parameter for the source object. The Parameter property appears only if you select parameter as the source type.
If you want to overwrite the parameter at run time, select the Allow parameter to be overridden at run time option.
When the task runs, the Secure Agent uses the parameters from the file that you specify in the advanced session properties.
Object
Source object for the mapping.
The following table describes the Amazon SageMaker Lakehouse query options that you can configure in a Source transformation:
Property
Description
Filter
Filters records based on the filter condition.
Click Configure to add conditions to filter records and reduce the number of rows that the Secure Agent reads from the source.
You can specify the following filter conditions:
  • - Not Parameterized. Use a basic filter to specify the object, field, operator, and value to select specific records.
  • - Completely Parameterized. Use a parameter to represent the field mapping.
  • - Advanced. Use an advanced filter to define a more complex filter condition that uses the Amazon SageMaker Lakehouse query format.
Sort
Not applicable.
The following table describes the Amazon SageMaker Lakehouse source advanced properties that you can configure in a Source transformation:
Property
Description
Iceberg Spark Properties
The Spark configuration properties as key-value pairs that you want to configure for the Iceberg tables at runtime.
Enter the properties in the following format:
<parameter name>=<parameter value>
If you enter more than one property, enter each property in a new line.
When you use the S3 Tables lakehouse pattern, you must specify the S3 table bucket ARN property in the following format:
TableBucketARN=arn:aws:s3tables:us-east-1:001234567890:bucket/sagemaker-s3tables
When the source and target are in different regions, you must specify the Amazon S3 bucket ARN property in the following format:
s3.access-points.<bucket name>=<S3 bucket ARN>
Pre-SQL
The SQL queries to run before reading data from Apache Iceberg tables.
Ensure that the SQL queries use a valid Spark SQL syntax.
You can enter multiple queries separated by a semicolon.
Post-SQL
The SQL queries to run after reading data from Apache Iceberg tables.
Ensure that the SQL queries use a valid Spark SQL syntax.
You can enter multiple queries separated by a semicolon.
Tracing Level
Sets the amount of detail that appears in the log file.
You can choose terse, normal, verbose initialization, or verbose data.
Default is normal.