Data Integration

Do you use a cloud data warehouse as your primary destination?

This panel appears if you haven't selected a cloud data warehouse as your primary destination. Click Yes, let's go to select a primary cloud data warehouse. If you load data to different destinations, you can dismiss this panel and move it to the bottom of the page.

Primary task panels

Use the following panels to load and transform your data:

• - Ingest. Create a data loader task to load large data sets to a cloud data warehouse.
• - Transform. Create a data transfer task or mapping to load and transform your data.
• - Orchestrate. Create a taskflow to define the control flow logic for your tasks.

If you've chosen a primary cloud data warehouse, the panel names include the data warehouse type, for example, Ingest into Snowflake.

Recent assets, Recent jobs

These panels display the assets you've most recently modified and the jobs you've most recently run.

Secondary task panels

These panels allow you to perform other tasks like inviting users to join your organization, creating connections, or opening the Explore page.

Help panels

Use these panels to view getting started information or access the community for help.

Usage, Runtime environments

These panels allow you to access your usage metrics and view the status of the runtime environments available to you, respectively.

Try out our mapping templates!

Click this panel to create a mapping based on a pre-defined mapping template.

• •New. Create mappings that define how to transform your data and tasks that move and transform your data. Create reusable mapping components. Create taskflows that can run multiple tasks in a series or in parallel.
• •Home. Return to the Home page.
• •Explore. Create and manage projects and assets.
• •My Jobs. View the status of the jobs you’re currently running or have run.

Data integration tasks

Mapping tasks

Mapping tasks process data based on the data flow logic defined in a mapping.

A mapping reads data from one or more sources, transforms the data based on logic that you define, and writes it to one or more targets. Create a mapping when you need to augment or manipulate your data before you load it to a target. For example, if you need to aggregate data, calculate values, perform complex joins, normalize data, or route data to different targets, you can create a mapping to do this.

A mapping task runs the data flow logic that you've defined in the mapping. Choose this task type after you've created a mapping so that you can run the data flow logic defined in the mapping.

Data transfer tasks

Data transfer tasks move data from one or two sources to a target. You can also choose to sort and filter the data before you load it to the target.

Choose this task type when you want to transfer data from a source object, optionally add fields from a second source object, and write the data to a new or existing target object without changing the source data. For example, if you want to move customer records from an on-premises database table to a table in your cloud data warehouse, create a data transfer task.

Data loader tasks

Data loader tasks provide secure data loading from multi-object sources to corresponding objects in your cloud data warehouse. They can load data incrementally and provide support for schema drift.

To optimize performance, data loading occurs in parallel batches. To fine-tune the data, you can exclude certain objects and fields and also apply some simple filters. If your source data changes frequently, you can load only new and changed records each time the task runs.

Choose this task type when you need to ingest data as-is from multiple objects into your cloud data warehouse. For example, if you need to repeatedly load all the data from files an Amazon S3 bucket to corresponding tables in Snowflake Data Cloud, create a data loader task.

Data transformation options

Using a wizard

Select this option to create a data transfer task.

A data transfer task uses a step-by-step, wizard-based interface to transfer data from your source to your target. For example, you might create a data transfer task to transfer data from an on-premises database to your cloud data warehouse.

If you wish, you can augment the source data with data from a lookup source and also sort and filter the data before loading it to the target.

Using our mapping designer

Select this option to create a mapping using the Mapping Designer.

Create a mapping when you need flexibility in your sources, targets, and transformation options. A mapping can read and write to a wide variety of heterogeneous data sources. It also offers a large variety of data transformation options.

When you run a mapping , Data Integration processes the transformation logic. However, you can choose to push some or all the transformation logic to the source, to the target, or both.

Using our SQL ELT mapping designer

Select this option to create a mapping in SQL ELT mode using the Mapping Designer.

Create a mapping in SQL ELT mode when your source and target are in the same cloud ecosystem and you want to perform the data transformation entirely within the cloud ecosystem. For example, you want to read data from your Snowflake cloud data warehouse or data lake, load it to your Snowflake cloud data warehouse, and perform all of the data transformation within Snowflake.

When you run a mapping in SQL ELT mode, Data Integration translates the transformation logic into ecosystem-specific SQL statements and commands that run in the underlying cloud data warehouse.

Connectors

Notifications