Learn about scalable, cost-efficient, and extremely productive data integration using Advanced Pushdown Optimization.

The importance of using ELT

Traditional data warehousing can’t keep up with the demands of rapidly growing data volumes, processing workloads, and data analysis use cases. Many organizations are moving their data warehouses to cloud, as a first step in the data warehouse modernization process. Moving to cloud helps organizations around the globe to become more efficient, agile and prepared for the demands of the digital age.

Most of the large on-premises appliances and purpose-built small-to-midsize data warehouse vendors are transitioning to cloud. Top cloud service platform providers have their own data warehouse and data lake solutions. And there is an emergence of new out-of-the-box solution providers, like Snowflake. Organizations are modernizing their data platforms by migrating them to the cloud, as a first step, but also re-architecting the data load process to include a data lake stage. As data volumes increase, loading the raw data to a data lake provides scale, ease of access, low storage cost, and operational efficiency.

If you have large volumes of data coming from different sources and you want to make it available in a Snowflake data warehouse for your users to use for ad hoc analytics, real-time reporting or data monetization, you want to make sure you load trusted, high-quality data to Snowflake data warehouse. You want to perform these processes in the fastest possible way and at scale.

Organizations that want to derive the greatest value from Snowflake’s modern data warehouse need a high-performance, easy-to-use, cloud-native solution to integrate and deliver timely, trusted data to Snowflake.

Understanding ETL vs. ELT

Informatica supports both ETL as well as ELT on Snowflake’s Data Cloud and provides the option for users to choose based on use cases and needs.

Informatica’s AI-powered, best-in-class Snowflake Data Cloudconnector allows you to develop powerful data integration mappings from any of your on-premises and cloud sources to Snowflake’s data warehouse. It significantly reduces development costs and allows you to get to work within minutes.

Diagram of a modern unified cloud architecture with Informatica and Snowflake data warehouse and data lake | Informatica

ETL (Extract, Transform, and Load): The ETL approach enables processing and transforming external data from single or multiple sources into your data warehouse such as Snowflake Data Cloud. Informatica has been an industry leader in ETL for decades. Informatica data integration products are best-in-class for quickly integrating data and applications—wherever they reside—and load them into your cloud data store.

Once the data is in your data store, it is highly recommended to use an ELT approach where you can leverage the power of a modern data warehouse.

ELT (Extract, Load, and Transform) or Pushdown Optimization: Once the data is in the Snowflake data lake, the ETL pattern of moving data in and out of Snowflake Data Cloud and Snowflake’s data warehouse is inefficient, time consuming and resource heavy. It is not only slower but also makes your solution more expensive due to additional data egress charges by your data warehouse provider.

Informatica’s Advanced Pushdown Optimization optimizes the process by processing the data directly inside Snowflake, leveraging SQL layer for operations such as filter, join, aggregation, or sorting. The solution enables faster data processing and avoids unnecessary data movements.

When you use Informatica Cloud Data Integration to load data to Snowflake, you don’t need to use any native, proprietary data processing commands. You can use a simple, drag-and-drop UI to create your flow and choose to turn on pushdown optimization mode to execute the designed data pipeline.

Informatica’s Advanced Pushdown Optimization / ELT is designed for modern use case patterns in the cloud data lake and data warehousing world such as:

Classical data warehouse load patterns in cloud, such as Slowly Changing Dimensions (SCD)
Loading data from data lakes to a data warehouse while also transforming the data

In Advanced Pushdown mode, the engine translates the data pipeline to an optimized SQL statement or native Snowflake command and pushes the business transformation logic to the native Snowflake data warehouse, where the underlying database or application handles the allocation of resources and scheduling.

Why Use Advanced Pushdown Optimization / ELT?

You gain three key benefits:

Simplicity – Use a single connector for all pushdown optimization and advanced features. A simple drop-down option in the GUI means there’s no need to learn proprietary commands.
Productivity – Pushdown optimization improves processing speed by many times and significantly reduces operating costs. There are zero data egress charges when the source and target systems are in Snowflake’s data cloud.
Scale – With Informatica you can leverage the broadest array of connectors and deliver scale for data, engine, and users.

Check out this video for more details: Learn how to lower cost and speed up data processing with Advanced Pushdown Optimization (APDO).

Together, Informatica and Snowflake enable you to integrate, synchronize, replicate, and combine data from a broad range of data sources, while ensuring quality and consistency, and with the agility and scalability needed to provide universal access to analytics for a multi-cloud and serverless world.

Learn more:

The post Why You Should Use Informatica’s ELT or Advanced Pushdown Optimization for Snowflake appeared first on The Informatica Blog - Perspectives for the Data Ready Enterprise.