As of 2026-09-03, Snowflake's external lineage feature reached General Availability, extending native lineage capabilities to include external data sources and destinations. This means your data team can now get a unified view of data flows, even when data moves between Snowflake and other systems. For many South African businesses, this isn't just a technical upgrade; it's a step towards more robust data governance and easier POPIA compliance, especially when dealing with data that originates outside your core Snowflake environment.
Most data strategies are a PDF. This feature helps you build a pipeline that actually shows you where your data has been. If you're a data engineer or platform lead, you know the headache of tracing data origins across a complex ecosystem. This update means less guesswork and more clarity. Here’s how to set it up.
Photo by Mekselina Güçer on Pexels.
Snowflake External Lineage allows you to track the journey of your data, not just within Snowflake itself, but also as it enters from and exits to external systems. It integrates with the OpenLineage standard, a vendor-agnostic specification for collecting and exchanging data lineage metadata.
This feature is about giving you a single pane of glass to understand your data's lifecycle.
| Feature | Native Snowflake Lineage (before GA) | Snowflake External Lineage (with OpenLineage) |
|---|---|---|
| Scope | Within Snowflake objects only | Within Snowflake + external systems |
| Integration | Automatic for Snowflake operations | OpenLineage-compatible events via REST API |
| Visibility | Snowsight, GET_LINEAGE function |
Snowsight, GET_LINEAGE function |
| POPIA Compliance | Helps with internal data audit | Enhances auditability for full data lifecycle |
| Use Cases | Data transformation, impact analysis | Data governance, regulatory compliance, debugging |
The ability to track data from its origin through various transformations is critical. For instance, if you're pulling customer data from an external CRM into Snowflake, processing it, and then pushing it to a Power BI dashboard, external lineage lets you see that entire flow. This is not just theoretical; it's about knowing exactly which source system contributed to a specific metric in your financial report.
Our experience with over 50 projects for clients like Hepstar and Glydepay has shown that clear data lineage is foundational. Without it, even the best AI agent just hallucinates confidently on bad data. This feature helps ensure the data feeding your analytics and AI models is trustworthy.
In South Africa, POPIA isn't an obstacle to automation — it's a design constraint that forces better systems. When we touch data, customer messaging, or AI, the compliance angle is stated up front. External lineage, by providing a transparent and auditable trail of data movement and transformation, directly supports POPIA compliance. You can demonstrate exactly where personal information originated, how it was processed, and where it ended up. This level of transparency is a differentiator in the South African market.
We build the data pipeline first. Skip it, and even the best AI agent just hallucinates confidently on bad data. This is why we prioritise clean, accessible data and robust lineage capabilities in all our data engineering projects.
Photo by panumas nikhomkhai on Pexels.
Deploying Snowflake external lineage with OpenLineage involves a few key steps:
INGEST LINEAGE privilege on your Snowflake account.openlineage-dbt package. Configure transport variables in a YAML file to specify the Snowflake external lineage endpoint and authentication details (e.g., private key path, user, account identifier).GET_LINEAGE SQL function to programmatically access the metadata.OpenLineage is an open standard for collecting and exchanging data lineage metadata. It's important for Snowflake because it allows you to track data flows from external systems into and out of Snowflake, providing a unified and comprehensive view of your data's journey, which is crucial for data governance and compliance.
Snowflake's external lineage feature requires an Enterprise Edition account or higher. This ensures you have the necessary capabilities and support for advanced data governance features.
Snowflake retains external lineage events for one year. This retention period allows for historical analysis and auditing of data flows over a significant timeframe.
Yes, as of September 2026, Snowflake supports capturing lineage between individual columns using the columnLineage facet within OpenLineage events, in addition to object-level lineage. This provides a more granular understanding of data transformations.
Snowflake supports various authentication methods for its external lineage endpoint, including Key Pair Authentication (recommended for automation), OAuth, and API Keys. Key Pair Authentication is generally preferred for secure, automated integrations.
External lineage enhances POPIA compliance by providing a transparent and auditable trail of data movement and transformation, even for data originating outside Snowflake. This allows businesses to demonstrate exactly where personal information comes from, how it's processed, and where it resides, supporting regulatory requirements.
Understanding your data's journey is no longer a luxury; it's a necessity for compliance, accuracy, and building reliable AI solutions. Snowflake's external lineage, combined with OpenLineage, provides the tools to achieve this.
Ready to implement robust data lineage and strengthen your data governance? Talk to us about how we can integrate this into your existing data architecture.
Avoid these 5 costly data engineering mistakes South African businesses make in 2026. Learn how to build clean, accessible data pipelines that deliver real…
Resilient data pipelines are critical for South African businesses. We build systems that adapt and recover, ensuring continuity despite local challenges.
Engineering data management is critical for South African firms. We cut through the hype to show how clean, accessible data drives real outcomes, not just…