Data Pipelines: Efficient Data Movement and Transformation
Learn how data pipelines facilitate smooth data flow from sources to destinations, ensuring efficient management.
What is this page about?
An explainer of data pipelines, the automated workflows that extract data from source systems, transform it, and load it into destinations for analysis, opening with Uber's nightly cascade producing company-wide dashboards by morning. It covers the foundational ETL-versus-ELT decision, the anatomy of a modern pipeline (source connectors, streaming infrastructure, processing engines, storage, orchestration), batch versus streaming, ensuring reliability through idempotency, data-quality validation, monitoring, and incremental processing, and schema evolution, slowly changing dimensions, and governance at scale.
What has been corrected on this page?
Every accepted correction to this page is recorded with the exact change, so readers can see how the page improved over time.
-
Kafka development began ~2010 at LinkedIn and was open-sourced in 2011
BeforeCreated at LinkedIn in 2011 to handle
AfterCreated at LinkedIn around 2010 and open-sourced in 2011 to handle
Why: Kafka development began ~2010 at LinkedIn and was open-sourced in 2011. Verified fix already live in body; no leftover instances found in faq/excerpt/meta.
View the full record →
Who checked this page?
1 contributor has checked "Data Pipelines: Efficient Data Movement and Transformation" on When Notes Fly. Each name below links to that person's public CitePep profile, where every contribution they have made is listed with the exact change they proposed.