Skip to content
Data Developer & Engineer
To Kafka
Initializing search
korawica/ddedocs
Home
Services
Tools
Blogs
Data Developer & Engineer
korawica/ddedocs
Home
Home
Methodology
Requirements Gathering
Daily Work
Data Storytelling
Abstraction
Abstraction
Data Management
Data Management
Data Model
Data Modeling
Data Modeling
Inmon
Inmon
Abstract
Kimball
Kimball
Abstract
Implement
Slowly Changing Dim
Rapidly Changing Dim
Techniques
Data Vault
Data Vault
Abstract
Implement
Buzz Act Schema
Anchor
One Big Table
Data Integration
Data Transformation
Data Quality
Data Storage
Data Storage
Data Warehouse
Data Mart
ODS
Data Lifecycle
Normalization
De-Normalization
SCD
Data Governance
Data Governance
Implement
Data Quality Framework
Modern Team
Data Architecture
Data Architecture
Data Lakehouse
Data Lakehouse
PBAC
Modern Data Stack
Secure Data Platform
Event-Driven Data Architect
Lambda & Kappa
Data Mesh
Data Mesh
Data as a Product
Data Domain Usage Monitoring
DataOps
DataOps
Data Product
Data CICD
Data Pipeline
Data Pipeline
Declarative
Data Observability
Data Observability
Data Orchestration
Data Quality Metric
Data Quality Pyramid
Data Lineage
Data Consistency
Viable Monitoring System
Data Strategy
Data Strategy
Semantic Layer
Data Driven
Sensitive Data
Advance
Advance
MLOps
MLOps
Challenge
CICD
Emerging Trends
Emerging Trends
Roles
Roles
Lead Data Engineer
Services
Services
Cloud Provider
Cloud Provider
Azure
Azure
OAuth
VNet
Storage
Key Vaults
Database
Database
Auth
Batch
Batch
Start Task
Auto Scalable
Run Pyspark
Connections
Connections
Azure
Google
Dockerize
Dockerize
Docker
Docker inside Node
Function App
Function App
Function V2
Dockerize
Connections
Connections
Azure
Databricks
Databricks
Init Script
Mount Storage
Secrets
Connections
Connections
To Azure
To Synapse
To Google
To AWS
Unity Catalog
Unity Catalog
Setup
Privileges
Event Hubs
Data Factory
Data Factory
Share IR
Link Services
Synapse
Synapse
Auth
External Data Source
Partition View
Monitoring
Low-Level Security
Date & Timezone
DevOps
DevOps
For Loop
Multi Repo
Self Hosted
Fabric
Fabric
AWS
AWS
IAM
VPC
IoT Core
IoT Core
Rules
Rules
to S3
to Kinesis
to Timestream Grafana
S3
S3
Filter Content
Trigger Lambda
Transform Lambda
EC2
EC2
Domain with Route53
ECS
ECS
With Fargate
Lambda
Lambda
With Docker
CICD
Step Functions
Step Functions
Getting Started
State Machine Language
Combine Parallel Results
Glue
Glue
Data Quality
With Iceberg
Local Env
Athena
Athena
With DeltaLake
EMR
EMR
Compare Databricks
Kinesis
Kinesis
Data Streams
Data Firehose
Secret
Secret
Across Account
Google
Google
OAuth
Cloud Functions
Cloud Functions
To Managing Secrets
BigQuery
BigQuery
Getting Started
Data Processing
Data Processing
Databricks
Databricks
Dynamically Workflow
With FastAPI to Serverless
Custom Python Docker
AWS Orchestration
Deploy with AWS
Custom Policy
Row & Column Level Filter
Data Quality
Workspace
Workspace
Migration Workspace
Functional Workspace Organization
Snowflake
Snowflake
Data Wash
IaC & Infra
IaC & Infra
Ansible
Ansible
Terraform
Terraform
Manage Secret
Providers
Providers
Databricks
Azure Databricks
AWS Glue
Pulumi
Pulumi
OpenTofu
OpenTofu
Infisical
Infisical
Server & Container
Server & Container
Server
Server
SSH
SSL/TSL
SFTP
Docker
Docker
Dockerfile
Commands
Commands
Management
Composes
Composes
Postgres
Kubernetes
Kubernetes
Pod Scheduling
Networking
RBAC
State Phase
Tools
Tools
Common
Common
Git
Git
Scenarios
Branching Strategies
Commit Release
Hooks
Orchestration
Orchestration
Airflow
Airflow
Sensor
Repeatable DAGs
Cost Optimize
Pool
Unittest
Implements
Implements
CICD
On K8s
Connections
Connections
To Kafka
To DBT
To Minio
Dagster
Dagster
Dynamic Partition
Connections
Connections
To DLT
Bacalhau
Bacalhau
Ingestion
Ingestion
Airbyte
Airbyte
With Terraform
DltHub
DltHub
Extract & Transform & Load
Extract & Transform & Load
Pandas
Pandas
SQL Conversion
Polars
Polars
Data Pipeline
DeltaLake SCD2
Connections
Connections
To Synapse
To DeltaLake
DBT
DBT
Kimbal Modeling
LakeHouse
DBT Loom
DBT Mesh
Connections
Connections
To Synapse
To Trino
To Athena
Spark
Spark
IO
UDFs
RDD
RDD
Foreach & Foreach Partition
Map & Map Partition
Optimizations
Optimizations
Bucketing
Joining
Repartitioning
Nested Data Types
Serialization
Data Skew
Data Spill
Shuffling
Storage
Pyspark
Pyspark
Select & SelectExpr
Data Wrangling Functions
SCD2
RegExp
Media Files
Unittest
Avoid These at Any Cost
Structured Stream
Structured Stream
Aggregate
Read Files
Multi Query
ForEach Batch
Deploy
Deploy
On Local
On Docker
On Kubernetes
On Local with DataProc Serverless
Updated
Updated
Spark 3.4
Spark 3.4
Parameterised SQL
DuckDB
DuckDB
Database
Connections
Connections
To DeltaLake
Streaming
Streaming
Kafka
Kafka
With Zookeeper
Use-Cases
Use-Cases
Agoda
Flink
Flink
Open Table
Open Table
DeltaLake
DeltaLake
Deletion Vector
Liquid Clustering
Partition Z-Order Cluster
Universal Format
Merge
Pyspark API
Pyspark API
Auto Schema Evolution
Image Files
SCD2
Star Schema
Keeping Fast & Clean
Best Practice
Handling Concurrent Write
Stream Data
Iceberg
Iceberg
Reduce Full Scan
Small Files
Concurrent Write
With Pyspark
Hudi
Hudi
Storage
Storage
Hadoop
Hadoop
On Mac
MinIO
MinIO
Quality
Quality
Great Expectations
Great Expectations
With Databricks
With Spark
CICD & Monitoring
CICD & Monitoring
Jenkins
Jenkins
Jira
Jira
Vault
Vault
ML & BI
ML & BI
MLflow
MLflow
Pinot
Pinot
Trino
Trino
Programing Languages
Programing Languages
Shell
Shell
App
PowerShell
PowerShell
Batch File
RestAPI
SQL
SQL
Optimizing SQL Queries
Python
Python
Wheel
Sync Multi-processes
Data Structure for DE
Threading
Libraries
Libraries
Pytest
Pre-Commit
Joblib
Pydantic
SQLAlchemy
Functional Programing
Functional Programing
Monad
Toolz
Rust
Rust
With Rust
Migration
Versions
Versions
Python 3.12
GO
GO
Command
Unittest
Connect Database
Tools
Tools
Connect Kafka
Connect Redis
Hexagonal Architect
Scala
Scala
Command
Advance Feature
Collection
OOP Concept
Rust
Rust
Learning
From Python
CLI Application
Blogs
Blogs
Datetime
To Kafka
https://adzic-tanja.medium.com/etl-and-data-pipelines-using-airflow-and-kafka-0f82c186c97e
Back to top