Skip to content

Data Developer & Engineer

Pipeline strategy

English
Thai

korawica/ddedocs

Data Developer & Engineer

korawica/ddedocs

Home
Home
- Methodology
- Requirement Gathering
- Transform Spec
- Daily Work
- Data Storytelling
- Abstraction
  Abstraction
  - Data Management
    
    Data Management
    
    Data Model
    
    Data Modeling
    
    Data Modeling
    
    Inmon
    Inmon
    
    Abstract
    
    Kimball
    Kimball
    
    Abstract
    
    Implement
    
    Slowly Changing Dim
    
    Rapidly Changing Dim
    
    Techniques
    
    Data Vault
    Data Vault
    
    Abstract
    
    Implement
    
    Buzz Act Schema
    
    Anchor
    
    One Big Table
    
    Data Integration
    
    Data Transformation
    
    Data Quality
    
    Data Storage
    Data Storage
    
    Data Warehouse
    
    Data Mart
    
    ODS
    
    Data Lifecycle
    
    Normalization
    
    De-Normalization
    
    SCD
  - Data Governance
    
    Data Governance
    
    Implement
    
    Data Quality Framework
    
    Modern Team
  - Data Architecture
    
    Data Architecture
    
    Data Lakehouse
    
    Data Lakehouse
    
    PBAC
    
    Modern Data Stack
    
    Secure Data Platform
    
    Event-Driven Data Architect
    
    Lambda & Kappa
  - Data Mesh
    
    Data Mesh
    
    Data as a Product
    
    Data Domain Usage Monitoring
  - DataOps
    
    DataOps
    
    Data Product
    
    Data CICD
  - Data Pipeline
    
    Data Pipeline
    
    Declarative
  - Data Observability
    
    Data Observability
    
    Data Orchestration
    
    Data Quality Metric
    
    Data Quality Pyramid
    
    Data Lineage
    
    Data Consistency
    
    Viable Monitoring System
  - Data Strategy
    
    Data Strategy
    
    Semantic Layer
    
    Data Driven
    
    Sensitive Data
- Advance
  Advance
  - MLOps
    
    MLOps
    
    Challenge
    
    CICD
  - Emerging Trends
    
    Emerging Trends
- Roles
  Roles
  - Lead Data Engineer
- Implement
  Implement
  - Data Framework
    Data Framework
    
    Pipeline Strategy
    
    Pipeline Strategy
    
    Metadata
  - Metadata
    Metadata
    
    Structures
    Structures
    
    ETL
    
    Master
    
    Transaction
Services
Services
- Cloud Provider
  Cloud Provider
  - Azure
    
    Azure
    
    OAuth
    
    VNet
    
    Storage
    
    KeyVaults
    
    ServiceBus
    
    Database
    Database
    
    Auth
    
    Monitoring
    
    Batch
    
    Batch
    
    Start Task
    
    Auto Scalable
    
    Run Pyspark
    
    Connections
    Connections
    
    Azure
    
    Google
    
    Dockerize
    Dockerize
    
    Docker
    
    Docker inside Node
    
    Function App
    Function App
    
    Introduction
    
    Function V2
    
    Dockerize
    
    Connections
    Connections
    
    Azure
    
    Databricks
    
    Databricks
    
    Init Script
    
    Mount Storage
    
    Secrets
    
    Connections
    Connections
    
    To Azure
    
    To Synapse
    
    To Google
    
    To AWS
    
    Unity Catalog
    Unity Catalog
    
    Setup
    
    Privileges
    
    Event Hubs
    
    Data Factory
    
    Data Factory
    
    Share IR
    
    Link Services
    
    Synapse
    Synapse
    
    Auth
    
    External Data Source
    
    Partition View
    
    Monitoring
    
    Low-Level Security
    
    Date & Timezone
    
    DevOps
    DevOps
    
    For Loop
    
    Multi Repo
    
    Self Hosted
    
    Fabric
    
    Fabric
  - AWS
    
    AWS
    
    IAM
    
    VPC
    
    IoT Core
    IoT Core
    
    Rules
    Rules
    
    to S3
    
    to Kinesis
    
    to Timestream Grafana
    
    S3
    S3
    
    Filter Content
    
    Trigger Lambda
    
    Transform Lambda
    
    EC2
    EC2
    
    Domain with Route53
    
    ECS
    ECS
    
    With Fargate
    
    Lambda
    
    Lambda
    
    With Docker
    
    CICD
    
    Step Functions
    Step Functions
    
    Getting Started
    
    State Machine Language
    
    Combine Parallel Results
    
    Glue
    
    Glue
    
    Data Quality
    
    With Iceberg
    
    Local Env
    
    Athena
    Athena
    
    With DeltaLake
    
    EMR
    EMR
    
    Compare Databricks
    
    Kinesis
    Kinesis
    
    Data Streams
    
    Data Firehose
    
    Secret
    Secret
    
    Across Account
  - Google
    
    Google
    
    OAuth
    
    OIDC
    
    Cloud Functions
    Cloud Functions
    
    To Managing Secrets
    
    BigQuery
    BigQuery
    
    Getting Started
    
    With Iceberg
    
    Utility Funcs
- Data Processing
  Data Processing
  - Databricks
    
    Databricks
    
    Dynamically Workflow
    
    With FastAPI to Serverless
    
    Custom Python Docker
    
    AWS Orchestration
    
    Deploy with AWS
    
    Custom Policy
    
    Row & Column Level Filter
    
    Data Quality
    
    Custom Spark Connector
    
    Workspace
    Workspace
    
    Migration Workspace
    
    Functional Workspace Organization
    
    UDF
    
    SQL Params
    
    SQL Script
  - Snowflake
    
    Snowflake
    
    Data Wash
- IaC & Infra
  IaC & Infra
  - Ansible
    
    Ansible
  - Terraform
    
    Terraform
    
    Manage Secret
    
    Providers
    Providers
    
    Databricks
    
    Azure Databricks
    
    AWS Glue
  - OpenTofu
    
    OpenTofu
  - Infisical
    
    Infisical
- Server & Container
  Server & Container
  - Server
    
    Server
    
    SSH
    
    SSL/TSL
    
    SFTP
  - Docker
    
    Docker
    
    Dockerfile
    
    Commands
    Commands
    
    Management
    
    Composes
    Composes
    
    Postgres
  - Kubernetes
    
    Kubernetes
    
    Pod Scheduling
    
    Networking
    
    RBAC
    
    State Phase
Tools
Tools
- Common
  Common
  - Git
    
    Git
    
    Scenarios
    
    Branching Strategies
    
    Commit Release
    
    Hooks
- Programing Langs
  Programing Langs
  - Shell
    
    Shell
    
    App
  - PowerShell
    
    PowerShell
    
    Batch File
    
    RestAPI
  - SQL
    
    SQL
    
    Optimizing SQL Queries
  - Python
    
    Python
    
    Wheel
    
    Sync Multi-processes
    
    Data Structure for DE
    
    Threading
    
    Libraries
    Libraries
    
    Pytest
    
    Pre-Commit
    
    Joblib
    
    Pydantic
    
    SQLAlchemy
    
    Functional Programing
    Functional Programing
    
    Monad
    
    Toolz
    
    Rust
    Rust
    
    With Rust
    
    Migration
    
    Versions
    Versions
    
    Python 3.12
  - GO
    
    GO
    
    Command
    
    Unittest
    
    Connect Database
    
    Tools
    Tools
    
    Connect Kafka
    
    Connect Redis
    
    Hexagonal Architect
  - Scala
    
    Scala
    
    Command
    
    Advance Feature
    
    Collection
    
    OOP Concept
  - Rust
    
    Rust
    
    Learning
    
    From Python
    
    CLI Application
Blogs
Blogs
- Datetime
  Datetime
  - March 2024
- Categories
  Categories
  - Knowledge

Pipeline strategy

This concept use pipeline to pass running date on all node processes.

core
 ╰─ pipeline
     ╰─ node-group
         ╰─ node