Spark Stream: Read from Files

Getting Started

Create Spark Session

# Create the Spark Session
from pyspark.sql import SparkSession

spark = (
    SparkSession
        .builder
        .appName("Streaming Process Files")
        .config("spark.streaming.stopGracefullyOnShutdown", True)
        .master("local[*]")
        .getOrCreate()
)

References

• Structured Streaming Read from Files
• Structured Streaming for multi format scalable Data Ingestion Workloads