Beginning Apache Spark 3 Pdf Now
df.createOrReplaceTempView("sales") result = spark.sql("SELECT region, COUNT(*) FROM sales WHERE amount > 1000 GROUP BY region") This makes Spark accessible to analysts familiar with SQL. 4.1 Reading and Writing Data Supported formats: Parquet, ORC, Avro, JSON, CSV, text, JDBC, and more.
# Read df = spark.read.option("header", "true").csv("path/to/file.csv") df.write.parquet("output.parquet") 4.2 Common Transformations | Operation | Example | |------------------|-------------------------------------------| | Select columns | df.select("name", "age") | | Filter rows | df.filter(df.age > 21) | | Add column | df.withColumn("new", df.value * 2) | | Group and aggregate | df.groupBy("dept").avg("salary") | | Join | df1.join(df2, "id", "inner") | 4.3 Handling Missing Data df.dropna(how="any", subset=["important_col"]) df.fillna("age": 0, "name": "unknown") 4.4 User‑Defined Functions (UDFs) When built‑in functions are insufficient: beginning apache spark 3 pdf
df = spark.read.parquet("sales.parquet") df.filter("amount > 1000").groupBy("region").count().show() You can register DataFrames as temporary views and run SQL: They provide fault tolerance via lineage
from pyspark.sql import SparkSession spark = SparkSession.builder .appName("MyApp") .config("spark.sql.adaptive.enabled", "true") .getOrCreate() 3.1 RDD – The Original Foundation RDDs (Resilient Distributed Datasets) are low‑level, immutable, partitioned collections. They provide fault tolerance via lineage. However, they are not recommended for new projects because they lack optimization. COUNT(*) FROM sales WHERE amount >