Skip to content

Smj Spill To Disk Q1

Spark Scenario - SMJ Spills to Disk🔗

Suppose we have skewed data, and the data for the key does not fit in memory and spills to the disk, can we still join using SMJ because entire data wont fit in executor right?🔗

Context

Which Join Strategy can handle skewed join relatively well?

When data is skewed — that is, when one partition contains a disproportionately large amount of records — Shuffle Hash Join faces major challenges because it needs to build a hash table locally during the join, demanding a lot of extra memory and often running into out-of-memory errors. On the other hand, Sort Merge Join copes much better: it doesn’t require additional memory for hash tables, and if a partition is massive, Spark can sort and merge the data while spilling to disk if necessary, ensuring the job completes successfully. Based on this intuition, we can say, Sort Merge Join is generally the more reliable and scalable choice for handling skewed data in Spark.

Would the Join Fail?

Let’s break it down:

1. How SMJ works🔗

  • Both sides of the join are partitioned and sorted on the join key.
  • For each partition, Spark performs a merge-like scan (similar to merge sort).
  • Unlike Broadcast Hash Join (BHJ) or Shuffle Hash Join, SMJ does not require the entire dataset or entire partition for a key to fit in memory.

2. What happens with skew & large keys🔗

  • If a single key has an extremely large number of rows, Spark will try to process them together in a streaming fashion during the merge phase.
  • Spark spills sorted runs to disk if the in-memory buffer is not enough.
  • At join time, it merges these spilled runs from disk and streams through them.
  • This means SMJ can still succeed even if data for a single key does not fit entirely into executor memory.

3. But what’s the catch?🔗

  • Performance degradation: spilling means heavy disk I/O, and processing skewed keys becomes very slow.
  • OOM risk in extreme cases: if the skewed key is so large that even the streaming join structures (e.g., buffers, iterators) cannot handle it, Spark can still run into OutOfMemoryError.
  • Shuffle skew: the partition holding that skewed key will be disproportionately large, leading to stragglers.

4. Compared to other joins🔗

  • BHJ: would fail outright if the broadcast side cannot fit in memory.
  • Shuffle Hash Join: needs the entire build side of the hash map for that partition in memory → skewed data is more likely to OOM.
  • SMJ: more tolerant since it can spill and stream, but performance can still tank.

✅ So the answer: Yes, you can still join using SMJ even if data for a key does not fit into executor memory, because Spark will spill to disk and stream. But it will be very slow and possibly unstable if the skew is extreme.

“Sort-Merge Join in Spark is tolerant to skew because it doesn’t require all data for a key to fit in memory. It sorts partitions using ExternalSorter, which spills sorted runs to disk if they don’t fit. During the merge phase, Spark streams through both sides with iterators and uses spillable buffers for keys with large numbers of rows. This means the join can still succeed even if one key doesn’t fit in executor memory, but it will be very slow due to repeated spill/merge cycles.”

“SMJ will succeed even with skewed data because Spark spills and streams, but performance tanks. To handle this in practice, we enable AQE skew join optimization (spark.sql.adaptive.skewJoin.enabled=true), which splits large partitions at runtime. We can tune thresholds (skewedPartitionFactor, skewedPartitionThresholdInBytes) and increase shuffle partitions. If skew is extreme, I’d consider salting keys or broadcasting the smaller side. I always check the Spark UI task distribution to confirm skew.”