Skip to content

Spark Transformations Actions

Lecture 9 : Transformations and Actions in Spark🔗

image

Types of Transformations🔗

  • Narrow Transformation
  • Wide Transformation

image

Example: image

Suppose data is of 200MB. 200MB / 128MB = 2 partitions

image

Let's say both partitions go to separate executors.

Q1 : Filtering Records image There is no data movement here.

Q2: Find Total Income of each employee image

One id = 2 record is in one partition and the other is in the second partition so we need to do wide transformation image

Data needs to be shuffled and records with same id must be moved to same partition.

  • filter,select,union etc are narrow transformations
  • join,groupby,distinct