Spark Dynamic Resource Allocation

Lecture 31 : Dynamic Resource Allocation

Cluster Configuration

There is no problem with above configuration.

But let's say another user comes and asks for more resources...

The red person can't get assigned any more memory since cluster is already full.

The resource manager works on FIFO process.

Now this small process that needs only 25GB may have to wait for hours.

In dynamic memory allocation, the data that is not used is released.

Resource Manager has no role, spark internal algo does this.

Let's say we have free 750GB and driver demands 500GB from resource manager but there might be other processes waiting for the memory so it may not get it.

We can use min executors and max executors to get around this. We set min executors in such a way that process does not fail.

Now let's say there is a process which has completed execution and so Dynamic Resource Allocator frees the data. But we want it for further calculations. Do we calculate it again? No.

We can use External Shuffle Service. This works independently on every worker node and data in this doesnt get deleted.

If executors idle from 60s then we release the data.

How does executor ask for resources?

If the executor does not get its required memory within 1 sec then it starts asking in two fold manner.

First it asks for 1GB then 2GB then 4GB and so on...

spark.scheduler.backlogTimeout = 2s the executor waits till 2s before asking for memory.

Parallel Execution and Multi Threading

When to avoid dynamic resource allocation?

For critical jobs that needs to be run within certain SLA avoid it.