Spark Partitioning Bucketing

Lecture 15 : How to write data on the disk?

Modes to write data

Create three files

  write_df = read_df.repartition(3).write.format("csv")\
    .option("header", "True")\
    .mode("overwrite")\  # Using .mode() instead of .option() for overwrite mode
    .option("path", "/FileStore/tables/Write_Data/")\
    .save()