Question

what is the benefit of repartition(1) and coalesce(1). When we save data we use df.repartition(1).so how many partition it will create

Comments

In case you use .repartition(1) it will only create a single file per partition. the main benefit is, the less the number of file per partition, the higher the reading speed will be. However if the file size becomes more than or almost a GB, then better to go for 2nd partition like .repartition(2).

In case or repartition all data gets re shuffled. and all the files under a partition have almost same size.

by using coalesce you can just reduce the amount of Data being shuffled.