Spark error throws stack overflow when union a lot

Question

Getting this error throws stackoverflow when union a lot of RDD. When I use "++" to combine a lot of RDDs, I got error stack over flow error.

This the the way I generated:

val collection = (for (
  path <- files
) yield sc.textFile(path)).reduce(_ union _)

Can anyone say how to resolve this?

Answer 1

Hey,

Use SparkContext.union(...) instead to union many RDDs at once

You don't want to do it one at a time like that since RDD.union() creates a new step in the lineage (an extra set of stack frames on any computation) for each RDD, whereas SparkContext.union() makes it all at once. This will ensure not getting a stack overflow error.

Since RDD.union() creates a new step in the lineage (an extra set of stack frames on any computation) for each RDD, whereas SparkContext.union() makes it all at once.