How to write Spark DataFrame to Avro Data File

Question

Hi Team,

I am new to Spark. I want to write Spark DataFrame to Avro Data File. How can I do that?

Answer 1

Hi@akhtar,

Since Avro library is external to Spark, it doesn’t provide avro() function on DataFrameWriter, hence we should use DataSource “avro” or “org.apache.spark.sql.avro” to write Spark DataFrame to Avro file as shown below.

$ df.write.format("avro").save("person.avro")