SQLInterpreter in Spark

Question

What's the role of SQL interpreter and Optimizer in Spark?

Answer 1

SQL Interpreter & Optimizer handles the functional programming part of Spark SQL. It transforms the DataFrames RDDs to get the required results in the required formats.

Some of it's Features are:

• Functional programming
• Transforming trees
• Faster than RDDs
• Processes all size data

e.g. Catalyst: A modular library for distinct optimization