I have a dataframe which has one of the column called "Query" having the select statement present. Want to execute this query and create a new column having actual results from the query within the same dataframe.
+--------------+-----------+-----+--------------------------------+
|DIFFCOLUMNNAME|DATATYPE |ISSUE|QUERY |
+--------------+-----------+-----+--------------------------------+
|Firstname |StringType |YES |Select Firstname from TempView limit 1|
|LastName |StringType |NO |Select LastName from TempView limit 1 |
|Designation |StringType |YES |Select Designation from TempView limit 1|
|Salary |IntegerType|YES |Select Salary from TempView limit 1 |
+--------------+-----------+-----+--------------------------------+
When I try below
DF.withColumn("QueryResult", SparkConfiguration.sparkSession.sql(col("QUERY")))
Getting error as Type mismatch, Required String found column.
I am thinking I need to use UDF here. But not sure how to write and use. Please helpspark