How to assign a column in Spark Dataframe PySpark as a Primary Key

Question

I've just converted a glue dynamic frame into spark dataframe using the .todf() method. I now need to assign a column as the Primary Key. How do I do that? Please help!

Comments

What you could do is, create a dataframe on your PySpark, set the column as Primary key and then insert the values in the PySpark dataframe.

---

Hi Kalgi! I do not see a way to set a column as Primary Key in PySpark. Can you please share the details (code) about how that is done? Thanks!

Answer 1

spark do not have any concept of primary key. As spark is computation engine not database.

Hope this helps!

To know more about Pyspark, it's recommended that you join PySpark course online.

Thanks.

Comments

Yes I just read a few articles and came to the conclusion that you cannot set primary key in apache spark.