Python is not working when I try to read an excel file from S3 inside of an AI flow dag.
It is quite strange because it works when I read it using pd.read excel(s3 excel path) from outside airflow.
How I acted:
Create an AWS account in Airflow (this works well as I can list my s3 bucket)
In my Docker environment, where I run Airflow, instal pandas and s3fs.
Pd.read excel(s3 excel path) should be used to attempt to read the file.
I've tried it outside of Airflow, and it functions as stated.
Furthermore, even after waiting 20 minutes, nothing happens; the dag just keeps running indefinitely (at the stage where it is meant to read the file).
(I am attempting to read the file directly from S3 since I want to have to download it from S3, process it, and then upload it back to S3)
Note that I do not also deal with CSV.
EDIT: In a same vein, df.to csv('s3 path') in the airflow dag does not allow me to save my dataframe directly to S3, but it does in Python.