How to create Athena tables for dynamic S3 paths using AWS Crawler

0 votes

Below are given my S3 paths under which multiple folders are present. Each folder contains a CSV file each with a different schema.

The values within the curly braces {} will be dynamic.

s3://test_bucket/{val1}/data/{val2}/input/latest/ s3://test_bucket/{val1}/data/{val2}/input/archived/timestamp={val3}/


I want to create the Athena tables using AWS Glue Crawler. We can have a separate database for input_data both for current and archive.

The tables formed should be such that it's partitioned over val1 and val2 both for the current and archive. And, an additional partition should be present in the table, that is, val3, in the case of the archived.

Kindly help me with any approach I can take to set the configuration for creating tables dynamically. I would appreciate your time. Please let me know in case more information is needed.

Apr 29, 2022 in AWS by Soham
• 9,710 points

edited Mar 4 7 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP