How to create Athena tables for dynamic S3 paths using AWS Crawler

0 votes

Below are given my S3 paths under which multiple folders are present. Each folder contains a CSV file each with a different schema.

The values within the curly braces {} will be dynamic.

s3://test_bucket/{val1}/data/{val2}/input/latest/

s3://test_bucket/{val1}/data/{val2}/input/archived/timestamp={val3}/

I want to create the Athena tables using AWS Glue Crawler. We can have a separate database for input_data both for current and archive.

The tables formed should be such that it's partitioned over val1 and val2 both for the current and archive. And, an additional partition should be present in the table, that is, val3, in the case of the archived.

Feb 16, 2022 in AWS by Rahul
• 2,080 points
1,096 views

1 answer to this question.

0 votes
The simplest and most efficient way to use partition projection. It speeds up the query processing of highly partitioned tables and automate partition management. Partition values and locations are calculated from configuration rather than the repository like AWS Glue Data Catalogue. In memory operations are faster than remote operations
answered Feb 16, 2022 by anonymous

Related Questions In AWS

+1 vote
3 answers

How to get ARN for s3 Bucket using aws cli .

An ARN is a non-opaque, constructible identifier, ...READ MORE

answered Aug 16, 2018 in AWS by Priyaj
• 58,100 points
5,608 views
0 votes
1 answer

How to create AWS S3 api logs using Cloud Trail?

CloudTrail basically captures the part of API calls ...READ MORE

answered Dec 6, 2018 in AWS by Nitesh
770 views
0 votes
1 answer

How to create a S3 bucket using AWS CLI?

You can use the following command:- C:\Users\priyj_kumar>aws s3api ...READ MORE

answered Feb 15, 2019 in AWS by Priyaj
• 58,100 points
6,109 views
0 votes
1 answer
0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14, 2019 in AWS by athenauserz
11,793 views
0 votes
2 answers

How to display just the name of files using aws s3 ls command?

aws s3 ls s3://<your_bucket_name>/ | awk '{print ...READ MORE

answered Mar 17, 2019 in AWS by anonymous
22,090 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,170 points
19,817 views
0 votes
1 answer

How to create new stack using create-change-set in AWS Cloudformation

The type of change set operation. To ...READ MORE

answered Sep 18, 2018 in AWS by Priyaj
• 58,100 points
3,902 views
0 votes
1 answer

How to use AWS S3 REST API ListObjects(v2) with multiple parent/child bucket directories?

Here you have to use the ListObjectsV2 ...READ MORE

answered Feb 15, 2022 in AWS by anonymous
1,299 views
0 votes
1 answer

Add a new item to a Dynamodb using a AWS lambda function each time a function is executed with Cloudwatch

The problem here is that the PutItem ...READ MORE

answered Feb 14, 2022 in AWS by Rahul
• 2,080 points
581 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP