Throttling S3 commands with AWS CLI

0 votes

I'm running a backup script using AWS CLI to perform an S3 sync command on my MediaTemple server.It was working properly for sometime , but I updated my Plesk installation and now when the backup script runs, MediaTemple disables my server due to excessive usage.

There is a limit on networking snapshots at the time they take the server offline which includes many open connections to Amazon IP addresses

Is there anything I can do to throttle the connections to AWS? Preferably I'm looking for an option within the AWS API , but other than that , is there something I can do on my end to manage the connections at the network level?

Aug 2, 2018 in AWS by datageek
• 2,540 points
1,229 views

1 answer to this question.

0 votes

The AWS CLI S3 transfer commands (which includes sync) have the following relevant configurations :

  • max_concurrent_requests -

    • Default: 10

    • The maximum number of concurrent requests.

  • multipart_threshold -

    • Default: 8MB

    • The size threshold the CLI uses for multipart transfers of individual files.

  • multipart_chunksize -

    • Default: 8MB

    • When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.

This isn't as efficient  as throttling packets per second, but it might help if you set concurrent request value and lower both multipart threshold and chunksize.

answered Aug 4, 2018 by Archana
• 4,170 points

Related Questions In AWS

0 votes
0 answers

Get ARN of S3 Bucket with aws cli

Is it possible to get the ARN ...READ MORE

Apr 21, 2022 in AWS by Rahul
• 9,680 points
903 views
+1 vote
3 answers

How to get ARN for s3 Bucket using aws cli .

An ARN is a non-opaque, constructible identifier, ...READ MORE

answered Aug 16, 2018 in AWS by Priyaj
• 58,020 points
5,645 views
0 votes
1 answer

AWS:Using S3 with EC2 instances

Yes, it can be used for instances ...READ MORE

answered Jun 26, 2018 in AWS by hemant
• 5,790 points
6,321 views
0 votes
1 answer
0 votes
1 answer

Invalid Bucket name when creating s3 bucket with AWS SDK

AWS provides naming standards when naming an ...READ MORE

answered Aug 30, 2018 in AWS by Priyaj
• 58,020 points
4,506 views
0 votes
1 answer

How to use AWS CLI with Elastic Beanstalk?

You have to create a source bundle ...READ MORE

answered Sep 28, 2018 in AWS by Archana
• 4,170 points
2,201 views
–1 vote
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 58,020 points
5,204 views
0 votes
1 answer

AWS S3 CLI : error while trying to copy files locally using terminal

For the first error you should add ...READ MORE

answered Aug 3, 2018 in AWS by Archana
• 4,170 points
11,405 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,170 points
19,855 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP