Table of Contents
Introduction to S3 and EC2 file copy
In this article, we are going to see how to copy files from ec2 instance to S3 bucket and vice versa, with the help of IAM roles and without having to use the AWS access key and secrets in the server.
In AWS technical terms.
Copying files from EC2 to S3 is called Uploading the file
Copying files from S3 to EC2 is called Downloading the files
The first three steps are the same for both upload and download and should be performed only once when you are setting up a new EC2 instance or an S3 bucket. the last and the fourth step is same except the change of source and destination.
Hope the following image helps you understand the difference and between upload and download.
Let see how to get both of this done.
Steps to copy files from EC2 instance to S3 bucket (Upload)
- Create an IAM role with S3 write access or admin access
- Map the IAM role to an EC2 instance
- Install AWS CLI in EC2 instance
- Run the AWS s3 cp command to copy the files to the S3 bucket
1. Create an IAM role with S3 write access or admin access
Login to your AWS Management Console and go to IAM.
Create a new IAM role with S3 Admin Access which can be lateral mapped to the EC2 instance for easy S3 and EC2 integration.
Refer the following GIF to know how to create a new IAM role for S3 access
2. Map the IAM role to an EC2 instance
Choose the EC2 instance you want to assign this IAM role to.
Click on Actions > Security> Modify IAM Role
Search for the IAM Role we have created in the previous step and select it and hit Save.
3. Install AWS CLI in EC2 instance
Login to your EC2 instance and follow the instructions corresponding to your Operating System and install the AWS CLI
Refer the following GIF to know how to install AWS CLI in Linux machine
to know more about AWS CLI commands and how to use them refer this article
4. Run the AWS s3 cp command to copy the files to the S3 bucket
Now as we have installed the aws command line you can simply use the following commands to copy the files to S3 Bucket from EC2.
# To List the S3 Bucket aws s3 ls s3://<S3bucketName># To copy the files from EC2 to S3aws s3 cp <Fully Qualified Local filename> s3://<S3BucketName>
Refer the following GIF to know how it works in real-time.
Hope this quick article helps you to copy files from S3 to EC2 instance directly without any programmatic access and Keys
Steps to copy files from S3 bucket to EC2 instance (Download)
To copy the files from S3 bucket to EC2 instance, you have to follow the same steps except changing the source and destination on the Fourth Step
- Create an IAM role with S3 write access or admin access
- Map the IAM role to an EC2 instance
- Install AWS CLI in EC2 instance
- Run the AWS s3 cp command to copy the files from S3 to EC2
To copy the files from S3 to EC2, Keep the source as the bucket URL and the destination to your local directory or filename
# To copy the files from S3 to EC2aws s3 cp s3://<S3BucketName> <Fully Qualified Local filename/Directory>
In the previous command, you can see the difference. Here the source is S3 Bucket URL and the destination is a local file name or directory name.
If you are interested to use S3 with ansible. refer this article.
How to use ansible with S3 – Ansible aws_s3 examples | Devops Junction
Hope it helps
What you think about this article ?. let me know in comments
Cheers
Sarav AK
Follow me on Linkedin My ProfileFollow DevopsJunction onFacebook orTwitterFor more practical videos and tutorials. Subscribe to our channel
Signup for Exclusive "Subscriber-only" Content
More from Middleware Inventory
AWS CLI EC2 Stop and Terminate command Examples
In AWS infrastructure, We create a lot of EC2 instances on demand and we tend to forget about the stopped instances considering that it is stopped and not being billed for. But the fact is that Even the stopped instances would cost you a couple of pennies in various forms…
AWS EC2 CLI List Examples - Describe instances | Devops Junction
AWS CLI is a very great help when it comes to efficiently managing your AWS Cloud Infrastructure and your EC2 instances. While we are managing our AWS Infrastructure, we cannot always afford to login to the AWS console all the time and it is not recommended from the security perspective…
Add SSH Key to EC2 instances with Ansible - Automated
Whether it is On-Prem (or) Cloud-hosted, A Typical Non-Containerized Enterprise infrastructure would have ample of Virtual Machines aka Servers [ Linux ] Let us suppose that you work for the DevOps team of a Big Organization where you manage 100+ ec2 instances. You have a new hire in your team…
Ansible EC2 Example - Create EC2 instance with Ansible
Infrastructure as Code is getting all attention it deserves and everyone is trying to find their way to the `Completely automated Infrastructure Provisioning & Management` While there are a lot of tools available now in the market starting from Terraform, AWS CloudFormation, Chef, Puppet, Salt Stack There are some differences…
Enabling Two Factor Authentication for EC2 SSH - AWS MFA Setup
Security is always a big thing. Protecting your data is very crucial when there are lot of people out waiting for a chance to exploit. In Enterprise Infrastructure. We have various levels of security as follows Internet Firewall and Gateway Load balancer Restrictions and Constraints Inbound and OutBound Proxy Servers…
