Questions tagged [amazon-web-services]
Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.
72 questions
4
votes
2
answers
1k
views
Distributed lock system using AWS S3
I need a distributed lock system for a service I'm writing. (There can be multiple instances of this service running at a time, and I need to make sure that only one instance does a particular cron ...
- 181
4
votes
2
answers
188
views
Reading S3 files from two different buckets but adding it to one list variable
I have a below code which reads all the S3 files from a particular S3 bucket and then it adds all those files in a ...
- 43
2
votes
1
answer
224
views
Bash script to take user input to set AWS envrionment variable the first time and automatically in subsequent calls
Background
Our team has a utility called envmgr to set aws iam roles. The credentials are only good for 4 hours or so. As such, I have a number of scripts that call the command ...
- 361
2
votes
1
answer
467
views
Read bunch of files in parallel and populate concurrent map
I am trying to read bunch of S3 parquet files in parallel from a S3 bucket. After reading all these files, I am populating my products and ...
- 249
5
votes
1
answer
352
views
Rust CLI tool to synchronize files to S3
Learning Rust, I've written a small CLI tool which will
Fetch the existing data in S3
Iterate through local folders
Upload the files which don't exist in S3
I'd be keen to hear anything I might have ...
- 11.6k
7
votes
2
answers
600
views
A Hit Counter For AWS Python Lambda powered with DynamoDB
I'm learning how to use DynamoDB and various AWS services, so I decided to write a simple hit counter application for my education of how to use DynamoDB. This application deduplicates hits from the ...
- 925
1
vote
1
answer
130
views
Archiving MongoDB data to S3, with retryable steps
I have a python script to archive some data (export from mongo to a local json file and then upload it to s3)
Is there a better way to implement/improve the current flow? Each step should be be ...
- 203
3
votes
1
answer
394
views
Delete older EBS Volume snapshots except few with certain tags using Python, Lambda
I am learning Python and managed to accomplish the above. But my script doesn't seem optimal. Could someone help to take a review and suggest for betterment?
...
2
votes
1
answer
120
views
PHP filesystem on a shared file system
I have a project which we need to migrate from an on-prem solution to the AWS cloud. Currently, the software is running on a single instance. It's working fine but the business want the software to be ...
- 23
1
vote
1
answer
85
views
Cleaning Netflix-style recommendation engine datasets
I am working on a portfolio project that deals with creating a Netflix-style recommendation engine for movies. This code is currently ran locally but I will need to upload it into Jupyter Notebooks ...
- 11
3
votes
0
answers
106
views
Bash script for rebalancing weighted round robin
I have written a tool that is used to rebalance weights in a weighted round robin in AWS. I am quite happy with it whereas I would welcome feedback from any Bash programmers.
...
- 315
4
votes
0
answers
4k
views
Copying multiple S3 files in parallel using aws shell command
Any improvements to this welcome, I have got it working and I'm happy with it. I'm not really all that proficient with bash shell scripts though.
Problem:
AWS can copy multiple files pretty much as ...
- 143
4
votes
1
answer
81
views
scrape AWS ECS and store summary in S3
This is my scraper for AWS ECS (Elastic Container Service). My primary use for this is to grep through the resulting file to see which cluster a given service is ...
- 2,889
5
votes
1
answer
612
views
Designing a Simple Class that caches files from AWS
Background
I have a pipeline that I run and it spins up multiple EC2 instances that process many jobs. I noticed that in many parts of my pipeline i am downloading files from amazon s3 multiple times. ...
- 387
2
votes
0
answers
44
views
Code and Inputs on to use AWS temporary Credentials
I am adding below code for review as i am bit new to boto3. This Script adds tags (key and value) to All Volumes of EC2 listed in Excel. Is there any more effective way i could have written this? Also ...
- 21