AWS Bites

We discuss common use cases and challenges for copying data between S3 buckets and S3-compatible object storage services. We share our experience building an open source Node.js CLI tool called S3-Migrate to efficiently migrate data with separate source and destination credentials. We cover performance considerations like streaming, chunk sizes, concurrency and parallelism.AWS Bites is brought to you in association with fourTheorem. If you need a friendly partner to support you and work with you to de-risk any AWS migration or development project, check them out at ⁠⁠⁠fourtheorem.com⁠⁠⁠In this episode, we mentioned the following resources:s3-migrate CLI tool: https://github.com/lmammino/s3-migrateDigitalOcean Spaces Object Storage: https://docs.digitalocean.com/products/spaces/Cloudflare R2: https://www.cloudflare.com/en-gb/developer-platform/products/r2/Backblaze B2: https://www.backblaze.com/cloud-storageWasabi Cloud Storage: https://wasabi.com/Linode / Akamai Object Storage: https://www.linode.com/products/object-storage/MinIO (Self-hosted S3-compatible storage): https://min.io/Basecamp / Hey’s move away from S3: https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563AWS re:Post - How to move objects between S3 buckets: https://repost.aws/knowledge-center/move-objects-s3-bucketAWS Labs - Utility for S3 Migration: https://github.com/awslabs/utility-for-s3-migrations3s3mirror (Java-based tool): https://github.com/cobbzilla/s3s3mirrorrclone S3 Support: https://rclone.org/s3/knox-copy (Ruby-based, deprecated): https://github.com/goodeggs/knox-copyFlexify.io (paid cloud migration service): https://flexify.io/Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠https://bsky.app/profile/eoin.sh⁠⁠ | ⁠⁠https://www.linkedin.com/in/eoins/⁠⁠- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠⁠⁠ | ⁠⁠https://bsky.app/profile/loige.co⁠⁠ | ⁠⁠https://www.linkedin.com/in/lucianomammino/

In this episode, we explore DuckDB, an open-source analytical database known for its speed and simplicity. Discover how DuckDB stands out in various applications and compare it to other tools like SQLite, Athena, Pandas, and Polars. We also demonstrate integrating DuckDB with AWS Lambda and Step Functions for serverless analytics.AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out ⁠fourtheorem.com⁠In this episode, we mentioned the following resources: Our `duck-query-lambda`, A Lambda runtime for DuckDB queries: https://github.com/fourTheorem/duck-query-lambda DuckDB's official website: https://duckdb.org/ LibSQL: https://github.com/tursodatabase/libsql Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠ | https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/

Ready to take your tech career to the cloud and build those awe-inspiring systems you see? Then you're in the right place. This episode of AWS Bites is your blueprint for becoming a successful cloud architect. We're not just going to talk about it; we'll show you what worked for us, sharing the critical skills you need, and a practical path to build your expertise. Whether you're a beginner or looking to take the next step, join us as we equip you with the knowledge and tools to make your mark as a cloud architect! In this episode, we mentioned the following resources: Google Cloud Architecture Definition: https://cloud.google.com/learn/what-is-cloud-architecture Market data about the Cloud Professional Services market: https://www.gminsights.com/industry-analysis/cloud-professional-services-market EP 91 - Our Journeys into Software and AWS: https://awsbites.com/91-our-journeys-into-software-and-aws/ AWS Well-Architected Framework: https://aws.amazon.com/architecture/well-architected/ EP 68 - Are you well architected?: https://awsbites.com/68-are-you-well-architected/ Cloud Design Patterns: https://learn.microsoft.com/en-us/azure/architecture/patterns/ The art of scalability (book): https://www.amazon.com/Art-Scalability-Architecture-Organizations-Enterprise/dp/0134032802 Enterprise integration patterns (book): https://www.amazon.com/Enterprise-Integration-Patterns-Designing-Deploying/dp/0321200683/ Designing Data-Intensive Applications (book): https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 AWS Networking Essentials (free guide): https://aws.amazon.com/getting-started/aws-networking-essentials/ Docker Curriculum (free): https://docker-curriculum.com/ How Linux works (book): https://www.amazon.com/How-Linux-Works-Brian-Ward/dp/1718500408/ Exercism coding challenges: https://exercism.org/ The tangled web (book): https://www.amazon.co.uk/Tangled-Web-Securing-Modern-Applications/dp/1593273886 Low Level YouTube Channel: https://www.youtube.com/lowlevellearning AWS - Best Practices for Security, Identity, & Compliance: https://aws.amazon.com/architecture/security-identity-compliance/ Supercommunicators (book): https://www.amazon.com/Supercommunicators-Unlock-Secret-Language-Connection/dp/0593862066/ An Elegant Puzzle (book): https://www.amazon.com/Elegant-Puzzle-Systems-Engineering-Management/dp/1732265186/ Staff Engineer (book): https://www.amazon.com/Staff-Engineer-Leadership-beyond-management/dp/1736417916/ EP 58 - What can kitties teach us about AWS: https://awsbites.com/58-what-can-kitties-teach-us-about-aws/ AWS User Groups: https://aws.amazon.com/developer/community/usergroups/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X/Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this pre-re:Invent 2024 episode, Luciano and Eoin discuss some of their favorite recent AWS announcements, including improvements to AWS Step Functions, Lambda runtime updates, DynamoDB price reductions, ALB header injection, Cognito enhancements, VPC public access blocking, and more. They share their thoughts on the implications of these new capabilities and look forward to seeing what else is announced at the conference. Overall, it's an exciting time for AWS developers with many new features to explore. Very important: no focus on GenAI in this episode :) AWS Bites is brought to you, as always, by fourTheorem! Sometimes, AWS is overwhelming and you might need someone to provide clear guidance in the fog of cloud offerings. That someone is fourTheorem. Check them out at ⁠fourtheorem.com⁠ In this episode, we mentioned the following resources: The repo containing the code of the AWS Bites website: https://github.com/awsbites/aws-bites-site Orama Search: https://orama.com/ JSONata in AWS Step Functions: https://aws.amazon.com/blogs/compute/simplifying-developer-experience-with-variables-and-jsonata-in-aws-step-functions/ EC2 Auto Scaling improvements: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-ec2-auto-scaling-highly-responsive-scaling-policies/ Node.js 22 is available for Lambda: https://aws.amazon.com/blogs/compute/node-js-22-runtime-now-available-in-aws-lambda/ Python 3.13 runtime: https://aws.amazon.com/blogs/compute/python-3-13-runtime-now-available-in-aws-lambda/ Aurora Serverless V2 now scales to 0: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-aurora-serverless-v2-scaling-zero-capacity/ Episode 95 covering Mountpoint for S3: https://awsbites.com/95-mounting-s3-as-a-filesystem/ One Zone caching for Mountpoint for S3: https://aws.amazon.com/about-aws/whats-new/2024/11/mountpoint-amazon-s3-high-performance-shared-cache/ Appending to S3 objects: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html 1 million S3 Buckets per account: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-up-1-million-buckets-per-aws-account/ DynamoDB cost reduction: https://aws.amazon.com/blogs/database/new-amazon-dynamodb-lowers-pricing-for-on-demand-throughput-and-global-tables/ ALB Headers: https://aws.amazon.com/about-aws/whats-new/2024/11/aws-application-load-balancer-header-modification-enhanced-traffic-control-security/ Cognito Managed Login: https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/ Cognito Passwordless Authentication: https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/ VPC Block Public Access: https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-block-public-access/ Episode 88 where we talk about VPC Lattice: https://awsbites.com/88-what-is-vpc-lattice/ Direct integration between Lattice and ECS: https://aws.amazon.com/blogs/aws/streamline-container-application-networking-with-native-amazon-ecs-support-in-amazon-vpc-lattice/ Resource Control Policies: https://aws.amazon.com/blogs/aws/introducing-resource-control-policies-rcps-a-new-authorization-policy/ Episode 23 about EventBridge: https://awsbites.com/23-what-s-the-big-deal-with-eventbridge/ EventBridge latency improvements: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-eventbridge-improvement-latency-event-buses/ AppSync web sockets: https://aws.amazon.com/blogs/mobile/announcing-aws-appsync-events-serverless-websocket-apis/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X/Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this episode, we discuss why IAM users and long-lived credentials are dangerous and should be avoided. We share war stories of compromised credentials and overprivileged access. We then explore solutions like centralizing IAM users, using tools like AWS Vault for temporary credentials, integrating with AWS SSO, and fully eliminating IAM users when possible. 💰 SPONSORS 💰 AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out ⁠⁠https://fourtheorem.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠. In this episode, we mentioned the following resources: Episode 118 "The landing zone: Managing multiple AWS accounts": https://awsbites.com/118-the-landing-zone-managing-multiple-aws-accounts/ Episode 96: "AWS Governance and Landing Zone with Control Tower, Org Formation, and Terraform" https://awsbites.com/96-aws-governance-and-landing-zone-with-control-tower-org-formation-and-terraform/ Datadog Security Report (IAM stats): https://www.datadoghq.com/state-of-cloud-security/ Credentials provider chain in the JavaScript SDK: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html Credentials provider chain in the AWS CLI: https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-authentication.html Episode 45 "What’s the magic of OIDC identity providers?": https://awsbites.com/45-what-s-the-magic-of-oidc-identity-providers/ Episode 112 "What is a Service Control Policy (SCP)?": https://awsbites.com/112-what-is-a-service-control-policy-scp Episode 115 "What can you do with Permissions Boundaries?": https://awsbites.com/115-what-can-you-do-with-permissions-boundaries/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this episode, we provided an overview of GitHub Action Runners and discussed the benefits of using self-hosted runners on AWS. We covered options including EC2 and CodeBuild for running GitHub Actions, compared pricing across solutions, and shared our hands-on experience setting things up. Overall, using AWS services can provide more control, lower latency, and cost optimization compared to GitHub hosted runners. 💰 SPONSORS 💰 AWS Bites is sponsored by fourTheorem, an Advanced AWS partner that works collaboratively with you and sets you up for long-term success on AWS. Find out more at ⁠fourtheorem.com⁠. The source code for the project we discussed is available on GitHub: ⁠fourTheorem/codebuild-gha-runners⁠! In this episode, we mentioned the following resources. ⁠Cloudonaut - Self-Hosted GitHub Runners on AWS⁠ ⁠AWS: Best Practices for Working with Self-Hosted GitHub Action Runners at Scale on AWS⁠ ⁠GitHub - philips-labs/terraform-aws-github-runner⁠ ⁠GitHub - garysassano/cdktf-aws-codebuild-github-runners-organization⁠ ⁠GitHub - machulav/ec2-github-runner⁠ ⁠AWS CodeBuild Managed Self-Hosted GitHub Action Runners⁠ ⁠HyperEnv - Self-hosted GitHub runners on AWS⁠ ⁠RunsOn - Self-hosted runners on AWS⁠ ⁠Actions Runner Controller for Kubernetes⁠ ⁠Biome⁠ ⁠SLIC Watch⁠ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this episode, we had the pleasure to interview Farrah Campbell, head of modern compute community at AWS, prolific speaker, and former AWS Hero. We discussed Farrah's career journey from healthcare into tech, tips on public speaking, dealing with imposter syndrome, the pace of innovation in the cloud, and predictions for the future. Farrah shared personal stories and advice for getting started in tech and being an active member of the community. It was inspiring to hear from someone so passionate about helping others learn and grow. 💰 SPONSORS 💰 AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out https://fourtheorem.com⁠⁠⁠⁠⁠ . In this episode, we mentioned the following resources: Farrah's favourite AWS Bites episode with Jeremy Daly: https://awsbites.com/102-getting-ampt-with-jeremy-daly/ Farrah on X (Twitter): https://x.com/FarrahC32 Farrah on Linkedin: https://www.linkedin.com/in/farrahcampbell/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠