AWS Bites

We take a deep dive into Amazon S3 Files, AWS's exciting new managed file system backed by S3! We kick things off by exploring why S3 isn't a traditional file system, covering everything from the lack of true directories and atomic renames to immutable objects and POSIX access control differences. We then walk through the existing solutions people have used to bridge that gap, like S3FS FUSE, MountPoint for S3, FSx for Lustre, and Storage Gateway. From there, we get into the heart of the episode: how S3 Files works, how to set it up, and how it uses EFS under the hood as a caching layer. We share our own real-world benchmarking results comparing S3 Files against various EFS configurations across Lambda and Fargate, and we discuss a real customer project where we put S3 Files to the test. We also cover the important caveats like eventual consistency, the 60-second write-back delay, the lack of cross-account bucket support, and the cost model so you can make an informed decision.Resources mentionedEpisode 124: S3 PerformanceEpisode 95: Mounting S3 as a FilesystemAmazon S3 FAQs: S3 FilesfourTheorem S3 Files demo code on GitHubAmazon documentation: Understanding how synchronization worksSponsor Thanks to fourTheorem for powering AWS Bites. We help teams build cloud systems that are simple, scalable, and cost effective. Visit fourtheorem.com.Chapters00:00 Introduction: Why S3 is amazing but not a file system, and what S3 Files promises to solve01:47 Why S3 is not a file system: no true directories, immutable objects, no atomic renames, expensive listings, and POSIX differences05:23 Existing solutions for mounting S3 as a file system: S3FS FUSE, Python fsspec, Hadoop S3A, MountPoint, FSx for Lustre, File Cache, and Storage Gateway07:16 How S3 Files works: NFS-based access, EFS caching layer, streaming from S3, and supported compute services like EC2, ECS, EKS, and Lambda09:49 Setting up S3 Files: buckets, file system resources, import and expiration rules, mount targets, access points, VPC requirements, and NFS port configuration13:42 S3 Files performance numbers from AWS documentation: throughput, IOPS, latency figures, and why real-world benchmarking is recommended15:39 Benchmarking S3 Files vs EFS configurations on Lambda and Fargate: small and large file reads and writes, memory/CPU impact, and key findings19:48 Downsides and limitations: NFS only, no hard links, no atomic renames, eventual consistency, the 60-second write-back delay, and large-scale rename performance warnings23:05 Real-world project experience: a SaaS multi-tenant architecture, cross-account bucket limitation discovered, and how the team worked around it27:52 Cost breakdown: EFS-equivalent cache pricing, S3 storage costs, reads from cache vs. S3 directly, and how S3 access tiers still apply29:50 Final recap and take: when S3 Files shines, when to be cautious, mixed access pattern warnings, and an invitation to share your own experiences33:42 ClosingSend us your AWS questions Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X/Twitter, Bluesky, or LinkedIn: Eóin: Bluesky | LinkedIn Luciano: X/Twitter | Bluesky | LinkedIn

We discuss common use cases and challenges for copying data between S3 buckets and S3-compatible object storage services. We share our experience building an open source Node.js CLI tool called S3-Migrate to efficiently migrate data with separate source and destination credentials. We cover performance considerations like streaming, chunk sizes, concurrency and parallelism.AWS Bites is brought to you in association with fourTheorem. If you need a friendly partner to support you and work with you to de-risk any AWS migration or development project, check them out at ⁠⁠⁠fourtheorem.com⁠⁠⁠In this episode, we mentioned the following resources:s3-migrate CLI tool: https://github.com/lmammino/s3-migrateDigitalOcean Spaces Object Storage: https://docs.digitalocean.com/products/spaces/Cloudflare R2: https://www.cloudflare.com/en-gb/developer-platform/products/r2/Backblaze B2: https://www.backblaze.com/cloud-storageWasabi Cloud Storage: https://wasabi.com/Linode / Akamai Object Storage: https://www.linode.com/products/object-storage/MinIO (Self-hosted S3-compatible storage): https://min.io/Basecamp / Hey’s move away from S3: https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563AWS re:Post - How to move objects between S3 buckets: https://repost.aws/knowledge-center/move-objects-s3-bucketAWS Labs - Utility for S3 Migration: https://github.com/awslabs/utility-for-s3-migrations3s3mirror (Java-based tool): https://github.com/cobbzilla/s3s3mirrorrclone S3 Support: https://rclone.org/s3/knox-copy (Ruby-based, deprecated): https://github.com/goodeggs/knox-copyFlexify.io (paid cloud migration service): https://flexify.io/Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠https://bsky.app/profile/eoin.sh⁠⁠ | ⁠⁠https://www.linkedin.com/in/eoins/⁠⁠- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠⁠⁠ | ⁠⁠https://bsky.app/profile/loige.co⁠⁠ | ⁠⁠https://www.linkedin.com/in/lucianomammino/

In this episode, we explore DuckDB, an open-source analytical database known for its speed and simplicity. Discover how DuckDB stands out in various applications and compare it to other tools like SQLite, Athena, Pandas, and Polars. We also demonstrate integrating DuckDB with AWS Lambda and Step Functions for serverless analytics.AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out ⁠fourtheorem.com⁠In this episode, we mentioned the following resources: Our `duck-query-lambda`, A Lambda runtime for DuckDB queries: https://github.com/fourTheorem/duck-query-lambda DuckDB's official website: https://duckdb.org/ LibSQL: https://github.com/tursodatabase/libsql Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠ | https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/

Ready to take your tech career to the cloud and build those awe-inspiring systems you see? Then you're in the right place. This episode of AWS Bites is your blueprint for becoming a successful cloud architect. We're not just going to talk about it; we'll show you what worked for us, sharing the critical skills you need, and a practical path to build your expertise. Whether you're a beginner or looking to take the next step, join us as we equip you with the knowledge and tools to make your mark as a cloud architect! In this episode, we mentioned the following resources: Google Cloud Architecture Definition: https://cloud.google.com/learn/what-is-cloud-architecture Market data about the Cloud Professional Services market: https://www.gminsights.com/industry-analysis/cloud-professional-services-market EP 91 - Our Journeys into Software and AWS: https://awsbites.com/91-our-journeys-into-software-and-aws/ AWS Well-Architected Framework: https://aws.amazon.com/architecture/well-architected/ EP 68 - Are you well architected?: https://awsbites.com/68-are-you-well-architected/ Cloud Design Patterns: https://learn.microsoft.com/en-us/azure/architecture/patterns/ The art of scalability (book): https://www.amazon.com/Art-Scalability-Architecture-Organizations-Enterprise/dp/0134032802 Enterprise integration patterns (book): https://www.amazon.com/Enterprise-Integration-Patterns-Designing-Deploying/dp/0321200683/ Designing Data-Intensive Applications (book): https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 AWS Networking Essentials (free guide): https://aws.amazon.com/getting-started/aws-networking-essentials/ Docker Curriculum (free): https://docker-curriculum.com/ How Linux works (book): https://www.amazon.com/How-Linux-Works-Brian-Ward/dp/1718500408/ Exercism coding challenges: https://exercism.org/ The tangled web (book): https://www.amazon.co.uk/Tangled-Web-Securing-Modern-Applications/dp/1593273886 Low Level YouTube Channel: https://www.youtube.com/lowlevellearning AWS - Best Practices for Security, Identity, & Compliance: https://aws.amazon.com/architecture/security-identity-compliance/ Supercommunicators (book): https://www.amazon.com/Supercommunicators-Unlock-Secret-Language-Connection/dp/0593862066/ An Elegant Puzzle (book): https://www.amazon.com/Elegant-Puzzle-Systems-Engineering-Management/dp/1732265186/ Staff Engineer (book): https://www.amazon.com/Staff-Engineer-Leadership-beyond-management/dp/1736417916/ EP 58 - What can kitties teach us about AWS: https://awsbites.com/58-what-can-kitties-teach-us-about-aws/ AWS User Groups: https://aws.amazon.com/developer/community/usergroups/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X/Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this pre-re:Invent 2024 episode, Luciano and Eoin discuss some of their favorite recent AWS announcements, including improvements to AWS Step Functions, Lambda runtime updates, DynamoDB price reductions, ALB header injection, Cognito enhancements, VPC public access blocking, and more. They share their thoughts on the implications of these new capabilities and look forward to seeing what else is announced at the conference. Overall, it's an exciting time for AWS developers with many new features to explore. Very important: no focus on GenAI in this episode :) AWS Bites is brought to you, as always, by fourTheorem! Sometimes, AWS is overwhelming and you might need someone to provide clear guidance in the fog of cloud offerings. That someone is fourTheorem. Check them out at ⁠fourtheorem.com⁠ In this episode, we mentioned the following resources: The repo containing the code of the AWS Bites website: https://github.com/awsbites/aws-bites-site Orama Search: https://orama.com/ JSONata in AWS Step Functions: https://aws.amazon.com/blogs/compute/simplifying-developer-experience-with-variables-and-jsonata-in-aws-step-functions/ EC2 Auto Scaling improvements: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-ec2-auto-scaling-highly-responsive-scaling-policies/ Node.js 22 is available for Lambda: https://aws.amazon.com/blogs/compute/node-js-22-runtime-now-available-in-aws-lambda/ Python 3.13 runtime: https://aws.amazon.com/blogs/compute/python-3-13-runtime-now-available-in-aws-lambda/ Aurora Serverless V2 now scales to 0: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-aurora-serverless-v2-scaling-zero-capacity/ Episode 95 covering Mountpoint for S3: https://awsbites.com/95-mounting-s3-as-a-filesystem/ One Zone caching for Mountpoint for S3: https://aws.amazon.com/about-aws/whats-new/2024/11/mountpoint-amazon-s3-high-performance-shared-cache/ Appending to S3 objects: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html 1 million S3 Buckets per account: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-up-1-million-buckets-per-aws-account/ DynamoDB cost reduction: https://aws.amazon.com/blogs/database/new-amazon-dynamodb-lowers-pricing-for-on-demand-throughput-and-global-tables/ ALB Headers: https://aws.amazon.com/about-aws/whats-new/2024/11/aws-application-load-balancer-header-modification-enhanced-traffic-control-security/ Cognito Managed Login: https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/ Cognito Passwordless Authentication: https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/ VPC Block Public Access: https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-block-public-access/ Episode 88 where we talk about VPC Lattice: https://awsbites.com/88-what-is-vpc-lattice/ Direct integration between Lattice and ECS: https://aws.amazon.com/blogs/aws/streamline-container-application-networking-with-native-amazon-ecs-support-in-amazon-vpc-lattice/ Resource Control Policies: https://aws.amazon.com/blogs/aws/introducing-resource-control-policies-rcps-a-new-authorization-policy/ Episode 23 about EventBridge: https://awsbites.com/23-what-s-the-big-deal-with-eventbridge/ EventBridge latency improvements: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-eventbridge-improvement-latency-event-buses/ AppSync web sockets: https://aws.amazon.com/blogs/mobile/announcing-aws-appsync-events-serverless-websocket-apis/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X/Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this episode, we discuss why IAM users and long-lived credentials are dangerous and should be avoided. We share war stories of compromised credentials and overprivileged access. We then explore solutions like centralizing IAM users, using tools like AWS Vault for temporary credentials, integrating with AWS SSO, and fully eliminating IAM users when possible. 💰 SPONSORS 💰 AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out ⁠⁠https://fourtheorem.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠. In this episode, we mentioned the following resources: Episode 118 "The landing zone: Managing multiple AWS accounts": https://awsbites.com/118-the-landing-zone-managing-multiple-aws-accounts/ Episode 96: "AWS Governance and Landing Zone with Control Tower, Org Formation, and Terraform" https://awsbites.com/96-aws-governance-and-landing-zone-with-control-tower-org-formation-and-terraform/ Datadog Security Report (IAM stats): https://www.datadoghq.com/state-of-cloud-security/ Credentials provider chain in the JavaScript SDK: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html Credentials provider chain in the AWS CLI: https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-authentication.html Episode 45 "What’s the magic of OIDC identity providers?": https://awsbites.com/45-what-s-the-magic-of-oidc-identity-providers/ Episode 112 "What is a Service Control Policy (SCP)?": https://awsbites.com/112-what-is-a-service-control-policy-scp Episode 115 "What can you do with Permissions Boundaries?": https://awsbites.com/115-what-can-you-do-with-permissions-boundaries/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this episode, we provided an overview of GitHub Action Runners and discussed the benefits of using self-hosted runners on AWS. We covered options including EC2 and CodeBuild for running GitHub Actions, compared pricing across solutions, and shared our hands-on experience setting things up. Overall, using AWS services can provide more control, lower latency, and cost optimization compared to GitHub hosted runners. 💰 SPONSORS 💰 AWS Bites is sponsored by fourTheorem, an Advanced AWS partner that works collaboratively with you and sets you up for long-term success on AWS. Find out more at ⁠fourtheorem.com⁠. The source code for the project we discussed is available on GitHub: ⁠fourTheorem/codebuild-gha-runners⁠! In this episode, we mentioned the following resources. ⁠Cloudonaut - Self-Hosted GitHub Runners on AWS⁠ ⁠AWS: Best Practices for Working with Self-Hosted GitHub Action Runners at Scale on AWS⁠ ⁠GitHub - philips-labs/terraform-aws-github-runner⁠ ⁠GitHub - garysassano/cdktf-aws-codebuild-github-runners-organization⁠ ⁠GitHub - machulav/ec2-github-runner⁠ ⁠AWS CodeBuild Managed Self-Hosted GitHub Action Runners⁠ ⁠HyperEnv - Self-hosted GitHub runners on AWS⁠ ⁠RunsOn - Self-hosted runners on AWS⁠ ⁠Actions Runner Controller for Kubernetes⁠ ⁠Biome⁠ ⁠SLIC Watch⁠ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠

In this episode, we had the pleasure to interview Farrah Campbell, head of modern compute community at AWS, prolific speaker, and former AWS Hero. We discussed Farrah's career journey from healthcare into tech, tips on public speaking, dealing with imposter syndrome, the pace of innovation in the cloud, and predictions for the future. Farrah shared personal stories and advice for getting started in tech and being an active member of the community. It was inspiring to hear from someone so passionate about helping others learn and grow. 💰 SPONSORS 💰 AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out https://fourtheorem.com⁠⁠⁠⁠⁠ . In this episode, we mentioned the following resources: Farrah's favourite AWS Bites episode with Jeremy Daly: https://awsbites.com/102-getting-ampt-with-jeremy-daly/ Farrah on X (Twitter): https://x.com/FarrahC32 Farrah on Linkedin: https://www.linkedin.com/in/farrahcampbell/ Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠