TestGuild Automation Podcast

Everyone is talking about AI replacing testers, writing tests, and transforming software quality. But what if we're asking the wrong question? In this solo episode, Joe Colantonio shares a growing concern he's seen while traveling across the country for TestGuild IRL events: a decline in testing fundamentals at the exact moment AI hype is reaching a fever pitch. Drawing insights from Carissa Véliz's book Prophecy: Prediction, Power, and the Fight for the Future, Wayne Roseberry's work on AI and meaning, and Tariq King's concept of Human Experience Testing, Joe explores why AI systems may be far less intelligent than many believe, and why human testers remain more important than ever. You'll discover: ✅ Why large language models generate plausible answers without understanding truth ✅ The difference between prediction, correlation, and genuine understanding ✅ Why AI can test software but cannot experience software ✅ What "Everything is tested, but nothing is experienced" really means ✅ How AI hype may be distracting teams from critical testing fundamentals ✅ Why empathy, context, and human judgment are becoming competitive advantages for testers Whether you're excited about AI, skeptical of it, or somewhere in between, this episode will challenge you to think more deeply about the future of testing and your role in it. Resources Mentioned 📖 Prophecy: Prediction, Power, and the Fight for the Future by Carissa Véliz 📖 Work and presentations by Wayne Roseberry 🎓 Free course: thebullshitmachines.com 🎤 Learn more about TestGuild IRL events: TestGuild.com/irl If you enjoy this episode, be sure to subscribe, leave a review, and share it with a fellow tester who's trying to navigate the AI era without losing sight of the fundamentals. #SoftwareTesting #AI #QualityEngineering #TestAutomation #SoftwareQuality #HumanExperienceTesting #ArtificialIntelligence #TestGuild #QA #TechPodcast

What happens when AI agents can not only write mobile app code, but also validate their own work automatically? In this episode, I sit down with Maestro Co-founder and CEO Leland Takamine to explore one of the biggest shifts happening in software testing right now: agentic mobile testing. Leland shares how his team went from solving mobile performance testing challenges to building one of the fastest-growing mobile automation frameworks used by companies like Microsoft, Meta, Amazon, and DoorDash. We dive deep into: How AI coding agents are changing mobile testing workflows What "closing the agentic feedback loop" actually means Why deterministic testing still matters in the age of AI How Maestro MCP lets AI agents validate mobile app changes automatically Why mobile test maintenance may finally become manageable The future role of testers as AI-generated code explodes Leland also gives a live demo showing an AI agent building, validating, debugging, and generating a reusable mobile test completely autonomously. If you care about AI testing, mobile automation, MCP servers, or the future of QA engineering, this episode will likely change how you think about testing workflows over the next few years. Try it out now for yourself: Maestro Studio: https://testgld.link/mstudio Maestro MCP docs: https://testgld.link/maestromcp

AI-powered testing tools promise faster automation and less maintenance, but most require teams to abandon their existing frameworks. In this episode, we explore Alumnium, an opensource AI-native end-to-end testing solution created by Alex Rodionov, an engineer at Airbnb and a tech lead on the Selenium project. Instead of replacing tools like Playwright or Selenium, Alumnium adds an AI layer on top, helping teams: Reduce test maintenance by removing brittle locators Build more resilient, self-healing tests Write less code while increasing coverage Run tests across web and mobile using intent-based steps We also go beyond the hype and break down what actually matters for real teams: Why AI-driven tests can still become flaky The performance and cost tradeoffs of LLM-based execution What "context rot" is—and how it impacts long test runs How to safely introduce AI into existing test suites without rewriting everything Check it out now: https://testguild.me/alumAI

AI coding tools are helping teams move faster than ever, but there's a hidden cost. In this episode, we break down new insights from a DevOps industry report revealing a growing "velocity paradox": teams are shipping more code, but experiencing more failures, rollbacks, and burnout. You'll discover why AI adoption is heavily skewed toward coding, but not testing, pipelines, or observability, and how that imbalance is creating fragile systems that break under pressure. More importantly, you'll learn what high-performing teams are doing differently to maintain quality while scaling speed. What You'll Discover: ✔️ Why AI is increasing deployment failures (and how to stop it) ✔️ The "velocity vs quality" trap hurting modern DevOps teams ✔️ How to reduce flaky tests and pipeline instability ✔️ Why observability and feature flags are now critical, not optional ✔️ Practical ways to improve your CI/CD pipeline for AI-driven development ✔️ The role of QA engineers in the age of AI (and why it's growing, not shrinking) If you're a tester, automation engineer, or DevOps leader trying to keep up

Mobile test automation is still one of the biggest bottlenecks in modern software delivery. In this interview, QApilot's Co-founder Aditya Challa explains why most AI testing approaches fail and how to fix them. Learn more about QApilot: https://links.testguild.com/flutterqa If your mobile tests are flaky, slow, or hard to trust, you're not alone. Most teams are trying to apply LLM-based AI to problems that actually require deterministic reliability—and that's where things break down. In this video, you'll learn: Why mobile test automation breaks at scale The real issue with "99% accurate" AI in testing LLMs vs deterministic AI (and why it matters for mobile apps) How flaky tests destroy confidence in your pipeline How QApilot approaches mobile testing differently What reliable, scalable mobile automation should look like What this means for you: Fewer false positives, faster releases, and mobile tests you can actually trust. 00:00 Why Mobile Test Automation Is Still Broken 01:10 QApilot Overview 01:51 Why Mobile Testing Tools Fail 03:13 Why Appium Isn't Enough 05:09 QApilot's Approach to Mobile Testing 07:10 Scaling Mobile Testing Across Devices 08:02 Autonomous Testing + Human in the Loop 10:55 How QApilot Works (Architecture + Agents) 13:45 Real Example: Mobile App Crawling in Action 16:31 Finding Bugs Automatically (Performance + Accessibility) 18:52 Device Farms & Real Device Testing 21:50 Future of Mobile Testing (SRE + AI + Quality Layer) 27:06 Real Customer Results & Case Study 31:02 Why QApilot Focuses Only on Mobile 34:04 Where QApilot Fits in CI/CD 36:00 How to Try QApilot + Final Advice

What if your production logs could automatically generate new test cases? In this episode, Joe Colantonio sits down with Tanvi Mittal to break down how AI-powered log mining is changing the way teams approach software testing, quality engineering, and DevOps. Most teams ignore production logs or use them only for debugging. But those logs contain real user behavior, real failures, and real edge cases—the exact scenarios your test suite is probably missing. 👉 Learn how to: Convert production logs into automated regression tests Use AI to detect real-world failure patterns Apply shift-right testing to catch bugs earlier (and smarter) Handle the challenge of testing non-deterministic AI systems Reduce flaky tests and automation debt with real data If you're working with Playwright, Selenium, Cypress, or AI-driven testing tools, this episode will give you a completely new way to think about test coverage.

What does it really take to build a test automation tool that millions of testers rely on, without venture capital, paid ads, or a massive team? In this episode, we explore how SelectorsHub grew into one of the most widely used productivity tools in software testing, reaching over 1.6 million testers worldwide. You'll discover: How to build test automation tools that solve real QA pain Why community-driven development beats chasing funding How to prioritize features when you have thousands of users Whether AI testing tools will replace selector-based automation How to choose between Playwright vs Selenium using automation analysis What founders and QA leaders can learn from scaling without VC If you're an automation engineer, QA lead, DevOps professional, or tool builder looking to scale smarter, this episode delivers real-world insight without hype. Whether you're building frameworks internally or launching your own automation product, you'll walk away with a clearer strategy for solving problems testers actually care about.

Is traditional performance testing becoming obsolete? In this episode, performance engineering expert Akash Thakur shares why AI is fundamentally transforming load testing, scripting, observability, and shift-left strategies. With 17 years of real-world enterprise experience, Akash explains how AI-augmented tools are already reducing scripting time by 30%, improving analysis speed, and helping teams move from reactive performance testing to predictive intelligence. You'll learn: How AI is accelerating performance scripting and analysis Why shift-left performance testing is finally becoming realistic The role of structured data in predictive QA models How to test AI applications (LLMs, GPUs, inference throughput) differently than traditional web apps What the future role of performance engineers looks like — architect, not script writer If you're a performance tester, SRE, QA leader, or DevOps engineer wondering how AI will impact your role — this episode gives you practical, actionable insights you can apply immediately.

Performance testing often fails for one simple reason: teams can't see where the slowdown actually happens. In this episode, we explore Locust load testing and why Python-based performance testing is becoming the go-to choice for modern DevOps, QA, and SRE teams. You'll learn how Locust enables highly realistic user behavior, massive concurrency, and distributed load testing — without the overhead of traditional enterprise tools. We also dive into: Why Python works so well for AI-assisted load testing How Locust fits naturally into CI/CD and GitHub Actions The real difference between load testing vs performance testing How observability and end-to-end tracing eliminate guesswork Common performance testing mistakes even experienced teams make Whether you're a software tester, automation engineer, or QA leader looking to shift-left performance testing, this conversation will help you design smarter tests and catch scalability issues before your users do.

This is a special end-of-year episode of the Automation Testing Podcast. With family in town and a busy holiday season, Joe didn't want to skip a week without checking in and saying thank you to the TestGuild community. In this short episode, Joe shares: A huge milestone as the podcast approaches its 13-year anniversary Why the Automation Testing Podcast is moving from Sundays to Tuesdays starting in 2026 How loyal listeners can still get $100 off a full 5-day Automation Guild 2026 pass A sneak peek at TestGuild IRL — live, in-person events coming next year Gratitude for the listeners, YouTube community, and sponsors who make TestGuild possible If you're a software tester, automation engineer, or QA leader looking ahead to 2026, this episode lays out what's coming — and how to stay connected. Discount code: 100GUILDCOIN (https://testguild.me/podag26) Questions or ideas? Email Joe directly at joe@testguild.com As always — test everything, and keep the good.

What if understanding your codebase was no longer a blocker for great testing? Most testers were trained to work around the code — clicking through UIs, guessing selectors, and relying on outdated docs or developer explanations. In this episode, Playwright expert Ben Fellows flip that model on its head. Using AI tools like Cursor, testers can now explore the codebase directly — asking questions, uncovering APIs, understanding data relationships, and spotting risk before a single test is written. This isn't about becoming a developer. It's about using AI to finally see how the system really works — and using that insight to test smarter, earlier, and with far more confidence. If you've ever joined a new team, inherited a legacy app, or struggled to understand what really changed in a release, this episode is for you. Registration for Automation Guild 2026 Now: https://testguild.me/podag26

Manual regression testing isn't going away—yet most teams still struggle with deciding what actually needs to be retested in fast release cycles. See how AI can help your manual testing now: https://testguild.me/parasoftai In this episode, we explore how Parasoft's Test Impact Analysis helps QA teams run fewer tests while improving confidence, coverage, and release velocity. Wilhelm Haaker (Director of Solution Engineering) and Daniel Garay (Director of QA) join Joe to unpack how code-level insights and real coverage data eliminate guesswork during regression cycles. They walk through how Parasoft CTP identifies exactly which manual or automated tests are impacted by code changes—and how teams use this to reduce risk, shrink regression time, and avoid redundant testing. What You'll Learn: Why manual regression remains a huge bottleneck in modern DevOps How Test Impact Analysis reveals the exact tests affected by code changes How code coverage + impact analysis reduce risk without expanding the test suite Ways teams use saved time for deeper exploratory testing How QA, Dev, and Automation teams can align with real data instead of assumptions Whether you're a tester, automation engineer, QA lead, or DevOps architect, this episode gives you a clear path to faster, safer releases using data-driven regression strategies.

AI is accelerating software delivery, but it's also introducing new security risks that most developers and automation engineers never see coming. In this episode, we explore how AI-generated code can embed vulnerabilities by default, how "vibe coding" is reshaping developer workflows, and what teams must do to secure their pipelines before bad code reaches production. You'll learn how to prompt more securely, how guardrails can stop vulnerabilities at generation time, how to prioritize real risks instead of false positives, and how AI can be used to protect your applications just as effectively as attackers use it to exploit them. Whether you're using Cursor, Copilot, Playwright MCP, or any AI tool in your automation workflow, this conversation gives you a clear roadmap for staying ahead of AI-driven vulnerabilities — without slowing down delivery. Featuring Sarit Tager, VP of Product for Application Security at Palo Alto Networks, who reveals real-world insights on securing AI-generated code, understanding modern attack surfaces, and creating a future-proof DevSecOps strategy.