A walkthrough of how an IcebergQA Rescue Sprint typically plays out — every question, every reply, every AI scan, every confirmed bug, every milestone. The starting bug map is grounded in a real scan of a public website so the data is honest; everything else is a constructed scenario, not a real engagement.
Names and dialog are invented for the example. Any resemblance to real Starbucks employees is coincidental.
Jason and Phil are real principals. Samantha is a representative XBOSOFT lead engineer for the example — she drives the AI tools and the engagement day-to-day. Voices in this thread are illustrative; real client threads stay private.
Every phase is AI-driven and AI-executed. Humans (Samantha + the XBOSOFT senior team) advise, validate, and stay in the loop — but they don't hand-write the tests or hand-execute the runs. The client is in control of every decision; we do all the work.
Two hard milestones gate everything: Day 8 — the Quality Baseline & Bug Map. Day 30 — the sprint demo + 90-day plan.
Hi team — exec just lit a fire under us about Holiday readiness. Last year our consent banner silently broke on launch day and we ate three news cycles.
Want this sprint to find what's actually hiding in production. Not another dashboard.
Welcome Maya. The next 30 days are tightly scoped: Days 1–3 access & scoping, Day 8 you'll have a real Quality Baseline & Bug Map, Day 30 demo + 90-day plan. Hard milestones.
To get there fast, we need three things in the next 48 hours:
1. A short call to walk your top user flows — what would hurt the business if it broke today?
2. Read-only access to your existing test repo (Selenium, Cypress, manual cases — whatever you have).
3. Permission to point our scanner at production for the homepage and the unauthenticated funnels. No PII, no writes.
Top flows off the top of my head — homepage, store locator, order ahead, sign-in & Rewards, gift cards, the consent banner. Will get Jordan to walk you through the test repos this afternoon.
Question: across our web and mobile teams we have about 420 Selenium / Cypress E2E tests and ~1,200 manual cases. They're ~60% flaky in CI. Are you going to throw them out?
No. We upgrade them. Day 1–8 we baseline what's broken; Days 9–16 we convert your suite to self-healing AI checks. Same coverage, much less brittle.
The flake usually isn't your tests' fault — it's locators chasing CSS class hashes. AI checks describe intent, not selectors.
OK that's the right answer. Connecting Jordan now.
Heads up — repo access is in. web-e2e and mobile-e2e on GitHub. 420 specs total, plus the ~1,200 manual cases from our QMS export dropped in the shared drive.
What can I expect to see in the next few days?
Got it. Samantha from XBOSOFT is taking point on discovery. By Friday end-of-day she'll have the AI run the first baseline scan against production. You'll see status as it runs.
By Day 8 (Monday) you'll get the full Quality Baseline & Bug Map — scored, prioritized, with repro evidence for every finding.
Two more things to make your life easier:
1. We've got a staging environment that mirrors prod ~95%. I can give you an X-Stage-Token request header your tools can send to bypass our edge WAF and hit staging.starbucks.com directly. Safe to break things there.
2. We provisioned 4 test accounts with deliberately different data states so your personas don't all see the same thing:
qa-clean — brand-new, empty cart, no historyqa-cart — items in cart, mid-customization, not checked outqa-rewards-green — Green tier, 30 days old, 1 saved storeqa-rewards-gold — Gold tier, 3 years history, payment on file, 5 favorite storesHi Jordan — Samantha here. That's everything we need. Staging + token + 4 data-state accounts means the personas hit realistic conditions, not synthetic ones. Privacy probe especially loves having a fresh account state to compare against the long-time Gold one.
On the test cases — we don't just import them blind. I'm reading through every one of your 420 specs and 1,200 manual cases this week, and the AI is helping me cluster them by intent, find assertion gaps, and flag the redundant duplicates. I'll come back with a gap report Friday so we can huddle on the ambiguous ones before we convert anything.
X-Stage-Token accepted on staging.starbucks.com · 4 test accounts validated Day 3 · 2:11 PMSamantha and the AI read every spec and every manual case shared with us. The AI clusters them by intent (not by selector), flags where the “what success looks like” assertion is missing or vague, and identifies duplicates so we don't carry dead weight into the new suite.
| Area | Why it matters |
|---|---|
| Consent integrity | Banner renders, accept/decline propagates, third-party scripts honor the choice |
| Identity health | mParticle / identity-provider requests actually succeed (not 0-byte CORS) |
| Analytics integrity | Optimizely + measurement events log on every key step, not silently dropped |
| Keyboard a11y | Tab order, focus traps, skip links — the audit script itself runs without throwing |
| Contrast | WCAG AA enforced automatically on every interactive element |
| Initial-load budget | Request count + weight gated per release; regressions break the build |
| Social / share metadata | OG tags + image render correctly when a page is shared |
| Search relevance | Autocomplete behavior, no-results UX, item-not-available paths |
| Error-state UX | 404, 500, empty cart, no-geo on locator, sign-in failure messaging |
The 80 cases with missing assertions need a 30-minute huddle with Jordan's team — we want product context, not selectors. Booked Wednesday.
One more piece of the Day-3 readout: now that the AI has parsed everything, we can quantify AI-readiness and ROI per cluster of tests. You then have four paths per cluster. You choose. We do the work.
For every cluster of tests the AI assessed, here are the four paths. Every path is AI-first — the difference is how much of your existing investment we preserve, augment, or net-add to.
Our tools convert most manual and automated cases into AI-first versions where the AI executes and validates the test. Same scenarios you cover today, no brittle selectors, no flake.
Drop our SDK into your existing automated test runs to layer AI assertions, smart waits, and self-healing on top of what you already wrote. Keep the investment, get the AI uplift.
Our browser and mobile extensions piggy-back on your existing test runs to capture massive extra coverage — accessibility, visual diffs, console errors, network signals — without writing one new test.
AI analyzes the gaps and auto-generates new coverage in whichever format you want: more manual cases, more automated scripts, AI-first journeys, or just additional permutations of what's already there.
| Cluster | Recommended path |
|---|---|
| 340 ready-to-upgrade E2E cases | Path 1 · convert to AI-first execution |
| 80 missing-assertion cases | Path 2 · SDK assertions + huddle for product context |
| ~1,200 manual cases | Path 1 + 3 · convert high-value, augment the rest via the extension |
| 9 coverage gaps | Path 4 · AI auto-generates 820 net-new AI-first journeys |
| 120 redundant duplicates | Collapse · no work needed |
| 60 stale / broken | Retire with sign-off · or rewrite via Path 4 if still valuable |
You sign off on the mix per cluster. Default we recommend above gets you the highest ROI for this sprint — but every cluster is your call.
I love that this is laid out as a choice. Approving Paths 1, 3, and 4 for this sprint. Hold Path 2 (SDK) for the 90-day plan — we want to see the Day-30 results before we touch the existing CI.
Locked in. Path 1 + 3 + 4 this sprint. Kicking off the AI conversion + extension instrumentation now. Path 2 (SDK) on the shelf for the extension.
Crawled, deduped, and graphed. Home at the hub, the eight critical flows on the first ring (gold), and the supporting sub-pages branching outward. Every node becomes a target for AI personas, accessibility checks, and the integrity probes.
22 nodes, 21 edges, scoped from a 39-request homepage crawl. Native apps and authenticated deep links are out of scope for this sprint — flagged for the 90-day extension.
starbucks.com Day 4 · 8:02 AMEach persona has its own goals, browsing habits, and patience threshold. We score what they experience, not just whether the page loaded.
All three of these are silent in your existing test runs — the page loads, your CI is green. Only the integrity probes catch them.
net::ERR_FAILED (CORS) Day 6 · 11:18 AM/identity returned 400 (Bad Header) Day 6 · 11:24 AMHeads-up Maya — early signal has a few hot spots. The fundamentals look solid (this is a mature codebase), but three of the third-party integrations you depend on for trust and measurement are silently failing on the live homepage. Detail is still being collected; full picture lands on Day 8 with severity, repro, and the fix prompts.
Not asking for action yet — just flagging so it's not a surprise on Monday.
That's a sentence I don't love hearing on a Saturday. Appreciate the heads-up. Will brief Rachel before the readout Monday.
The honest picture of what's broken on the Starbucks homepage today — scored, prioritized, and with evidence per finding.
Bug map is ready. Headline: Quality 72/100, Grade C+. Solid foundation — visual storefront is best-in-class — but the trust and measurement layers underneath have real cracks. Posting the scorecard now.
| P | Finding | Cat | Sev |
|---|---|---|---|
| P10 | CORS policy blocking Privacy Consent Manager (TrustArc) | Privacy | High |
| P9 | Identity request failures (mParticle) — header misconfiguration | Reliability | High |
| P8 | JavaScript exception in keyboard accessibility audit | A11y | Med |
| P7 | Optimizely event logging blocked by CORS | Reliability | High |
| P6 | Low contrast on outline buttons | A11y | Med |
| P5 | Excessive network requests on initial load | Perf | Med |
| P4 | Inconsistent button styling in hero sections | Visual | Low |
| P3 | Non-standard meta tags for social sharing | Content | Med |
| P3 | Generic Open Graph image | Content | Med |
| P2 | Vertical alignment of 'Find a store' icon | Visual | Low |
I'm joining this thread. TrustArc and mParticle being broken in production is not something I want to learn from Twitter. Walk me through #1.
Welcome Rachel. Posting the full repro on P10 now. Short version: the consent manager is being blocked by the browser before it can even render the banner — so a subset of users may be loading the site with no consent gate at all, which is the GDPR/CPRA exposure you're worried about.
https://consent.trustarc.com/notice?... on every pageAccess to script blocked by CORS policy: No 'Access-Control-Allow-Origin' header on the requested resource.Access-Control-Allow-Origin: https://www.starbucks.com on the script delivery endpoint, or switch to first-party script proxy via the existing CDN.Filing internal ticket today. Phil, Jason — keep going. The fact that 60% of our test suite is green while this is happening is exactly why we hired you.
Starting the upgrade of your 420 E2E tests today. Each one becomes a self-healing AI journey — same scenarios you already cover, but the locators stop chasing CSS-hash churn.
And we're not stopping at parity. Samantha is having the AI autonomously generate net-new journeys across 9 coverage areas the existing suite is blind to — consent integrity, identity health, analytics integrity, keyboard a11y, contrast, initial-load budget, social/share metadata, search, and error-state UX.
Honest question — what does an AI check actually look like when you commit it? My team is suspicious of "AI magic."
Fair. Here's the diff for one of the brittle ones — the "tap Order Now in the hero" check. Before, after.
The intent-based check survives the next CSS class-hash regen — the exact change that broke the original ~60% of the time.
The AI check survives the next CSS class-hash regen — which is the change that broke the original 60% of the time.
OK that's not magic, that's just sane. Carry on.
1,200 journeys total — 380 from your originals (40 redundant pairs collapsed into single AI checks), plus 820 net-new that Samantha had the AI generate autonomously to cover the 9 areas the existing suite couldn't reach.
No human had to click anything. The AI ran 1,200 journeys across 4 personas while you slept, surfaced 412 signals, and queued them for the human filter.
Quick note on how the nightly runs feed back to you — we don't dump raw AI output into your tracker. Every finding gets reviewed by a senior XBOSOFT QA engineer before it crosses your desk. That filter is what keeps the signal real.
Good. The last AI tool we tried filed 2,000 issues in week one, 90% nonsense. My team stopped reading them. What's the ratio you're seeing so far?
Last night Samantha had the AI run the full 1,200-journey suite end-to-end on its own. 412 raw → 187 confirmed real after our XBOSOFT senior team filtered. The 225 we dropped were duplicates of known findings or false positives (icon mis-classified as a button, that kind of thing). We'll keep posting the ratio so you can watch it stay healthy.
Posting Week-3 validated findings. The Day-8 bug map (10 items) was all reproduced and confirmed.
The bigger story is what happened after Samantha had the AI autonomously generate 820 net-new journeys for the coverage areas your existing suite never touched and execute all 1,200 nightly without supervision. Hundreds of signals surfaced. XBOSOFT's senior validators worked them down to 177 confirmed net-new real bugs a homepage scan would never have caught — they live inside flows.
Samantha had the AI autonomously generate 820 net-new journeys for coverage gaps the original 420-test suite never touched, then execute all 1,200 nightly on its own. That's where the hundreds of new findings came from. XBOSOFT seniors validated each one before it crossed into Jira.
| Finding | Flow | Sev |
|---|---|---|
| Consent choice not applied before analytics fires (race condition) | Consent → home | High |
| Rewards sign-in drops session token on slow 3G · silent re-login loop | Sign in / Rewards | High |
| Store locator returns 0 results when geo is denied (no fallback) | Find a store | High |
| Customization options lost when switching size mid-order | Order ahead | Med |
| Gift-card balance not announced to screen readers | Gift cards | Med |
The Rewards sign-in loop is interesting. We've had support tickets about that for months and never reproduced it. You're saying it's slow-3G specific?
Yes. The token write loses its race with the next request on connections under ~400Kbps down. We attached the repro video and the network trace to the ticket. Goes straight to Jira when you greenlight it.
Greenlit. Sending to platform team. Thank you.
Benchmark done. We pointed the same scan rig at your category peers. Result is going to be useful for the Rachel conversation.
Right at category average — visual brand is best-in-class, but the broken trust layer and the 177 flow bugs are what's keeping you out of the leader band. All addressable.
That slide is going in Monday's exec readout. What's the operating model you'd recommend after the sprint?
Hybrid. IcebergQA runs the 1,200 AI journeys nightly + a senior-validated deep pass monthly. Your team owns triage and fixes. Release gate: consent, identity, analytics, top-tier a11y must pass before ship. Cadence: weekly trend, monthly deep run, quarterly benchmark refresh.
What changed in 30 days. What it's worth. Exactly what to do over the next quarter.
Demo deck attached. Headline number: closing the three Highs from the Day-8 map plus the top flow bugs Samantha's nightly runs surfaced moves you from 72 → a projected 88 (C+ to A−), restores the trust layer, and turns the nightly AI gate on so this doesn't slide back.
| By day | What ships |
|---|---|
| 30 | Top 3 Highs shipped · nightly AI gate on · consent + analytics signal restored |
| 60 | 177 validated flow bugs worked down by ~half · journeys extended to native apps · WCAG AA across primary flows |
| 90 | Re-benchmark vs category · hybrid model fully handed off · quality trend owned in-house |
This is exactly what I needed. Three things I want to say in front of the team:
1. A C+ overall didn't make me feel safe — the 3 reds inside it did. Day 8 changed how I think about “green dashboards.”
2. The TrustArc finding alone paid for the sprint.
3. 177 net-new flow bugs that our 420-test suite never found is not a small number. Approving the 90-day extension. Next target is order-ahead — that's where the revenue risk lives.
Thank you both. This is the most honest QA engagement my team has ever been part of. Let's do order-ahead.
Honored to keep going. Kicking off the 90-day plan tomorrow. Samantha stays your day-to-day point.
Same 30-day sprint — same hard milestones, same honest findings, same human-validated AI — pointed at your app.