--- title: Where Is Your Product Losing Agents? description: "AI agents skip vendors one time in three and fail 28% of the time even when told which product to use. Here's where your product loses them, with data." date: 2026-10-21 url: "https://www.tryscope.com/blog/agent-experience-failures" --- # Where Is Your Product Losing Agents? Oct 21, 2026. Research. 8 min read. Your customers are handing their work to AI agents. One in five developers now writes no code by hand ([JetBrains, 2026](https://blog.jetbrains.com/research/2026/08/how-much-code-do-developers-really-let-agents-write/)). So an agent gets asked to solve a problem your product was built for. It might find you, sign up, read the docs, and integrate your API. Or it might write a few lines of code and skip you entirely. In our tests, that second outcome happened far more often than most software companies would expect. Across 41,249 runs where agents were free to choose how to complete a task, they built their own solution in 14,311 cases, nearly one in three. And even when we named the product and told the agent to use it, 4,875 of 17,385 runs (28%) still failed. That’s the [agent experience (AX)](/blog/what-is-agent-experience) problem. It breaks in three places, whether the agent finds you, whether it picks you, and whether it can finish once it does. ## Would the agent find you? *“Never heard of you.”* Suppose the agent decides it wants a vendor. That still doesn’t mean it will find yours. - In 953 of 1,206 runs where an agent took a position on a product, it never ran a web search. It went on what the model already knew, plus whatever was in the prompt and the repo. - When it did search, it was looking for a way to solve a task, not for a name. One product we tested surfaced in **89 of 1,777** of those searches, about **5%**, for the exact problem it solves. That’s not a dead end, it’s a different front. - **Search-based picks** are decided at the moment. They respond to what’s on the web right now. - **Memory-based picks** come from what the model already knows. That knowledge is frozen at training and only changes when a new model ships. You can move both, but not with the same work and not on the same timeline. And the balance between them keeps shifting as new models ship and agent tools change how often they search. This is why Scope tracks both. For every task, we measure how often agents pick from memory and how often they pick from search, and we watch how that split moves month to month. The useful test, then, is the one a customer would run: describe the task, leave out your name, see what happens. ## Would it pick you? *“I’ll just build it myself.”* Your competitor isn’t another vendor. It’s the agent’s own workaround. In our tests, when an agent could get a usable result without a vendor, it often did: - **GTM data tools:** 71 to 83% of runs skipped the product and scraped, stitched, or inferred the data themselves. - **Image generation:** 68 of 88 runs called a model directly instead of using a product built on top of one. - **Retrieval and search:** nine out of nine built their own pipeline. These were agents standing in for a sales ops lead, a marketer, a developer, and in each case they judged the task close enough to solve on their own. The reasoning the agents offered was practical, not hostile. Building locally looked simpler. It didn’t require an account or API key. From the agent’s perspective, using your product means signing up, finding credentials, configuring a workspace, and piecing together a quickstart before it can test whether you even do the job. The bar isn’t beating a workaround on features. It’s making the path to a working result shorter than the path to building one. In practice, that means: - **A first call without a signup form.** A test-mode key issued in one request, a rate-limited demo endpoint, or a device-code login a human approves with one click. The agent can try you before anyone fills in a form. - **A one-step quickstart.** One command or one request gets to a real result, not a configuration screen. - **A visible “why us” where the agent lands.** One line on why your output beats the thing it was about to write. ## Could it finish? *“I got stuck on your login page.”* Now imagine the agent picks you. In our tests, 28% of the time it still couldn’t finish. Here’s where it stalled. - **It couldn’t find the answer in your docs.** The biggest source of friction we found: 410 findings across 23 companies, mostly basic questions. Where’s the workspace ID? Which credential does this command want? What has to be set up before the first call? [Research on agent tool use](https://arxiv.org/pdf/2503.04479) finds the same thing we did: docs are written for a human who can fill in the blanks, then handed to an agent that can’t. Content hidden behind tabs or client-side rendering, examples missing imports and env vars, prerequisites nobody wrote down. - **It never read the page you wrote for it.** [Ahrefs](https://searchenginejournal.com/97-of-llms-txt-files-got-no-requests-ahrefs-data-shows/579478): 97% of llms.txt files across 137,000 domains got zero requests. Our runs: zero fetches across 358 runs at one company, and when an agent did open one, none of its 683 links were followed. Publishing a page isn’t the same as an agent reading it. - **It couldn’t get past login.** 26 of 29 runs blocked at auth at one product. 16 of 30 gave up and handed the task back to another. Same hard stop at 10 companies. Keep the security, and give agents an approved way in: API keys, service accounts, or OAuth device flow. - **It guessed your API.** Agents bypassed the SDK for raw HTTP in 30 of 30 runs, 12 of 13, 20 of 29, seven of nine. Skipping the SDK isn’t the failure. Reconstructing the integration without it is: nine of 13 runs guessed API paths and got “not found,” and some invented MCP hostnames. - **It couldn’t recover.** Eight runs hit the same 404 over and over. Five of 15 copied the error into a search engine looking for an explanation your product didn’t give. A 404 is the right answer to a wrong path; an empty one isn’t. Name what didn’t match and point to the right route. Each piece works on its own; the agent fails between them. Your logs show the symptoms, like a spike of 404s, but not what the agent was trying to do or why it stopped. ## Why one eval isn’t enough *“Ask me again next month.”* Say you run the test once and the numbers look fine. Agents find you, pick you, finish. Are you good to go? You’re good *today*. [One study](https://www.advancedwebranking.com/blog/ai-brand-mention-volatility-study) tracked 481 brands in AI answers for three weeks and watched half of them disappear. Three things keep moving after you look. And even if nothing moved, the same agent won’t give the same answer twice. ### The models move. - Every model has its own learned associations with your category, and each agent’s setup decides when it searches and how much the results count. [12 frontier or near-frontier models shipped in April 2026 alone.](https://presenc.ai/llm-releases/april-2026) - Even one model won’t repeat itself: [a 2026 SparkToro study](https://www.airops.com/glossary/answer-variance) found less than a 1-in-100 chance that ChatGPT gives the same brand list twice for the same prompt. ### Your rivals move. - Every pick that isn’t you went to someone. That’s ‘share’, and nobody in your category is counting it. - Memory share is who the model names from its training data. Search share is who it finds on the live web. Our working bet: search moves first, and memory follows once a new model is trained on that newer web. The rival ahead of you in search is the one to watch. ### You move. - Every new feature is a task some rival already wins. Every new segment is a set of prompts where the model’s default isn’t you. - Run the map before you ship and you know who holds the ground you’re walking onto, on which agent, through which door. Put the three together and it stops being an audit and becomes a standing picture of your market as agents see it: every model, every rival, every segment, updated as each one changes. If our bet holds, a competitor’s rise shows in their search column before it reaches their memory column, and who owns a space is visible before you decide to enter it. Most companies get that view from analysts, late and secondhand; this one comes from the agents themselves, and it’s current. ## See what the agent sees *“Here’s where I stopped.”* None of this is a mystery once you can watch it happen. Every failure from our own runs was found the same way: an agent tried to do a customer’s job, stopped, and someone was recording where. That’s what we do at Scope. We monitor agents through your product the way your customers’ agents already do, and we show you where they stop. Not a score, not a list of best practices, the actual step, on the actual page, with the number of runs that broke there. You make a change, we run it again, you see what moved. Somewhere right now, an agent is trying to use your product. You should get to watch. --- [More from the Scope blog](https://www.tryscope.com/blog). [Book a demo](https://www.tryscope.com/demo).