Why AI Assistants Like ChatGPT Are Building Their Own Search Index
ChatGPT, Perplexity, and Google are all moving the same direction: away from licensed search data and toward owning their own index. Here's why that keeps happening, and what it changes for GEO.
Most GEO advice still describes ChatGPT as a client. It asks Google or Bing a question, gets an answer back, and hands it to you with better formatting. That description used to be accurate. It’s already out of date.
Here’s the pattern I keep noticing: every assistant that gets big enough stops being a wrapper around someone else’s search engine and starts building its own index instead.
Perplexity did this years ago. It started as a GPT wrapper sitting in front of Bing and didn’t stay one. Once it had the traffic and the money to justify it, it built its own crawler and its own hybrid index. Google never had to make the move, because Google was the origin the whole time. Its AI Overviews and AI Mode sit on the same index it spent two decades building.
ChatGPT is the newest one to cross over.
The clearest example so far
Peec AI’s investigation, led by Tomek Rudzki, found that ChatGPT runs its own retrieval index internally called Labrador. Not one index. A family of them: general web, PDFs, YouTube, news split by freshness, arXiv, Wikipedia, local, finance, legal, medical, shopping, images. Each stores what any real index stores: full page content, crawl date, publish date.
Outside Labrador, ChatGPT still calls Google, Microsoft’s Web IQ, Yelp, and TripAdvisor. Owning most of your answers doesn’t mean owning all of them.
They backed this with more than a hunch. OpenAI’s job postings describe teams building indexing and retrieval infrastructure at exabyte scale. Sworn testimony from Google’s antitrust trial has ChatGPT’s head, Nick Turley, giving the reason outright: quality problems with the data they were licensing, a direct offer to Google that got turned down, and a decision made in 2023 to build the alternative themselves. And Peec caught OpenAI running live A/B tests, weighing its own index against scraped results in shopping specifically, which is exactly where you’d test something you aren’t fully confident in yet.
Their piece is worth reading in full. What I want to write about is the bigger pattern it belongs to.
Why this keeps happening
The economics only point one way. Licensed search data is slow and expensive at scale, and it puts you at the mercy of a provider who, in OpenAI’s case, is also your biggest competitor for the same users.
That’s not a partnership. That’s a liability with an invoice attached.
Once an assistant has enough traffic to justify the infrastructure cost, dropping the outside dependency stops being optional. You get to own the ranking logic, the caching layer, and the freshness rules end to end, instead of inheriting someone else’s idea of what a good result looks like.
I don’t think ChatGPT is the last one to make this move. I think it’s the most visible one so far because Peec happened to catch it mid-transition, thanks to a field in ChatGPT’s own server data that gave the whole thing away for two months before it presumably got cleaned up. Not every assistant is there yet. Several still route web results entirely through licensed APIs. But the direction of travel looks the same everywhere.
What this breaks in how GEO has been done
The old model was simple. Rank in the search engines, structure your content well, and the AI answers mostly took care of themselves. That made sense when most assistants really were passing queries through someone else’s search engine.
It assumed one funnel. It doesn’t survive five separate indexes.
Being visible to Google says nothing about whether GPTBot has actually crawled you. It says nothing about whether Perplexity has a current copy of your page. It says nothing about whether you’re in the general web index or the specific vertical one, shopping, local, finance, that a given query actually pulls from. Labrador having a separate index for each of those is the detail I keep coming back to. Generic visibility and being in the right index are not the same thing.
Caching adds another layer. If an assistant is serving answers from a stored copy of your page instead of fetching it live, then how often that copy gets refreshed matters as much as whether the page ranks at all. That’s a new kind of technical question, and most tools I’ve used aren’t built to answer it yet.
Where I’d put the effort now
I’m not throwing out the fundamentals. Structured, crawlable content is still rewarded by every one of these systems, because they all reward some version of what Google always did. That part hasn’t changed and I don’t expect it to.
What’s changed is that there’s no longer one funnel to optimize for. I’m treating each major assistant as its own crawl target, checking server logs for the individual bots rather than assuming Google covers everything, and paying attention to which vertical a page is likely to land in, not just whether it’s indexed at all. Where something like lockdown mode exists, I’m using it to check whether a page has actually been cached.
There’s no single funnel left
This is more work than the old model, not less. That’s what happens when an industry stops being three players and becomes ten.
However many indexes exist now, each one decided that borrowing had gotten too expensive. Show up in all of them, or accept that your visibility is someone else’s rounding error.