Sustaining websites and user engagement in the AI era

Section 01 — How bad the damage actually is

The numbers are inconsistent, and that’s somewhat intentional. Anyone writing an „SEO is dead“ piece cites the worst outliers. Anyone trying to keep AI companies calm cites the aggregate average. Both sides are technically correct.

What’s true: organic traffic declined by an average of 2.5% across 40,000 of the largest US websites in 2025 — a moderate drop that undercuts the most dramatic claims. Also true: Forbes lost 50% of its organic traffic. HubSpot lost between 70 and 80%. Business Insider 55%. CNN 38%. The average is pulled upward by the biggest players — mid-sized publishers ranked roughly between 100 and 10,000 carry the steepest part of the decline.

„More rankings, less traffic — impressions are up, clicks are down.“

That’s the pattern multiple SEO agencies are documenting simultaneously: pages rank better than ever, appear more frequently in SERPs — and still generate fewer clicks. The reason is Google’s AI Overview, which has appeared in around 30% of all searches since its broad rollout in March 2025. The click-through rate on organic results below an AI Overview sits at 0.61% — compared to 1.62% without one. Sixty percent of all Google searches in 2025 end without a single click to an external website.

Section 02 — The real problem: bots arrive, people leave

While human traffic is declining, something else is exploding: bot traffic. According to TollBit, by Q4 2025, one AI bot visit occurred for every 31 human website visits. In Q1 2025, that ratio was still 1:200. Bots are growing faster than humans — in both directions at once.

The uncomfortable part is the asymmetry between taking and giving back. Cloudflare has documented what it calls the crawl-to-referral ratio — how many pages a bot crawls to send back a single human visitor. Googlebot’s number: roughly 5:1. Anthropic’s ClaudeBot peaked at 286,000:1 in January 2025. Even after a significant drop, it sat at 38,000:1 in July 2025 — still the worst ratio among major AI players. According to the CEO of Conductor, ChatGPT crawls around 60,000 pages and sends back exactly one visitor.

What that means in practice: the server carries the load, someone pays for the bandwidth, and the content that teams were paid to create gets hoovered up at industrial scale — with nothing coming back. Small blogs and large publishers alike sit in the same structural trap. Digital Trends documented 4.1 million bot scrapes in a single week that generated 4,200 human referrals. The numbers speak for themselves.

„AI needs the web to exist. The web needs people to survive. The machines are showing up — but they’re not paying the bill.“

There’s a side effect that rarely gets discussed: AI bots disguised as headless browsers consume ad impressions without ever being a real human. Advertisers pay for traffic that is partly bot-generated, slipping past anti-bot systems. Publishers appear to be earning ad revenue — but the numbers are increasingly unreliable.

Section 03 — Using AI to bring users back

It would be too easy to frame AI as purely the problem. It’s also the tool website operators can use to push back — but not with the naive approach of producing AI-generated content and hoping it ranks. The market is already flooded with generic AI output. What AI can actually do is enable precise personalization that gives people a concrete reason to return to a platform.

Personalized email sequences that adapt dynamically based on user behavior continue to show some of the best ROI of any digital channel — researchers at Kellogg put it at 36:1. That holds because email subscribers made an active choice: they want content from this specific source. AI helps deliver the right piece to the right person at the right time, without requiring them to remember to come back.

On-platform recommendation engines — not as an advertising mechanism, but as genuine content discovery — can significantly improve dwell time and return visits. Netflix and Spotify have trained users to expect personalized suggestions. A niche community that meets that expectation gives its members a reason to come back instead of just asking an AI chatbot for a summary.

For user-generated content specifically: people create content when they get something back — status, direct compensation, visibility, or the satisfaction of being genuinely useful. Stack Overflow ran this system on reputation scores for years. The decline shows what happens when the incentive erodes because answers are available faster and more conveniently elsewhere. Using AI to lower the barrier to contribution — structured templates, assisted writing flows, automatic categorization — reduces friction and gives the community a tangible quality advantage over a generic chatbot response.

Section 04 — Technical countermeasures: what works and what doesn’t

Robots.txt was designed for 2005. It’s a polite request — not enforcement. One study documented an average of 156 violations per website over a three-week period in 2025. Anyone who thinks a robots.txt directive will stop AI training is underestimating the situation.

Llms.txt serves a different purpose and is frequently misunderstood. It’s not a blocking mechanism — it’s a structured hint for LLMs during inference-time retrieval, roughly equivalent to a curated sitemap written for machines rather than crawlers. No major AI provider has implemented it as a blocking standard. An analysis of 500 websites found zero correlation between having the file and receiving fewer AI citations. Useful as a navigation guide for well-behaved crawlers — useless as an access barrier.

What actually works: Cloudflare flipped the default on July 1, 2025 — internally referred to as „Content Independence Day“ — switching AI crawler blocking to opt-in for all new users. That shift matters because Cloudflare handles roughly 20% of global web traffic. The Pay-Per-Crawl model, generally available since August 2025, is conceptually the most interesting development: instead of block or allow, there’s a third option — an HTTP 402 response that tells crawlers access has a price. Those unwilling or unable to pay don’t get in. Those who pay enter a commercial relationship.

In practice, the sensible strategy runs on two tracks: block or charge training crawlers (GPTBot, CCBot, Meta-ExternalAgent), while explicitly allowing search and retrieval crawlers (OAI-SearchBot, PerplexityBot) — they’re what ensures a site gets cited in AI responses. Citations in AI Overviews and LLM answers are the new form of visibility, even when direct traffic from them is minimal. A WAF rule takes precedence over any robots.txt directive — it’s evaluated before robots.txt is ever read, so a WAF block always wins.

The granularity available now is worth understanding. GPTBot trains the model — block it. ChatGPT-User fetches a page because a user explicitly asked — allow it. Google-Extended controls Gemini training, not search ranking — opting out is possible without losing search visibility at all.

„Blocking costs citability. No blocking costs content value. Pay-per-crawl is the middle ground — the web is slowly learning to price itself.“

Section 05 — Revenue models beyond display advertising

Anyone whose business model is built on CPM-based display advertising is caught in a double bind: fewer human visitors, more bot traffic consuming ad impressions without converting. The problem isn’t new — it’s old, just now acute. The answer is also not new, but many website operators have been putting it off: diversification.

Subscriptions and memberships. The most direct way to reduce dependence on traffic volume. A small paying community is more stable than a large passive audience that arrived through Google. Substack and Beehiiv have validated this for newsletter creators; the model works equally well for niche communities, specialist platforms, and review sites. The key requirement: the content has to offer something AI can’t easily replicate — timeliness, local expertise, verified primary sources, genuine community interaction.

Content licensing to AI providers. Several large publishers are pursuing this directly: a commercial deal instead of blocking. Reddit sold a data license to OpenAI. Axel Springer reached an agreement with OpenAI. That requires negotiating leverage — which comes from unique, high-quality content, not generic SEO articles. For niche platforms with specialized datasets (employer ratings, trade reviews, industry data), this is an underestimated option. Structured datasets that AI companies can use for RLHF or RAG systems have real market value.

Paid-access APIs. Anyone sitting on data that other systems — including AI systems — can use meaningfully can monetize access via an API. That decouples the business model entirely from the page-view mindset. TollBit already runs a marketplace where AI companies pay publishers directly for content; almost 20% of its 7,000 connected publishers have generated measurable revenue through it.

Direct product monetization. Platforms that have relied entirely on advertising are discovering that their real asset is community trust — not traffic. Premium listings, verified profiles, sponsored content from players who are part of the ecosystem: these revenue streams don’t depend on Google CTR. They depend on whether the platform is genuinely useful to the people and organizations it connects.

Events, courses, consulting. The content on the website becomes proof of expertise — the actual product is access to the people who embody that knowledge. This works particularly well for B2B content sites and specialist communities where readers aren’t looking for generic information but for decision support.

Section 06 — Something has to give

The internet was built on an exchange. People created content — reviews, forum answers, articles, tutorials, documentation. Other people found it, read it, linked to it, built on top of it. Search engines indexed it and sent traffic back. It wasn’t a perfect system, but it was a functioning loop. Everyone contributed something, and something came back.

That loop is breaking. If users stop returning anything to the internet — because AI gives them everything they need without requiring them to click, read, engage, or create — the web starts to hollow out. Not overnight, but gradually. Less content gets created because the incentive disappears. What does get created gets absorbed immediately. The pool shrinks.

„AI learns from what people put on the web. If people stop putting things on the web, AI starts eating its own tail.“

There will be a correction at some point — legal, economic, or at the infrastructure layer. Pay-per-crawl, licensing deals, litigation, regulation: the mechanisms exist or are being built. Whether any of it moves fast enough for publishers losing traffic right now is a different question, and an honest answer would be: probably not for everyone.

What might be more interesting to watch is what happens at the user level. At some point, people will develop a clearer sense of the difference between AI-summarized content and something a human actually wrote, tested, experienced, or got wrong. For a lot of queries — convert centimeters to inches, summarize a tax law, explain how a for loop works — AI is genuinely the right tool and nobody needs a blog post. But for other things, the origin will start to matter. Real reviews from real customers. An opinion from someone who has skin in the game. A repair guide from someone who actually broke the thing first. A forum thread where people argued and eventually reached something true.

People still buy vinyl. Not because it sounds objectively better, and not out of nostalgia alone, but because the format carries something that a Spotify stream doesn’t. The friction is part of the point. It wouldn’t be surprising if something similar happens with parts of the web — a growing preference for content with a visible origin, a traceable author, a comment section that isn’t entirely bots. Not for everything. But for the things where it matters.

This article will almost certainly be crawled, summarized, and served to someone who never clicked on it. That’s fine. The argument still stands either way.