Will AI Assistants Bring About the End of Technical Content Sites? 😱

The change in behavior caused by AI assistants will not be forgiving to technical content sites like...KIRUPA.com!

For many years, developer-centric content sites like KIRUPA have attracted traffic through a popular, well-trodden path. This path starts with Google and ends with a website containing the relevant information:

In this world, the developer is happy. The content creator is happy because the developer (thanks to Google) visited their site. Everybody wins!

This Path is Reaching the End of the Line

Increasingly, developers are no longer searching the web blindly for a snippet of code that may be found buried somewhere deep inside a website.

Instead, they are relying on AI assistants inside their code editor to help them out:

When we take a broader look, this makes a lot of sense. Why context switch away from your coding environment to get an answer to a coding problem? AI assistants can answer your questions inline and keep you in the flow state. That’s a great thing.

It’s not a great thing if you are a content creator who relies on coding-related search traffic to bring visitors to your site. StackOverflow is a great example, given how relevant their results are for most coding questions. Their site traffic has been steadily decreasing:

I have seen similar trends in analyzing the KIRUPA traffic. From talking to other content creators, they are also seeing similar traffic declines.

Chicken and Egg Problem

This brings up a more fundamental problem. How does an AI assistant know how to respond to a question? It checks against its massive database of stored knowledge. Who created the stored knowledge for, let’s say, coding questions? It’s the very content creators whose content is now being summarized and presented through the UI of ChatGPT or Gemini:

If we drag the timeline out far enough and the current trends remain, content creators will see very little traffic going to the content they create. This will reduce incentives for them to create content. Not every content creator is at the scale of Reddit where AI companies will pay for permission to scan and crawl the content. There is a long-tail of smaller content creators whose content will simply be absorbed.

Nobody really knows how this will play out in the long run. If the industry can’t find a good solution here, what we’ll find is that AI assistants are great at answering all coding content created up until 2024. After this point, enough content creators might have stopped creating content at a scale where there isn’t enough new knowledge to index on the latest changes.

Bad Idea: Blocking AI Crawlers

One potential solution is for content creators to block AI crawlers from indexing their content by using robots.txt and adding the following values:

# Used for many other (non-commercial) purposes as well
User-agent: CCBot
Disallow: /

# For new training only
User-agent: GPTBot
Disallow: /

# Not for training, only for user requests 
User-agent: ChatGPT-User
Disallow: /

# Marker for disabling Bard and Vertex AI
User-agent: Google-Extended
Disallow: /

# Speech synthesis only?
User-agent: FacebookBot
Disallow: /

# Multi-purpose, commercial uses; including LLMs
User-agent: Omgilibot
Disallow: /

Personally, this is a bad idea. If AI assistants become the primary way people get their questions answered, it’s better to figure out how you can play in this area as opposed to quitting and going home. As we’ve seen with paywalled content, high-quality information will be difficult to access, and content of a questionable quality will prevail.

Till Next Time

AI assistants are here to stay, and I personally find them very convenient for answering coding questions. That’s the content consumer in me speaking. As a content creator as well, the way I’ve reacted to this is by creating less technical coding content. Instead, I’m now creating more high-level conceptual content such as my latest experiments (and book!) around Data Structures and Algorithms.

I’m also focusing on more Product Development content and videos, both of which are safe (for now!) from being made obsolete by AI. So, I have a question for you:

If you are a content creator, how are you handling this shift where AI presents your content from its own UI?

Feel free to reply in this forum topic dedicated to this question.

Cheers,
Kirupa 😇