AI Models Built by Scraping Internet Content: The Fight to Get Paid 2026

Daniel Harrolds
AI Models Built by Scraping Internet Content: The Fight to Get Paid
This page may contain affiliate links.

AI models are built by scraping internet content on an industrial scale, and now content owners are demanding payment for the use of their material. This practice, known as scraping, involves bots that crawl millions of web pages, download the HTML, extract clean text, and store links to continue their digital sweep. It's a recurring complaint: AI models devour content without permission or compensation.

Each AI company has its own scraper: OpenAI uses GPTBot, Google uses Googlebot for Gemini, and Anthropic relies on ClaudeBot. These bots suck up information, and the resulting datasets train models that can generate text, images, and even videos. But the ease of scraping has led to a growing backlash from publishers, artists, and other content creators who see their work used without credit or payment.


Furniture with Meaning.

Find Original Home Decor sparking conversation at Grand Goldman Shop.


How AI Training Data Is Collected Through Scraping

Training a large AI model requires vast amounts of data. Scrapers systematically visit websites, extract text, and follow links to gather as much content as possible. This process is largely automated and indiscriminate, capturing everything from news articles to personal blogs.

ADVERTISEMENT
Goldman Media article image
Image: Goldman Media

Once trained, models may also call on web search tools when they lack sufficient information to answer a query. This often happens with current events or topics that have new details since the model's training. In such cases, another bot extracts content from websites to spice up the AI's response.

Goldman Media article image
Image: Goldman Media

The scale of scraping is enormous. For example, to train a model like GPT-3, OpenAI used hundreds of billions of words from web pages, books, and other sources. Much of this data was scraped without explicit permission, raising questions about the ethics and legality of such practices.

Goldman Media article image
Image: Goldman Media

Copyright and Intellectual Property Concerns

The most visible problem with scraping is related to intellectual property. This was evident when ChatGPT generated images in the style of Studio Ghibli or videos made with Seedance 2.0 featuring real actors in artificially generated scenes. Such outputs blur the lines between inspiration and infringement.

Goldman Media article image
Image: Goldman Media

Over the past three years, copyright infringement lawsuits have proliferated. The New York Times sued Microsoft and OpenAI, accusing them of using millions of its articles to train AI models without authorization. Similar cases have been filed by authors, visual artists, and music labels.

ADVERTISEMENT

These legal battles highlight a fundamental tension: AI companies argue that training on publicly available data is fair use, while content creators contend that it's theft. The outcomes of these cases could reshape the entire AI industry.

Who Wants to Charge for AI Scraping?

Content owners are increasingly vocal about wanting compensation. Publishers, in particular, see their valuable journalism being used to train models that then compete with them for readers. They argue that AI companies should pay licensing fees for the content they scrape.

Some have already taken action. News Corp, the parent company of The Wall Street Journal, signed a deal with OpenAI reportedly worth over $250 million over five years. Other publishers, like Axel Springer and The Associated Press, have also struck licensing agreements.

But many smaller creators lack the bargaining power to negotiate such deals. They are calling for collective licensing schemes or legislative action to ensure they get a fair share.

ADVERTISEMENT

The Rise of Content Licensing and Opt-Out Mechanisms

In response to the backlash, AI companies have introduced ways for websites to opt out of scraping. OpenAI allows sites to block GPTBot via robots.txt, and Google offers similar controls for Googlebot. However, these measures are voluntary and not always respected.

Some companies are going further by creating marketplaces for licensed content. For instance, Adobe launched a compensation program for artists whose work is used in its AI tools. Startups like Scale AI are building platforms where content owners can set prices for their data.

These efforts aim to create a more sustainable ecosystem where AI development doesn't come at the expense of content creators. The longer history behind the story is traced in AI Models Built by Scraping Internet Content: The Fight to Get Paid.

What's Next for AI and Content Compensation?

The debate over AI scraping is far from over. Lawsuits will likely continue, and regulators may step in. The EU's AI Act includes provisions requiring AI companies to disclose copyrighted material used in training, which could set a global precedent. The background to this story is set out in AI Copyright Reforms: Australians' Online Content Scraping. A related thread runs through Sony and Warner Sue Anthropic Over AI Music Copyright Infringement. Earlier coverage in Book Publishers Sue Google Over Gemini AI Copyright Infringement shows how the argument developed.

ADVERTISEMENT

In the US, the Copyright Office is examining the issue and may issue guidance. Meanwhile, content owners are exploring technical solutions like watermarking and blockchain to track usage. A closer look at the same subject appears in Quantum Network Passes Its Most Demanding Test Yet for Real-World Deployment. Anyone following the wider picture can start with Wolfoo Copyright Infringement: Peppa Pig Wins Legal Battle. The same theme surfaces in AI Can't Crawl Under Houses: Why Skilled Trades Are Booming, which takes a different angle.

Ultimately, the goal is to find a balance that allows AI to innovate while ensuring creators are fairly compensated. As AI models become more advanced, the demand for high-quality training data will only grow, making this a critical issue for years to come. Context on how this played out before sits in AI Can't Crawl Under Houses: Why Trade Jobs Are Booming in the US and France. A companion piece, School Phone Bans: Why Teens Are Demanding More Offline Support, approaches the same subject from another angle.

FAQ

What is AI scraping?

AI scraping is the automated process of extracting content from websites to create training datasets for AI models. Bots crawl millions of pages, download HTML, and extract text, often without permission.

Why are content creators upset about AI scraping?

Content creators argue that AI companies profit from their work without compensation or credit, undermining their livelihoods and intellectual property rights.

ADVERTISEMENT

Can websites opt out of AI scraping?

Yes, websites can use robots.txt to block specific bots like GPTBot. However, compliance is voluntary, and some AI companies may not honor these requests.

What legal actions are being taken against AI companies?

Several lawsuits have been filed, including by The New York Times against Microsoft and OpenAI, alleging copyright infringement. These cases are ongoing and could set important precedents.

ADVERTISEMENT
Daniel Harrolds

Author

Daniel Harrolds

With a career spanning four decades, Daniel is almost a library in the field of precious metals investing and Gold IRAs. His insightful strategies and pragmatic results-oriented approach make him a resource in safeguarding wealth, and financial foresight.


SPONSORED

Get Lifetime Access to the Lastest Movies, with Exclusive Offers & Free Express Order Delivery.

ADVERTISEMENT