The carbon cost of scraping the web with AI

The short version

• The more text a model reads, the more energy it uses. A web page is a long input: the practice pages in our benchmark were 7,000 to 10,000 tokens each, and real product pages are often much longer.
• Per page, published estimates put a request of about 10,000 tokens at roughly 0.8 to 6 Wh, depending on the model. Extraction code on the same page uses roughly 0.00003 to 0.00006 Wh.
• Per million pages, that's about 830 to 5,700 kWh for an AI model reading each page, against well under 0.1 kWh for code. On the New South Wales grid, that's roughly 530 to 3,600 kg CO2e against a few hundredths of a kilogram.
• The AI figures are estimates by researchers, not numbers published by the AI providers, and they could be off by several times in either direction. The gap is large enough that the conclusion holds anyway.

Why reading a page costs more than answering a question

AI models work in tokens, pieces of text about 4 characters long.⁹ Before a model writes anything, it has to process every token you send it. A short chat question is a few hundred tokens. A web page is thousands or tens of thousands, because the model reads the page's HTML markup as well as its words.

The best-known energy figures are for short chat prompts. Google reports 0.24 Wh for a median Gemini text prompt.¹ OpenAI's CEO has given 0.34 Wh for an average ChatGPT query.² Microsoft researchers estimate a median of 0.31 Wh for a query of 500 tokens in and 300 out on frontier models.³

Those numbers don't carry over to web pages, because energy rises with input length. Epoch AI's estimate for a GPT-4o-class model:⁴

Epoch AI, with output fixed at 500 tokens. Measurements on GPUs show the same pattern: the cost of reading the input rises with its length.⁶

How big is a web page, in tokens?

• Our practice pages: a book catalogue page on books.toscrape.com was about 10,100 prompt tokens as raw HTML and 7,072 after stripping scripts and styles. The models also wrote 2,400 to 2,900 tokens of output per page, about half of it reasoning. See the benchmark.
• A typical web page: the median HTML document is 33 to 35 KB,⁸ which is roughly 8,000 tokens.
• Real product pages: published studies measured an average of 44,706 raw tokens per page¹⁰ and 114,319 raw tokens per page,¹¹ and 477 KB of HTML per product page, roughly 120,000 tokens.¹² Careful cleaning brought pages down to 5,735 and 16,985 tokens in those studies.

So "about 10,000 tokens" is a fair size for a small or well-cleaned page. A raw product page from a large retailer can be ten times that.

Energy per page

^a Code: our benchmark measured about 59 ms of CPU time per page for downloading, decompressing and extracting, on a laptop. We assume 50 to 59 ms on a cloud server, where one virtual CPU draws 2.1 W at half load to 3.5 W at full load, times 1.135 for data-centre overhead (Cloud Carbon Footprint's AWS coefficients¹³). Low: 2.12 W × 1.135 × 0.050 s ÷ 3,600 = 0.000033 Wh. High: 3.5 W × 1.135 × 0.059 s ÷ 3,600 = 0.000065 Wh. Memory, network and storage are left out, as is a headless browser, which pages that need JavaScript would add.

The models in that table are mostly from 2025, because those are the ones researchers have estimated. Nobody publishes energy per token for the newest models.

Divide one by the other and an AI model reading a ~10,000-token page uses roughly 13,000 to 86,000 times the energy of code doing the same extraction (0.827 Wh ÷ 0.000065 Wh at the low end, 2.875 Wh ÷ 0.000033 Wh at the high end). Larger, reasoning or long-page rows are further apart still.

Both approaches have to download the page first. That part is the same either way, so it doesn't change the comparison.

Per million pages

Energy is the same wherever it's used. Carbon depends on how the electricity was made, so we show three grids: New South Wales, where AWS's Sydney region runs (0.64 kg CO2e per kWh¹⁴), the United States (about 0.35¹⁵), and the world average (0.458¹⁶).

Energy per page × 1,000,000 ÷ 1,000 = kWh; kWh × grid factor = kg CO2e. The last row assumes about 6,500 tokens per planning call and about 2 Wh per call, a GPT-4o-class upper bound. US factor: EPA eGRID2023, 770.884 lb CO2e per MWh, converted to 0.3497 kg per kWh.

A year of daily harvests

Say you collect 50,000 pages every day for a year: 18.25 million page reads. With a GPT-4o-class model reading each ~10,000-token page, that's about 43,800 to 52,500 kWh, or roughly 28 to 34 tonnes CO2e on the NSW grid. Extraction code doing the same work uses about 0.6 to 1.2 kWh.

What if these estimates are wrong?

They might be, in either direction, so here's how much it would take to change the picture.

• Too high? Microsoft researchers argue that common per-query estimates are overstated by 4 to 20 times.³ Divide the small model's 0.83 Wh by 20 and it's still about 0.04 Wh per page, roughly 640 to 1,240 times the code path, or about 41 kWh per million pages.
• Too low? The estimates above assume 1,500 output tokens or fewer. In our benchmark the models wrote 2,400 to 2,900, about half of it reasoning. Hugging Face measured open models using about 30 times more energy on average with reasoning switched on.⁷
• Cleaner electricity? AI models usually run in their provider's data centres, on a different grid from the scraper. Google reports 94 g CO2e per kWh for its fleet when it counts its clean-energy purchases, against 345 g on the local grids.¹ That would cut the AI rows' CO2e by several times. The energy used, and the ratio to code, stays the same.

How Siftyr does it

Siftyr uses AI once per site: it reads a sample page, proposes selectors from your plain-English description of the fields, and someone checks them. From then on, code reads every page and every harvest is checked against the last one. For 100 sites, that's around 0.2 kWh of AI energy at the generous estimate above, and then code at a few hundred-thousandths of a Wh per page.

Siftyr runs on AWS in Sydney, on the New South Wales grid. AWS reports a power usage effectiveness of 1.14 for its Sydney data centres in 2025.¹⁷ Our AI planning calls go to the model provider listed on our privacy page, so they use that provider's electricity.

We're not claiming Siftyr is carbon neutral. We haven't measured our whole footprint, and we aren't making offset or neutrality claims. The point here is narrower: for data collected on a schedule, reading pages with code instead of a model removes most of the energy from the job.

Assumptions, in one place

• AI energy figures are third-party estimates for specific models, mostly from 2025. No provider publishes per-token energy for its current models.
• "About 10,000 tokens per page" matches our measured practice pages (7,072 compacted, about 10,100 raw). Real product pages are often bigger.
• Code energy is based on our measured CPU time and server power coefficients from 2020–21 hardware data, so treat it as an order of magnitude.
• Downloading pages, network transfer, headless browsers and the energy used to make the hardware are left out of both sides.
• Grid factors are location-based averages: NSW 2025–26, US 2023, world 2025.
• We round to two significant figures and show ranges where sources disagree.

Sources

All sources were read on October 20, 2026. Measured page sizes and CPU times are from our benchmark.

1. Google Cloud, "Measuring the environmental impact of AI inference", August 2025 (0.24 Wh and 0.03 gCO2e per median Gemini Apps text prompt; fleet 94 g market-based vs. 345 g location-based per kWh). cloud.google.com; paper: arXiv 2508.15734
2. Sam Altman, "The Gentle Singularity", around June 10, 2025 (about 0.34 Wh per average query; method not disclosed). blog.samaltman.com
3. Oviedo et al. (Microsoft Research), Joule, 2026, DOI 10.1016/j.joule.2026.102430 (median 0.31 Wh per frontier-model query; common estimates "overstated by 4–20x"). arXiv 2509.20241
4. Epoch AI, "How much energy does ChatGPT use?", February 7, 2025 (about 0.3 Wh per query; about 2.5 Wh at 10k input tokens; almost 40 Wh at 100k). epoch.ai
5. Jegham et al., "How Hungry is AI?", arXiv 2505.09598, v6 November 24, 2025 (Wh per query at 10k in / 1.5k out: GPT-4.1 nano 0.827, GPT-4o 2.875, Claude 3.7 Sonnet 5.671, o3 12.222). arXiv 2505.09598
6. Fernandez et al., arXiv 2504.17674, April 24, 2025 (prefill energy rises with input length). arXiv 2504.17674
7. Hugging Face, AI Energy Score v2, December 4, 2025 (reasoning uses about 30 times more energy on average). huggingface.co
8. HTTP Archive, Web Almanac 2025, Page Weight, January 15, 2026 (median HTML 33 KB mobile, 35 KB desktop). almanac.httparchive.org
9. Google, Gemini API docs, "Understand and count tokens" (a token is about 4 characters). ai.google.dev
10. AICC / MinerU-HTML, arXiv 2511.16397, November 20, 2025 (44,706 raw tokens; 5,735 after preprocessing). arXiv 2511.16397
11. SCRIBES, arXiv 2510.01832, revised August 17, 2026 (114,318.6 raw tokens per page; 16,985.1 after de-duplication). arXiv 2510.01832
12. Pinterest, arXiv 2508.01096, August 1, 2025 (mean HTML 477 KB per product page; the token estimate is ours). arXiv 2508.01096
13. Cloud Carbon Footprint, methodology (AWS: 0.74 W min and 3.5 W max per vCPU; PUE 1.135). cloudcarbonfootprint.org
14. Australian Government, National Greenhouse Accounts factors via the Clean Energy Regulator (NSW and ACT, 0.64 kg CO2e per kWh, 2025–26, scope 2, location-based). cer.gov.au
15. US EPA, eGRID2023 summary data (770.884 lb CO2e per MWh, US average). epa.gov
16. Ember via Our World in Data, carbon intensity of electricity, updated June 30, 2026 (world 458.29 g CO2e per kWh, 2025). ourworldindata.org
17. AWS, Australia sustainability fact sheet (Sydney PUE 1.14 in 2025; company-reported). sustainability.aboutamazon.com

Send us a site. We'll send you a sample.

Tell us the sites, the fields and how often you need them. You'll get a checked sample before you commit to anything.

Request a sample

Model names are trademarks of their owners; Siftyr isn't affiliated with any AI provider named here. Energy estimates for AI models change as hardware and software improve, and providers may publish better figures.