Can't I just scrape it with ChatGPT?

The short answer

Sometimes, yes. If you need one small table from one public page, once, an AI assistant like ChatGPT or Gemini is a perfectly good tool, and you don't need us.

The difference shows up when the data has to arrive correct, every week, from sites that don't want to be scraped, and someone in your company is making decisions with it. That isn't a question of whether a page can be read once. It's whether a feed can be trusted for a year.

What AI assistants do well

• Pulling a small table from a simple, public page.
• Reshaping, summarising or cleaning data you already have.
• Writing a first draft of a scraping script.

If that's all you need, use them. Here's where they run out.

Where asking an AI assistant falls short

1. Many sites block AI assistants by name

AI assistants read the web through their own, publicly declared bots. A growing number of publishers, retailers and marketplaces block those bots outright. When that happens the assistant can't see the page. Some will tell you. Others fill the gap from what they learned in training, which may be months out of date.

2. Bot protection

Many travel, ticketing, retail and marketplace sites sit behind bot-protection services that show a challenge page, a CAPTCHA or an empty result to anything that doesn't behave like a real visitor's browser. The assistant reads the challenge, not the prices.

3. Pages that need interaction or a location

"Load more" buttons, infinite scroll, filters, and prices that change by country. Chat assistants generally read what's on the page when it first loads, from wherever their servers happen to be.

4. Volume

Ask for 20 rows and you'll usually get 20 rows. Ask for 20,000 and the answer gets cut short, skips pages or, worse, fills in plausible-looking rows that aren't on the site. A language model's job is to produce likely text. When it can't see the data, likely text is what you get.

5. No paper trail

A chat answer doesn't tell you which page each row came from or when it was read. If a number looks wrong, there's nothing to check it against, and no way to tell whether last week's answer and this week's were collected the same way.

“Then I’ll have the AI write a scraper.”

This is the better idea, and the script often works on the first try. Here's what happens next:

1. Someone has to run it. Install Python, keep a computer or server on, remember to start it.
2. It gets blocked. Hundreds of requests from one office or laptop look like a bot, because they are. Getting past that means paying for proxies, rotating them, adding a real browser and retrying failures.
3. It breaks quietly. Sites change their layout. The script usually doesn't crash; it returns empty columns or the wrong field, and nobody notices until a report is wrong.
4. It has to run on a schedule. "Every Monday at 6am, into the dashboard" means a scheduler, alerts when it fails, and someone on the hook when it does.
5. Nothing checks the data. A price of 0, a missing product, the same row twice. A script passes all of it straight through.
6. It needs an owner. The person who wrote the prompt moves on. The script stays, and so does the business that depends on it.

None of these is hard on its own. Together they're a small, permanent engineering job, and it's usually one of your most expensive people doing it.

“But we scraped 50,000 rows with an AI agent.”

Fair. Coding agents and browser agents are a real step up from a chat window. Give one a terminal and a browser, and it will write a scraper, run it, read the errors, fix them and keep going until it has 50,000 rows. For a one-off pull by someone who knows what they're doing, that's a legitimate way to work.

But "the agent got 50,000 rows" answers the easiest question. These are the ones that decide whether you can rely on it.

Did the agent read the pages, or write code that did?

An agent can do the job in one of two ways, and each has a catch.

• The agent reads every page itself. Every page passes through the model. A product page is often tens of thousands of tokens of HTML. Even trimmed to 5,000 tokens, 50,000 pages is 250 million input tokens per run, before retries and output. At a few dollars per million tokens, that's hundreds of dollars a run, every run, and the model can still misread a page or fill in a field it couldn't find.
• The agent writes a script and runs it. This is cheap per page, and it's what most good agent runs do. But what you end up with is an AI-written script on someone's laptop, with every problem in the section above. The agent just got you to day one faster.

How many of the 50,000 are right?

Agents are built to report success. "Done: 50,000 records saved" doesn't tell you how many were blocked pages saved as if they were products, rows with an empty price, duplicates from overlapping pages, or a list that stopped at page 200 of 400. Unless someone checked a random sample against the live site, nobody knows.

Will next week's run match this one?

Run the same prompt again and an agent may write different code, choose different fields, name columns differently or take a different path through the site. That's fine for exploring. It's a problem when a dashboard, a pricing model or a client report expects the same columns, the same way, every week.

Who runs it on Monday?

An agent works while someone is watching a terminal. A feed has to run at 6am with no one there, retry what failed, hold back a bad run and tell someone when the site changed. With an agent, that "someone" is still on your payroll.

What did it do to get the data?

An agent told to "get all the rows" will look for a way through whatever stops it. You may not know whether it ignored robots.txt, ran thousands of requests from your office network, or used credentials it found in your environment. When the data feeds a business decision, how it was collected matters as much as what was collected.

Side by side

We use AI too

Siftyr uses AI where it helps: reading a page from a plain-English description of the fields you want, and suggesting checks from the shape of your data. The difference is everything around it. Fetching that gets through, checks that catch mistakes before they ship, a history of every run, and people whose job is to fix breakages.

Which should you use?

• An AI assistant if you need it once, it's small, and you'll eyeball the result.
• An AI agent or AI-written script if you need a big dataset once, you have an engineer who'll check it, or the site is friendly and an occasional gap is fine.
• Siftyr if prices, reports or decisions depend on the data, the sites push back, it needs to arrive on a schedule, or nobody on your team should be babysitting scrapers.

A five-minute test

Before you decide, ask your AI tool for the data you actually need, then check:

1. Did it get every row? Compare the count with the site.
2. Pick five rows at random and open their pages. Do they match?
3. Run it again tomorrow. Same result, or a different one?
4. Try page 50, or another country. Does it still work?
5. If it stops working next month, who finds out, and how?

If every answer is fine, keep using it. If not, send us the sites and fields you need.

Already have 50,000 rows from an agent?

Ask whoever ran it:

1. How many rows did a person check against the live site, and how many were wrong?
2. How many pages came back blocked, empty or as a CAPTCHA, and were any saved as data?
3. Is the total what the site actually lists, or just where the run stopped?
4. What did the run cost, in tokens, proxies and hours of someone's time?
5. If you run it again next Monday without them, do you get the same columns?

If the answers hold up, you've got a good one-off dataset. If the plan is to do it every week, that's the job Siftyr does.

Send us a site. We'll send you a sample.

Tell us the sites, the fields and how often you need them. You'll get a checked sample before you commit to anything.

Request a sample

Siftyr follows robots.txt by default. ChatGPT is a trademark of OpenAI and Gemini is a trademark of Google; Siftyr isn't affiliated with either. AI assistants change quickly, so what they can and can't do may differ from what's described here.