Web Scraping With AI Agents That Handle Anti-Bot (2026 Guide)

The enrichment agent was asked to collect opening hours for every clinic on a directory site, and it reported success before lunch. The spreadsheet said otherwise: row after row read "Verify you are human", copied faithfully from a challenge page the model had taken for content.

That is the real anti-bot problem for an AI agent: a language model extracts whatever text a page shows, so a blocked page turns into confident bad data instead of an error. A scraping agent that handles anti-bot does three things in code: it spots the block before extraction, it moves that site to a stronger browser setting and remembers what worked, and it paces itself under the plan's limits. Solari offers every setting on that ladder (stealth, proxies, captcha solving and saved logins) as an option on the same cloud browser session, so escalation changes the create call and nothing else. This guide builds the loop step by step.

Detect the Block Before the Model Reads the Page

Detection comes first, because every later step depends on knowing a wall was hit. The cheapest signal is the response status: a 403 Forbidden or a 429 on the main document means the site refused the visit. The next signal is the page itself, such as a title like "Just a moment" or "Access denied", or a selector the extraction expects on every real page that is missing.

Run those checks in plain code before the model reads the page, and treat a match as a failed fetch rather than as content. A check that runs after extraction is too late, since the model has already summarised the challenge text into the row.

When a check fires and nobody can see why, a recording settles it. A Solari session created with the recording option can be replayed in the console, so the challenge page is on screen rather than guessed from logs. Recording captures input values by default, so a session that types a password should run without it. Once a block is detected reliably, the agent can react to it.

Escalate per Site, Not per Fleet

The reaction is to move only the blocked site up a ladder of settings, because most sites never need the heavier rungs. The figure below lists the rungs in the order an agent should try them.

The ladder starts at Solari's default mode, the fastest to start and to load pages, which suits public pages and structured-data scraping. Stealth makes the browser look like an ordinary person's, which answers a site that reads the browser fingerprint. A proxy then changes where the traffic appears to come from: Solari proxies offer rotating residential, static ISP and mobile carrier IPs by country, and a shared proxy.session label keeps a multi-step flow on a single exit IP. Proxies and captcha solving both require stealth and a paid plan, so each rung carries the one before it.

The loop below starts a site at the rung where it last succeeded and climbs only on a block. Its memory is kept per domain, so the agent pays for stealth on the shop that needs it and stays on default sessions everywhere else.

const ladder = [{}, { stealth: true }, { stealth: true, proxy: "us" }]

async function scrape(url: string) {
  const site = new URL(url).hostname
  for (let rung = memory.get(site) ?? 0; rung < ladder.length; rung++) {
    const browser = await client.launch(ladder[rung])
    try {
      const page = await browser.newPage()
      const res = await page.goto(url)
      if (!looksBlocked(res?.status(), await page.title())) {
        memory.set(site, rung)
        return await page.content()
      }
    } finally {
      await browser.close()
    }
  }
  throw new Error(`blocked on every rung: ${site}`)
}

const ladder = [{}, { stealth: true }, { stealth: true, proxy: "us" }]

async function scrape(url: string) {
  const site = new URL(url).hostname
  for (let rung = memory.get(site) ?? 0; rung < ladder.length; rung++) {
    const browser = await client.launch(ladder[rung])
    try {
      const page = await browser.newPage()
      const res = await page.goto(url)
      if (!looksBlocked(res?.status(), await page.title())) {
        memory.set(site, rung)
        return await page.content()
      }
    } finally {
      await browser.close()
    }
  }
  throw new Error(`blocked on every rung: ${site}`)
}

const ladder = [{}, { stealth: true }, { stealth: true, proxy: "us" }]

async function scrape(url: string) {
  const site = new URL(url).hostname
  for (let rung = memory.get(site) ?? 0; rung < ladder.length; rung++) {
    const browser = await client.launch(ladder[rung])
    try {
      const page = await browser.newPage()
      const res = await page.goto(url)
      if (!looksBlocked(res?.status(), await page.title())) {
        memory.set(site, rung)
        return await page.content()
      }
    } finally {
      await browser.close()
    }
  }
  throw new Error(`blocked on every rung: ${site}`)
}

The top rungs need one more check each, and that is where escalation fails quietly.

Confirm Each Setting Actually Applied

A rung that silently did not apply looks exactly like a rung that did not work, so the agent should verify each one. For proxies, Solari returns a proxy object on the session with the country, the tier and a timezone matched to the IP; check that session.proxy is present rather than trusting a successful create call.

Captcha solving covers reCaptcha, hCaptcha and Turnstile widgets and runs in the background from the moment the widget appears, but it is not instant. The agent waits for the widget's response field to fill before it submits, or the form goes out without a token.

A login wall is the last rung. A Solari profile saves a login with its cookies and site data, and it seeds only the session's default context, so the agent opens pages from that context rather than from a new one it creates itself. With every rung verified, the remaining limit is how many escalated sessions run at once.

Pace the Fleet Under the Plan's Limits

Pacing matters because escalated sessions are scarcer than default ones. Solari counts stealth sessions under a separate cap on top of the browser limit, 10 at once on Starter and 50 on Professional, and a stealth session over the cap gets a 429 even when browser slots are free. Size the escalated worker pool to that cap and retry with exponential backoff rather than a tight loop.

Politeness is part of not getting blocked. A crawler that reads robots.txt, spaces its requests to a single host and stops after repeated refusals trips fewer defenses than one that only rotates addresses.

Cost follows the same per-site logic. Proxy traffic bills at $4.00 per GB on Professional, and the blockHeavy option drops video, audio and oversized images before they cross the proxy. Captcha solving bills $0.005 per solve on the same plan, so it belongs only on the sites whose replays show a widget.

Choosing the Right Setup

A scraping agent that handles anti-bot is a loop, not a configuration. Detect the block in code before extraction, climb the ladder for that site alone, verify that each setting took effect and size the escalated pool to the stealth cap. Every site starts on the cheapest, fastest session and earns its way up. The Solari introduction covers the sandboxes on the same account, which suit cleaning the data once it lands.

Get Started With Solari

The first job is to run the block check against the sites that already return odd rows, then put only those sites on stealth with a proxy country. Solari browsers are ready in 8 ms and Solari is about 2x cheaper than Browserbase per browser hour, so a fleet that climbs the ladder only where it must stays fast and cheap. Create a free Solari account to build the detection loop on default sessions, then add stealth on a paid plan for the sites that need it.