Web Scraping Infrastructure for AI Agents (2026 Guide)

The list of catalogue sites the data team wanted tracked kept growing, and the crawler behind it still ran as a cron job on a lone server. When it stalled overnight, the morning report had gaps, and nobody could say which page had broken it or why.

Web scraping infrastructure for AI agents is everything under the agent's logic: browsers it can start on demand, a network path sites accept, saved logins, a record of each run and a machine to process the output. Solari supplies those pieces as cloud browsers, proxies, profiles, recording and sandboxes on one account and one API, while the agent, the model and the extraction code stay the team's own. This guide walks through each layer in the order a scraping job meets it, then what a fleet costs.

What the Infrastructure Includes

The easiest way to plan it is to list the jobs a crawl does that have nothing to do with the data. Web scraping is fetching a page and extracting from it, and every fetch needs a browser, an address, sometimes a login, and a place for the output. The figure below pairs each layer with the Solari piece that covers it.

Each layer adds cost, so a job should add one only when a site asks for it. The browser fleet comes first because everything else attaches to it.

Browsers an Agent Can Start on Demand

The first layer replaces the lone server with sessions created by API. A Solari session is a real Chrome in the cloud driven with the Playwright API, and existing tools attach through its CDP endpoint with Playwright's connectOverCDP. The default is a fast headless browser, and every option is opt-in, so a plain session carries no proxy, profile or recording until the job sets one.

A worker opens a session, reads the page and closes it in a finally block, so a crash never leaves a browser running:

const browser = await client.launch()
try {
  const page = await browser.newPage()
  await page.goto(url)
  rows.push(await page.locator("table").innerText())
} finally {
  await browser.close()
}

const browser = await client.launch()
try {
  const page = await browser.newPage()
  await page.goto(url)
  rows.push(await page.locator("table").innerText())
} finally {
  await browser.close()
}

const browser = await client.launch()
try {
  const page = await browser.newPage()
  await page.goto(url)
  rows.push(await page.locator("table").innerText())
} finally {
  await browser.close()
}

Width comes from the plan: Solari's Professional plan runs 150 concurrent browsers, so the crawler becomes a pool of workers instead of a queue on a single machine. A fleet that wide is also easier for sites to notice, which is where the network layer comes in.

Network and State: Proxies and Profiles

Many sites judge a visitor by where its traffic comes from before anything else. A proxy server puts another address between the browser and the site, and Solari proxies offer rotating residential IPs by default, a static ISP address held all session, or mobile carrier IPs, in the country the session names. Sessions that share a proxy.session label share an exit IP, so a multi-step flow keeps its address.

Proxies require stealth, and stealth sessions sit under a separate stealth cap counted on top of the browser limit; on Professional it is 50, and a session over it gets a 429. A fleet that scrapes both open and guarded sites therefore runs as two pools. The companion piece on scraping agents that meet anti-bot walls matches each block to its setting.

Some data sits behind a login. A Solari profile saves a login with its cookies and site data, and a session that attaches it by profileId starts signed in, so the agent never types a password mid-crawl. With access settled, the next layer explains the runs that still fail.

Recording as Evidence When a Run Breaks

A report with gaps needs evidence, not a guess about a selector. A session created with the recording flag can be replayed in the console from a temporary link, or downloaded as a line-by-line log of what the page served. Replays are kept for 30 days on Professional.

Recording captures input values by default, so a session that types a secret should run unrecorded. Recording a sample of workers is usually enough to explain a bad batch. Once pages arrive intact, the output needs somewhere to go.

Processing Results in a Sandbox

Raw page text is rarely the deliverable, and parsing it on the crawler's own machine brings back the lone-server problem. A Solari sandbox is an isolated Linux machine on the same account, where runCode runs Python that keeps its variables between calls and the files API writes and downloads results. Cleaning code the agent wrote runs there, away from the systems that hold credentials.

By default a sandbox pauses when idle and resumes as it was, so a nightly clean-up job picks up where it stopped. That leaves the bill.

What a Scraping Fleet Costs

Cost grows in three places: browser time, proxy traffic and solved puzzles. The browser hour falls with each plan, as the figure shows.

Solari bills browsers at $0.07 per hour on Professional, whose $200 monthly fee comes back as usage credits, and proxy traffic at $4.00 per GB against $6.00 on Starter. Captcha solving costs $0.005 per solve from Professional up.

Proxy bandwidth is usually the line that grows. The blockHeavy option refuses video, audio and large images before their bodies download, so they never cross the proxy, and sites that accept a plain session should stay on one.

Choosing the Right Setup

Scraping infrastructure for an agent is a stack added in order: a browser fleet sized to the plan, proxies and profiles where a site asks for them, recording where a batch goes wrong, and a sandbox for the parsing. Start every site on a plain session and keep the stealth pool sized to its cap. The Solari introduction shows how browsers and sandboxes share the account.

Get Started With Solari

The first job to move is the crawler's least reliable site: a plain session with recording on, then a proxy country if the replay shows a block. Solari keeps that fleet fast, with browsers started on demand by API, and cheap, at the Professional rate above and lower again on Team. Create a free Solari account and start on the Free plan's $3 of monthly credits.