Our AI agent pipeline kept stalling on CAPTCHA-heavy scraping until we rebuilt it around queues and short browser sessions

Title: Our AI agent pipeline kept stalling on CAPTCHA-heavy scraping until we rebuilt it around queues and short browser sessions Summary: The scraper got reliable only after we stopped fighting CAPTCHAs directly and rebuilt the whole thing around queues, short browser sessions, and fallback lanes so downstream AI agents could keep running without burning money on retries and stalled workflows.

AI scraping CAPTCHA bypass works best when you design for failure up front: put every URL in a queue, keep browser jobs short, retry blocked pages aggressively, and send only the hardest cases to a solver. Apify and Crawlee both point toward this pattern, with crawlers scaling from 1 to 200 concurrent requests and anti-blocking setups often using 10 retries plus fast session rotation.

I learned this the annoying way.

We had one of those scraping automations that looked great in a demo and acted possessed in production. It could click through JavaScript-heavy pages with Playwright, dodge a few basic blocks, and feed clean data into an agent workflow. Then traffic spiked, a target site tightened defenses, and the whole thing turned into a browser graveyard.

Sessions hung. CAPTCHA pages piled up. A few workers got stuck for minutes at a time. Meanwhile the queue behind them kept growing like nothing was wrong.

And the part that hurt most was not the scraper itself. It was everything downstream. Our n8n flow stopped getting fresh inputs. A classification step kept waiting on records that never arrived. A lead-enrichment agent started reprocessing stale items. A support triage workflow in Make fired on partial data and produced junk. The browser problem had become an AI agent pipeline problem.

That was the moment I stopped thinking about ai scraping captcha bypass as “how do we beat this challenge?” and started treating it as an operations problem for always-on automations. That shift fixed more than any stealth plugin ever did.

The big mistake was starting with the browser

A lot of teams build scrapers browser-first.

Open Playwright. Open Puppeteer. Open Browserless. Hit a page. If it works, keep going. If it fails, maybe retry. If a CAPTCHA appears, bolt on 2Captcha or some custom handler and pray.

That feels natural. It is also backwards.

The production pattern that actually holds up is queue-first.

This matters most if you are running AI agents in n8n, Make, Zapier, OpenClaw, or a custom worker stack that scrapes, enriches, classifies, and acts on web data. In those systems, scraping is usually the first domino. If it stalls, your extraction prompts never run, your deduping logic gets partial batches, your lead scoring agent makes decisions on old data, and your orchestration layer starts doing expensive nonsense to compensate.

Apify Request Queue gets this exactly right. You enqueue URLs with a uniqueKey, so duplicates disappear before they waste work. You can crawl breadth-first or depth-first depending on the target. And the retention detail matters more than most people realize: named queues are retained indefinitely, while unnamed queues expire after 7 days.

That sounds boring until your job crashes halfway through a 4-day crawl and you realize “boring” is another word for “we didn’t lose the crawl state.”

Then Crawlee layers autoscaling on top. It starts at 1 concurrent request and scales upward over time to a default maxConcurrency of 200 unless you cap it. That’s not just a performance detail. It changes how failure behaves.

When a page gets blocked, you want it to go back into a queue with metadata about what happened. You do not want one long-lived browser worker sitting there, holding memory, holding a proxy, holding session state, waiting for a miracle.

Once you fix that, the next bottleneck is usually not scraping anymore. It is LLM cost and orchestration behavior under retries, enrichment loops, and 24/7 agent runs. That is exactly where predictable flat-rate AI compute changes what you can safely automate. If your cleaned scrape output plugs into existing OpenAI-compatible SDKs or HTTP clients, you can keep the agent layer simple while removing the per-call panic that shows up when workflows retry, branch, and run nonstop.

And that leads to the next uncomfortable truth.

CAPTCHAs usually aren’t the real problem, are they?

Most scraping teams obsess over the puzzle on the screen.

That’s understandable. A giant Cloudflare Turnstile or Arkose Labs challenge feels like the enemy because it is visible. But ScrapingBee’s anti-blocking guidance makes a point I wish more people took seriously: CAPTCHAs are a symptom, not the problem.

Apify’s anti-scraping academy says basically the same thing from a different angle. Fix upstream signals first:

• use residential proxies when the target is aggressive
• send realistic headers and fingerprints
• use a full browser only where it’s actually needed
• increase retries a lot more than your instincts say is reasonable

The most counterintuitive production lesson here is that success does not mean every request passes on the first try. Apify is explicit about this: if even 10% of requests get through, retries plus session rotation can still scrape the whole site.

That number sounds absurd until you’ve seen it work.

A weak scraper treats a block as a fatal error. A strong scraper treats a block as routing information.

For AI agents, that difference is huge. If a blocked page kills the workflow, you waste downstream model calls on half-built records or trigger retries that rerun extraction and classification on the same bad input. If a blocked page just requeues cleanly, the rest of the automation can keep moving.

That’s where session design starts to matter more than stealth tricks.

Why do bounded browser sessions change the whole architecture?

Because browsers are not free, and hosted browsers are even less free.

Browserless makes this painfully concrete. It charges in 30-second units. Reconnect and you start a fresh unit. Successful CAPTCHA solves cost 10 units. Residential proxy traffic costs 6 units per MB and datacenter proxy traffic costs 2 units per MB.

Then there are the plan limits. Free gets 2 concurrent browsers and 2-minute max sessions. Prototyping gets 10 browsers and 15-minute sessions. Starter gets 40 browsers and 30 minutes. Scale gets 100 browsers and 60 minutes.

That pricing model is basically a design review in disguise. It rewards short, bounded browser sessions and punishes giant sticky sessions that try to crawl, solve, retry, and paginate forever inside one worker.

If your browser lane is doing everything, your cost curve gets ugly exactly when traffic spikes.

And in an AI automation stack, browser cost is only half the story. The bigger problem is end-to-end pipeline economics: a stuck scrape can trigger duplicate enrichment, repeated extraction prompts, extra deduping passes, and agent decision loops that keep firing because upstream state never settled. Browserless units and 2Captcha solves are visible costs. Wasted LLM calls from unstable orchestration are the sneakier one.

That is why flat-rate AI compute matters here. Once the scrape layer is queue-first and failure-tolerant, you can let classification, extraction, support triage, or lead scoring agents run continuously without babysitting token spend every time a workflow retries or branches.

And traffic spikes are when your architecture tells the truth.

The session rotation trick that feels wrong but works

The thing that finally made our scraper less fragile was giving up on “saving” a session.

Playwright’s BrowserContext model is perfect for this. Each context gets isolated cookies, storage, and session state. That means you can stop pretending one heroic browser session should survive the whole crawl.

Crawlee’s SessionPool pushes the same idea further by tying cookies and identifiers to a specific proxy or session, then retiring blocked sessions fast. Apify’s own recommendation for blocked sites is aggressive enough that it should reset how you think about retries: maxRequestRetries: 10 and maxErrorScore: 1.

That is not “be careful.” That is “expect failure constantly and design for rapid replacement.”

Here’s the shape of it:

import { BasicCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({});

const crawler = new BasicCrawler({
  useSessionPool: true,
  sessionPoolOptions: { maxPoolSize: 100 },
  async requestHandler({ request, session }) {
    const proxyUrl = await proxyConfiguration.newUrl(session.id);
    // retire blocked sessions fast
    // session.retireOnBlockedStatusCodes(response.statusCode)
  },
});

That tiny design choice changes everything. Instead of nursing broken sessions back to health, you rotate them out and keep the queue moving.

For teams running LLM-powered automations, this is the difference between a scrape stage that starves the rest of the system and one that keeps feeding it. Your classifier can keep labeling new records. Your extraction step can keep structuring page data. Your agent can keep deciding whether to alert, enrich, score, or ignore. The pipeline stays alive because the browser layer stops acting like a single point of failure.

What should actually happen when a CAPTCHA appears?

Not what most teams do.

The wrong answer is: “send every challenge to 2Captcha.” That turns your fallback into your main lane, which is both expensive and slow.

The right answer is a branching pipeline:

1. Try the cheapest path first: plain HTTP, Playwright.request, or a curl-impersonate style approach if the page allows it.
2. Escalate to a browser only when JavaScript or anti-bot logic makes that necessary.
3. If blocked, retry with a fresh session and rotated proxy.
4. Only after repeated failure, hand off to a solver like 2Captcha.
5. Put the result back into the queue and continue.

That last part matters. Solver handoff should be a lane, not a detour that hijacks the whole worker.

ZenRows has a nice concrete Playwright + 2Captcha example against a reCAPTCHA demo page. The flow is clean: launch Playwright, locate the CAPTCHA iframe, extract the site key from the iframe src, click the checkbox, request a token from 2Captcha, inject the token, continue.

from playwright.sync_api import sync_playwright
from twocaptcha import TwoCaptcha

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    solver = TwoCaptcha("<YOUR_API_KEY>")
    page.goto("https://patrickhlauke.github.io/recaptcha/")
    captcha_frame = page.wait_for_selector("iframe[src*='recaptcha']")
    site_key = captcha_frame.get_attribute("src").split("k=")[-1].split("&")[0]

Useful? Absolutely.

But the real lesson is architectural, not tactical. The solver is a specialized service sitting behind your queue, not the beating heart of the crawler.

And once the page finally clears, that output should move straight into the next AI step: classify the page, extract structured fields, dedupe against prior records, score the lead, or let an agent decide whether to trigger an email, CRM update, or support action. If the scrape stage is unstable, every one of those model-powered steps becomes noisier and more expensive.

And the throughput numbers make that obvious.

The solver math gets weird fast

2Captcha publishes enough numbers to make budgeting real.

Cloudflare Turnstile is listed at $1.45 per 1,000 solves with 2,883 free capacity per minute. reCAPTCHA v3 runs $1.45 to $2.99 per 1,000 with 757 per minute capacity. Arkose Labs ranges from $1.45 to $50 per 1,000 with 578 per minute capacity.

Those are not scary numbers if solver usage is rare.

They get scary when your architecture starts funneling every blocked request into that lane because your upstream anti-blocking is weak and your retry semantics are sloppy.

They get even worse when you look at the whole workflow instead of the CAPTCHA bill. One blocked product page can mean a missed extraction, a delayed classifier, a duplicate retry in Zapier, and another LLM pass to reconcile inconsistent records. One broken support page can mean a triage agent never sees the ticket context it needs. One flaky lead source can cause a scoring model to rank stale prospects because the fresh ones never made it through.

This is why I’m opinionated about it: if your first instinct is “we need better CAPTCHA bypass,” you probably need better queue behavior, better session retirement, and better browser boundaries.

The CAPTCHA is often just the invoice for mistakes you made earlier.

Do you even need to run your own browser fleet?

Sometimes no.

For some targets, ScrapingBee, ZenRows, or Browserless is the sane choice. Browserless gives you direct Playwright and Puppeteer WebSocket endpoints, plus Browserless-specific CDP extensions like solveCaptcha and reconnect. It also offers regional endpoints in San Francisco, London, and Amsterdam, which is genuinely useful when geography affects block rates.

If you do not want to manage Chrome versions, browser crashes, and fleet capacity, outsourcing that layer can be a relief.

But there’s a tradeoff. You usually give up some control over retry semantics, session lifecycle, and exactly how cost behaves under spikes.

That’s why I like a split-brain design:

• keep queueing, retries, and routing in your own logic
• outsource browser execution only if it saves operational pain
• reserve solvers for the fallback lane

That way your architecture still belongs to you.

And if the cleaned scrape output is heading into an OpenAI-compatible SDK or plain HTTP client anyway, keeping control of the queue and retry layer makes the rest of the agent stack much easier to reason about.

The boring crawler example that taught me the most

One of the most useful examples I’ve seen is not flashy at all: a Crawlee PlaywrightCrawler that recursively scrapes Hacker News with RequestQueue, enqueues next-page links, and defines a failedRequestHandler for pages that exceed retry limits.

That’s it. No magic stealth sauce. No macho “zero blocks” fantasy.

Just queueing, recursion, retries, and explicit failure handling built into the crawler itself.

That’s what production looks like.

If you want a lighter lane for less hostile targets, even a simple CheerioCrawler setup can go surprisingly far:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  maxConcurrency: 100,
  maxRequestsPerMinute: 250,
});

And that might be the most underrated move in this whole stack: don’t spend browser budget where plain HTTP will do.

Especially when your agent pipeline is doing more than scraping. If the next steps include extraction, classification, deduping, or decision-making, every unnecessary browser session adds latency and failure risk before the valuable model work even begins.

The production rule I wish someone had drilled into me earlier

Treat every request as disposable. Treat every session as temporary. Treat every CAPTCHA as a fallback event.

If you build around those three assumptions, a lot of design decisions get easier.

You stop asking, “How do I make this browser survive forever?” and start asking, “How quickly can this work item fail, requeue, rotate identity, and try again?”

That sounds less heroic. It is also how scrapers stay alive when the target changes behavior at 2 a.m.

My practical takeaway is simple: start with Apify Request Queue or something equivalent, let Crawlee scale concurrency deliberately instead of chaotically, isolate sessions with Playwright BrowserContext or SessionPool, and keep Browserless or 2Captcha behind a queue boundary. Then connect the cleaned output to the LLM layer in a way that can survive retries, spikes, and 24/7 operation.

That is the part more teams miss: queue-first scraping solves the browser meltdown, but predictable unlimited LLM compute is what makes the full agent workflow viable at scale. Once both layers stop punishing you for retries and nonstop automation, always-on agents become much easier to trust.