Web Scraping Without Getting Blocked: 2026 Guide
Web scraping without getting blocked comes down to one principle: behave like a considerate human client, not an abusive bot. That means respecting the target site's rules, spreading your requests thin, presenting realistic browser signals, and using a captcha solver to solve the occasional CAPTCHA cleanly. This guide walks through the full stack for legitimate, authorized data collection: QA testing your own forms, monitoring, accessibility, and contracted research, so your crawler stays reliable at scale.
Before any technique: only scrape data you are permitted to collect. Public data, data you own, or data you have written authorization to gather. The tactics below keep authorized crawlers stable; they are not a license to ignore terms of service.
Start With Rules, Not Tricks
The fastest way to avoid getting blocked scraping is to not trigger defenses in the first place.
- Read robots.txt. Fetch
https://example.com/robots.txt and honor Disallow paths and Crawl-delay. See the official robots.txt spec (RFC 9309) (https://www.rfc-editor.org/rfc/rfc9309.html) for parsing rules.
- Respect Terms of Service. If the ToS forbids automated access, get written permission or use an official API instead.
- Rate-limit yourself. Honor Retry-After headers and back off on 429 / 503 responses.
- Identify yourself when appropriate. For authorized crawls, a descriptive User-Agent with contact info builds trust with the site owner.
A polite crawler that a site operator would tolerate is one that almost never gets banned.
Rotate Residential and Mobile Proxies
Datacenter IPs are the first thing anti-bot systems flag. For serious scraping, scraping proxies from residential or mobile pools blend into normal traffic.
- Datacenter: detection risk high, cost low, best for non-hostile targets and internal QA
- Residential: detection risk low, cost medium, best for most public-web scraping
- Mobile (4G/5G): detection risk lowest, cost high, best for aggressively defended sites
Rotate the exit IP per session or per N requests, keep one IP per logical session so cookies stay consistent, and geo-match the proxy to the content you request. Never hammer a single IP - that is the clearest bot signal there is.
import random
import requests
PROXIES = [
"http://user:pass@resi1.example:8000",
"http://user:pass@resi2.example:8000",
]
def get(url):
proxy = random.choice(PROXIES)
proxies = dict(http=proxy, https=proxy)
return requests.get(url, proxies=proxies, timeout=20)
Send Realistic Headers and Rotate User-Agents
A bare HTTP client sends a fingerprint no browser ever would. Match a real browser's header order and values.
HEADERS = dict([
("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/128.0 Safari/537.36"),
("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
("Accept-Language", "en-US,en;q=0.9"),
("Accept-Encoding", "gzip, deflate, br"),
("Referer", "https://www.google.com/"),
("Upgrade-Insecure-Requests", "1"),
])
Rotate the User-Agent from a small pool of current, real strings; outdated versions stand out more than a static one. Keep the rest of the headers internally consistent (an Accept-Language that matches your proxy's geo, for example).
Beat Fingerprinting With Anti-Detect Browsers
Modern anti-bot walls read far more than headers: JavaScript execution, canvas/WebGL fingerprints, the navigator.webdriver flag, TLS/JA3 signatures, and mouse timing. To bypass anti-bot detection on JS-heavy sites, drive a real browser and strip the automation tells.
- undetected-chromedriver patches Selenium's obvious markers.
- playwright-stealth hides navigator.webdriver and normalizes fingerprints in Playwright.
from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
stealth_sync(page)
page.goto("https://example.com")
print(page.title())
browser.close()
Add a realistic viewport, timezone, and locale, and prefer headless=new or headful mode; legacy headless is trivially detected.
Throttle, Add Jitter, and Cap Concurrency
Robots are betrayed by their rhythm. Constant 200ms intervals scream automation.
- Randomize delays. Sleep a random 2-8 seconds between requests, not a fixed value.
- Add jitter so no two sessions share a pattern.
- Cap concurrency to a handful of workers per domain.
- Exponential backoff on errors instead of instant retries.
import random
import time
time.sleep(random.uniform(2.0, 8.0)) # human-like pacing
Cache and Crawl Incrementally
The politest request is the one you never send. Cache aggressively and only fetch what changed.
- Store responses and honor ETag / Last-Modified with conditional If-None-Match requests to get cheap 304 responses.
- Track a last_seen timestamp per URL and skip unchanged pages.
- Deduplicate your URL frontier so you never crawl the same page twice in one run.
Incremental crawling slashes request volume, which is the single biggest factor in staying under the radar.
Handle CAPTCHAs With a Solver
Even a well-behaved crawler eventually meets reCAPTCHA, hCaptcha, Turnstile, or GeeTest. To handle captcha scraping without stalling your pipeline, solve captcha challenges by handing them to an AI solver. OMOCaptcha (https://omocaptcha.com/en?utm_source=blog&utm_medium=organic) solves 14 captcha systems from a single API: 0.42s average solve time, up to 99% accuracy, from $0.27 per 1000 solves, with no human-farm queue delay.
The API uses a simple two-step flow: create a task, then poll for the result. HTTP status is always 200; success is decided by errorId (0 = success).
import time
import requests
API = "https://api.omocaptcha.com/v2"
KEY = "YOUR_API_KEY"
# 1) Create the task
payload = dict(
clientKey=KEY,
task=dict(
type="RecaptchaV2TokenTask",
websiteURL="https://example.com/login",
websiteKey="SITE_KEY_HERE",
),
)
task = requests.post(API + "/createTask", json=payload).json()
task_id = task["taskId"]
# 2) Poll until ready
token = None
while True:
res = requests.post(API + "/getTaskResult", json=dict(clientKey=KEY, taskId=task_id)).json()
if res["status"] == "ready":
token = res["solution"]["gRecaptchaResponse"]
break
time.sleep(3)
print("Token:", token[:40], "...")
Read the token from solution (solution.gRecaptchaResponse for reCAPTCHA/hCaptcha, solution.token for most others) and inject it into the form submission. For non-reCAPTCHA types such as HCaptchaTokenTask, TurnstileTokenTask, FunCaptchaTokenTask, or GeeTestTask, confirm the exact type string in the OMOCaptcha API docs before use.
For step-by-step, per-captcha walkthroughs, see How to solve reCAPTCHA (https://blog.omocaptcha.com/how-to-solve-recaptcha), How to solve hCaptcha (https://blog.omocaptcha.com/how-to-solve-hcaptcha), and the Cloudflare Turnstile solver (https://blog.omocaptcha.com/cloudflare-turnstile-solver) guide.
The Anti-Block Checklist
- [ ] Checked robots.txt, ToS, and confirmed authorization
- [ ] Residential/mobile proxy rotation with sticky sessions
- [ ] Realistic, internally consistent headers + current User-Agent pool
- [ ] Anti-detect browser (undetected-chromedriver / playwright-stealth) for JS sites
- [ ] Randomized delays, jitter, and capped concurrency
- [ ] Exponential backoff on 429 / 503
- [ ] Caching + incremental crawls with ETag/Last-Modified
- [ ] CAPTCHA solver wired in for challenges
FAQ
Why do I keep getting blocked even with proxies?
Proxies fix your IP reputation but not your behavior. If your headers are inconsistent, navigator.webdriver is exposed, or your timing is robotic, sites still detect you. Combine proxies with an anti-detect browser and human-like pacing.
Is web scraping legal?
Scraping public data is broadly permitted in many jurisdictions, but it depends on the data, the site's ToS, and local law. Always scrape only authorized or public data, respect robots.txt, and consult counsel for anything sensitive.
How many requests per second are safe?
There is no universal number. Start slow: one request every few seconds per domain, watch for 429 responses, and back off. Politeness beats speed; a slow crawler that never gets banned wins.
Which proxies are best for avoiding blocks?
Residential proxies suit most public-web work. Reserve pricier mobile proxies for aggressively defended targets. Datacenter proxies are fine only for non-hostile or internal QA sites.
How do I handle CAPTCHAs at scale?
Route challenges to an AI solver like OMOCaptcha via its createTask / getTaskResult API. It returns a token in well under a second, which you inject into the form - no human queue, no pipeline stall. Compare options in best captcha solving service (https://blog.omocaptcha.com/best-captcha-solving-service-2026).
Start Scraping Reliably Today
Do the polite-crawler basics, add clean proxy and fingerprint hygiene, and let an AI solver clear the CAPTCHAs so your authorized pipeline never stalls.
Sign up for OMOCaptcha (https://omocaptcha.com/en?utm_source=blog&utm_medium=organic) and get 1000 free solves, no card required. Explore transparent pricing from $0.27/1000 (https://omocaptcha.com/en#pricing), or email support@omocaptcha.com (24/7) with any question. Full refund if your success rate drops below 95%.