Selenium Web Scraping: The Complete Python Guide

Selenium is the oldest and most widely deployed browser automation framework: it drives a real Chrome, Firefox, Edge, or Safari through the W3C WebDriver protocol, so your scraper sees exactly what a user sees — JavaScript executed, AJAX applied, DOM fully built. This guide covers the complete scraping workflow in Python: setup with Selenium Manager, finding elements, waiting correctly, proxies, browser profiles, downloads, screenshots, exception handling, parallel execution with Grid, an honest look at bot detection, and how Selenium compares to Playwright and Puppeteer in 2026.

Key Takeaways

• pip install selenium is the whole install — Selenium Manager (built in since 4.6) downloads matching drivers automatically, so webdriver-manager and manual chromedriver downloads are obsolete
• Use find_element(By.CSS_SELECTOR, ...); the old find_element_by_* methods were removed in Selenium 4.3
• element.text returns only visible text — for hidden or collapsed elements use get_attribute("textContent")
• Replace every time.sleep() with WebDriverWait + expected conditions, and never mix implicit and explicit waits
• --proxy-server doesn't accept credentials; authenticated proxies need a CDP handler or a proxy extension
• Selenium can't set request headers natively — that's a CDP (execute_cdp_cmd) job, and Chrome-only
• Real browsers don't defeat modern anti-bot systems: undetected-chromedriver helps on soft checks, but Cloudflare and DataDome still detect automated Chrome

Why Selenium — and when to skip it

Selenium's advantage is fidelity and reach. It runs a genuine browser, so client-side rendering, lazy loading, and interaction-gated content all work; and it supports more browsers and languages than any alternative, including real Safari and Internet Explorer mode. It's also everywhere: a decade of Stack Overflow answers, corporate test grids, and CI pipelines already speak Selenium.

The cost is speed and weight. Every page means a full browser rendering pipeline — 100–400 MB of RAM and a second or more of CPU-bound work per page, against a few milliseconds for an HTTP request. If the data you need is already in the HTML response, Requests plus Beautiful Soup is 10–50× cheaper, and you should check that first: open the target page with JavaScript disabled, or curl it and search the raw HTML for a value you need. If it's there, you don't need a browser. Our headless browser guide covers that decision in depth, and the Python scraping libraries roundup maps the rest of the toolbox.

For a brand-new browser-automation project in 2026, Playwright is usually the better default — the comparison section below is honest about why. Selenium remains the right call when you're maintaining existing Selenium code, need a browser or language Playwright doesn't cover, or already run a Selenium Grid.

Installation and setup

pip install selenium

That's it. Since version 4.6, Selenium ships Selenium Manager, which detects your installed browser, downloads the matching driver, and caches it — all automatically on first run. The webdriver-manager third-party package and hand-downloaded chromedriver binaries that older tutorials insist on are no longer needed, and pinning a stale driver is now a common source of SessionNotCreatedException.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")          # modern headless mode
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome(options=options)      # driver resolved automatically
driver.get("https://example.com")

print(driver.title)
print(driver.page_source[:500])                 # rendered HTML

driver.quit()                                    # always quit, not close

driver.quit() ends the session and kills the browser process; driver.close() only closes the current window. Leaking browser processes is the classic Selenium memory bug — use a try/finally or a context manager so quit() runs even when scraping raises.

Firefox is a one-word change (webdriver.Firefox with selenium.webdriver.firefox.options.Options), and the rest of this guide applies unchanged except where noted. Selenium 4.46 is current at the time of writing; anything 4.6+ has the behavior described here.

Finding elements

All location goes through the By class:

from selenium.webdriver.common.by import By

driver.find_element(By.ID, "search")
driver.find_element(By.CSS_SELECTOR, "div.product > h2.title")
driver.find_element(By.XPATH, "//div[@class='price']")
driver.find_element(By.CLASS_NAME, "product-item")
driver.find_element(By.NAME, "email")
driver.find_element(By.TAG_NAME, "h1")
driver.find_element(By.LINK_TEXT, "Next page")
driver.find_element(By.PARTIAL_LINK_TEXT, "Next")

The find_element_by_id()-style shortcuts were removed in Selenium 4.3 — if you're porting old code, that's the rename you'll be doing most.

find_element vs find_elements

The difference matters more than the plural suggests:

That makes find_elements the idiomatic existence check, because it never throws:

# Presence check without try/except
if driver.find_elements(By.CSS_SELECTOR, ".cookie-banner"):
    driver.find_element(By.CSS_SELECTOR, ".cookie-banner button.accept").click()

# Iterate results
for card in driver.find_elements(By.CSS_SELECTOR, ".product"):
    print(card.find_element(By.CSS_SELECTOR, ".name").text)

Searches are scoped: calling find_element on an element searches only its subtree, which is how you avoid brittle absolute selectors. Prefer stable hooks (IDs, data-* attributes) over generated class names, and prefer CSS to XPath unless you need XPath's axes or text matching — see our CSS selectors FAQ and XPath cheat sheet.

Getting text and attributes

Extracting text is where Selenium surprises people most:

el = driver.find_element(By.CSS_SELECTOR, ".description")

el.text                              # visible, rendered text only
el.get_attribute("textContent")      # all text, including hidden nodes
el.get_attribute("innerHTML")        # markup inside the element
el.get_attribute("href")             # any HTML attribute or DOM property
el.get_attribute("value")            # current value of a form input

element.text returns what a user could see: it respects CSS, so text inside display: none, collapsed accordions, or off-screen containers comes back as an empty string, and whitespace is normalized the way the browser renders it. This is the single most common "Selenium returns nothing but the element exists" bug. When you need the underlying text regardless of visibility, use get_attribute("textContent").

get_attribute also blurs attributes and properties — it returns the live DOM property when one exists, which is why it gives you the current value of an input rather than the original HTML attribute. For the raw HTML attribute specifically, Selenium 4 added get_dom_attribute().

For bulk extraction, pulling many values in one execute_script call is dramatically faster than looping over elements, because each Selenium call is a round trip to the browser:

products = driver.execute_script("""
    return [...document.querySelectorAll('.product')].map(el => ({
        name: el.querySelector('.name')?.textContent.trim(),
        price: el.querySelector('.price')?.textContent.trim(),
        url: el.querySelector('a')?.href,
    }));
""")

A common hybrid is to let Selenium render and then hand driver.page_source to Beautiful Soup for parsing — you get the browser's rendering with a far more pleasant extraction API.

Waiting: the part that causes flaky scrapers

Most Selenium bugs are timing bugs. driver.get() returns when the load event fires, which says nothing about whether the JavaScript that builds your content has run. The fix is never time.sleep() — it's waiting on a condition:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".results")))
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".price")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more")))
wait.until(EC.text_to_be_present_in_element((By.CSS_SELECTOR, ".status"), "Ready"))
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, ".spinner")))

The three conditions worth distinguishing: presence means in the DOM (possibly invisible), visibility means rendered with non-zero size, clickable means visible and enabled. Waiting for presence and then clicking is a frequent cause of ElementNotInteractableException.

For anything the built-in conditions don't express, wait on a JavaScript predicate:

wait.until(lambda d: d.execute_script(
    "return document.querySelectorAll('.product').length >= 20"))

wait.until(lambda d: d.execute_script("return document.readyState") == "complete")

Don't mix implicit and explicit waits. driver.implicitly_wait(10) tells the driver to poll for every element lookup, and combining it with WebDriverWait produces unpredictable, sometimes multiplied timeouts — a documented Selenium pitfall. Pick explicit waits and leave the implicit wait at zero.

StaleElementReferenceException

An element reference points at a specific DOM node. If the page re-renders — a framework re-draw, an AJAX refresh, a navigation — your reference dies, even though a visually identical element is now on screen. Re-find the element instead of caching it:

from selenium.common.exceptions import StaleElementReferenceException

def click_safely(driver, selector, attempts=3):
    for _ in range(attempts):
        try:
            driver.find_element(By.CSS_SELECTOR, selector).click()
            return
        except StaleElementReferenceException:
            continue
    raise RuntimeError(f"element {selector} kept going stale")

The deeper fix is structural: collect the data you need from each element immediately rather than building a list of element handles and revisiting them after the page has changed.

Clicking, typing, and forms

driver.find_element(By.CSS_SELECTOR, "button.submit").click()

field = driver.find_element(By.NAME, "q")
field.clear()
field.send_keys("wireless headphones")

from selenium.webdriver.common.keys import Keys
field.send_keys(Keys.ENTER)

from selenium.webdriver.support.ui import Select
Select(driver.find_element(By.ID, "sort")).select_by_value("price_asc")

When a normal click fails because an overlay intercepts it (ElementClickInterceptedException), the options in order of preference are: dismiss the overlay, scroll the element into view, or fall back to a JavaScript click.

el = driver.find_element(By.CSS_SELECTOR, ".buy-now")
driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", el)
driver.execute_script("arguments[0].click();", el)   # bypasses overlays

A JavaScript click ignores whether a real user could have clicked, so it will happily "click" hidden elements — useful as an escape hatch, misleading as a default.

For hovers, drags, and key combinations, use ActionChains:

from selenium.webdriver.common.action_chains import ActionChains

menu = driver.find_element(By.CSS_SELECTOR, ".menu")
ActionChains(driver).move_to_element(menu).pause(0.3).click(
    driver.find_element(By.CSS_SELECTOR, ".menu .submenu-item")
).perform()

ActionChains(driver).key_down(Keys.CONTROL).send_keys("a").key_up(Keys.CONTROL).perform()

Dynamic content and infinite scroll

For endlessly scrolling listings, scroll and wait for the item count to grow, with a stop condition when it doesn't:

def scroll_until_stable(driver, item_selector, max_rounds=20, pause=1.5):
    count = len(driver.find_elements(By.CSS_SELECTOR, item_selector))
    for _ in range(max_rounds):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        try:
            WebDriverWait(driver, pause * 4).until(
                lambda d: len(d.find_elements(By.CSS_SELECTOR, item_selector)) > count)
        except TimeoutException:
            break                       # nothing new loaded — we're done
        count = len(driver.find_elements(By.CSS_SELECTOR, item_selector))
    return count

Before writing scroll loops, open the Network tab and look for the XHR the page uses to fetch each batch. If it's a plain JSON endpoint, calling it directly with requests is faster, more reliable, and easier to paginate than driving a browser — and you can keep Selenium only for obtaining the session cookies it needs. Infinite-scroll listings are the standard shape of job board and property listing scrapes, and both are far cheaper against the JSON than the DOM.

Windows, tabs, and iframes

Selenium's focus is explicit: it only sees the window and frame it's currently switched to.

original = driver.current_window_handle

driver.switch_to.new_window("tab")            # Selenium 4
driver.get("https://example.com/other")

for handle in driver.window_handles:          # switch to a popup
    if handle != original:
        driver.switch_to.window(handle)
        break

driver.close()                                 # close the popup
driver.switch_to.window(original)              # focus must be restored

Content inside an <iframe> is invisible to selectors until you switch into it — the reason so many "element not found" reports involve embedded players, checkout widgets, and consent dialogs:

driver.switch_to.frame(driver.find_element(By.CSS_SELECTOR, "iframe#checkout"))
driver.find_element(By.ID, "card-number").send_keys("4242424242424242")
driver.switch_to.default_content()             # back to the top-level document

Browser profiles and persistent sessions

A Chrome profile directory holds cookies, local storage, saved logins, extensions, and preferences. Pointing Selenium at a persistent profile means logging in once and reusing that session on every later run:

options = Options()
options.add_argument("--user-data-dir=/home/scraper/chrome-profiles/session-1")
options.add_argument("--profile-directory=Default")
driver = webdriver.Chrome(options=options)

Three practical rules. Never point at the profile of a Chrome you use interactively — Chrome locks the directory, and you'll get SessionNotCreatedException (or corrupt your real profile). Copy it instead. One profile per concurrent browser, since two drivers sharing a directory conflict. And profiles grow: cache and history accumulate across runs, so prune or recreate them periodically.

Preferences that don't have command-line flags go through prefs:

options.add_experimental_option("prefs", {
    "profile.managed_default_content_settings.images": 2,      # block images
    "profile.default_content_setting_values.notifications": 2,  # block popups
    "credentials_enable_service": False,                        # no password prompts
})

Blocking images alone typically cuts page weight enough to noticeably speed up large crawls.

Authentication and cookies

For form logins, drive the form once and save the cookies:

import json, os

driver.get("https://example.com/login")
driver.find_element(By.ID, "email").send_keys(os.environ["SCRAPE_USER"])
driver.find_element(By.ID, "password").send_keys(os.environ["SCRAPE_PASS"])
driver.find_element(By.CSS_SELECTOR, "button[type=submit]").click()
WebDriverWait(driver, 15).until(EC.url_contains("/dashboard"))

with open("cookies.json", "w") as f:
    json.dump(driver.get_cookies(), f)

# Later run — restore instead of logging in again
driver.get("https://example.com")            # must be on the domain first
for cookie in json.load(open("cookies.json")):
    cookie.pop("sameSite", None)             # some drivers reject this key
    driver.add_cookie(cookie)
driver.get("https://example.com/dashboard")

You must load a page on the target domain before add_cookie — cookies can't be set for a domain the browser isn't currently on.

HTTP Basic auth dialogs are native browser UI, not JavaScript alerts, so switch_to.alert won't help. Embedding credentials in the URL (https://user:pass@host/) is deprecated and blocked in modern Chrome. The reliable approach is an Authorization header via CDP:

import base64
token = base64.b64encode(b"user:pass").decode()
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders",
                       {"headers": {"Authorization": f"Basic {token}"}})

Keep credentials in environment variables or a secrets manager, and only automate logins on accounts you're authorized to use — scraping behind a login is where terms-of-service and legal exposure concentrate. See our guide to web scraping legality.

Custom headers and user agents

Selenium has no API for request headers. This is a deliberate consequence of the WebDriver protocol, which automates the browser rather than the network stack. The user agent is the exception, because it's a browser-level setting:

options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                     "AppleWebKit/537.36 (KHTML, like Gecko) "
                     "Chrome/126.0.0.0 Safari/537.36")

For arbitrary headers, use the Chrome DevTools Protocol:

driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", {"headers": {
    "Accept-Language": "en-US,en;q=0.9",
    "X-API-Key": os.environ["API_KEY"],
}})

Two caveats: execute_cdp_cmd is Chrome/Edge only — Firefox has no equivalent — and headers set this way apply to every request the page makes, including subresources. The once-standard selenium-wire package offered a nicer interface but is no longer actively maintained, so CDP is the safer choice for new code.

Keep your headers internally consistent: a Windows Chrome user agent paired with a Linux platform fingerprint and a mismatched Accept-Language is exactly the incoherence anti-bot systems look for. Our user agent rotation guide covers doing this well.

Proxies

A basic proxy is a launch flag:

options.add_argument("--proxy-server=http://proxy.example.com:8080")
options.add_argument("--proxy-server=socks5://proxy.example.com:1080")

The complication is authentication: --proxy-server accepts no credentials, and http://user:pass@host:port is ignored by Chrome, which instead pops a native auth dialog that blocks your scrape. There are two workable answers.

Handle the challenge over CDP — no extra files, Chrome only:

driver.execute_cdp_cmd("Fetch.enable", {"handleAuthRequests": True})
# then answer Fetch.authRequired events with Fetch.continueWithAuth

Or ship a tiny proxy extension, which is what most production Selenium setups do — a two-file .zip with a background.js that calls chrome.webRequest.onAuthRequired and returns the credentials, loaded via options.add_extension("proxy_auth.zip"). It's more setup, but it survives across pages and works with Selenium Grid.

Rotation happens per browser session, not per request: a launched Chrome keeps its --proxy-server for its lifetime, so rotating IPs means starting a new driver with a new proxy (or pointing every session at a rotating gateway endpoint that changes the exit IP for you). Our proxy provider comparison covers choosing between datacenter and residential pools.

Note that a proxy fixes IP reputation only. It does nothing about the browser fingerprint that marks your Chrome as automated — see the bot detection section.

File downloads

Downloads need an explicit directory, set through preferences:

download_dir = os.path.abspath("downloads")
os.makedirs(download_dir, exist_ok=True)

options.add_experimental_option("prefs", {
    "download.default_directory": download_dir,
    "download.prompt_for_download": False,
    "plugins.always_open_pdf_externally": True,    # download PDFs, don't preview
})

Selenium fires no download-complete event, so wait for the file — specifically, for Chrome's .crdownload temp file to disappear:

import time, glob

def wait_for_download(directory, timeout=60):
    deadline = time.time() + timeout
    while time.time() < deadline:
        if not glob.glob(os.path.join(directory, "*.crdownload")):
            files = [f for f in glob.glob(os.path.join(directory, "*"))]
            if files:
                return max(files, key=os.path.getctime)
        time.sleep(0.5)
    raise TimeoutError("download did not finish")

When the file is at a plain URL, skip the browser: copy the session cookies into requests and download it directly. It's faster, streams to disk, and gives you real error handling.

import requests

session = requests.Session()
for c in driver.get_cookies():
    session.cookies.set(c["name"], c["value"], domain=c["domain"])
with session.get(file_url, stream=True) as r:
    r.raise_for_status()
    with open("report.csv", "wb") as f:
        for chunk in r.iter_content(8192):
            f.write(chunk)

Screenshots

driver.save_screenshot("page.png")                      # viewport only
driver.find_element(By.CSS_SELECTOR, ".chart").screenshot("chart.png")
png_bytes = driver.get_screenshot_as_png()              # for in-memory use

save_screenshot captures the viewport, not the full page — a persistent difference from Puppeteer and Playwright, which offer full-page capture directly. Two workarounds: set a tall window size before capturing, or use CDP:

result = driver.execute_cdp_cmd("Page.captureScreenshot", {
    "captureBeyondViewport": True, "fromSurface": True})
with open("fullpage.png", "wb") as f:
    f.write(base64.b64decode(result["data"]))

Screenshots are also the single best debugging tool for headless scrapers: capture one in your exception handler and you'll usually see the cookie wall, CAPTCHA, or empty state that your selector hit.

SSL certificate errors

Scraping targets with self-signed or expired certificates fails at the browser level. Selenium 4 exposes this as a capability on the options object:

options.set_capability("acceptInsecureCerts", True)

(desired_capabilities, which older tutorials pass to the constructor, was removed in Selenium 4.10 — configure everything through options.) Chrome also accepts --ignore-certificate-errors, though the capability is the portable form. Only disable certificate validation for targets you control or trust; on the public internet it removes your protection against interception.

Browser extensions

options.add_extension("/path/to/extension.crx")            # packed
options.add_argument("--load-extension=/path/to/unpacked")  # unpacked

Extensions are how ad blockers, proxy authenticators, and fingerprint tweaks get injected. Two constraints: extensions don't load in old headless mode (another reason to use --headless=new), and each one adds startup time and memory to every browser you launch.

Handling exceptions

Selenium's exception names are precise, and knowing which is which shortens debugging enormously:

A resilient scrape wraps per-page work so one bad page can't kill the crawl:

from selenium.common.exceptions import TimeoutException, WebDriverException

def scrape_page(driver, url, retries=2):
    for attempt in range(retries + 1):
        try:
            driver.get(url)
            WebDriverWait(driver, 15).until(
                EC.presence_of_element_located((By.CSS_SELECTOR, ".content")))
            return driver.find_element(By.CSS_SELECTOR, ".content").text
        except TimeoutException:
            if attempt == retries:
                driver.save_screenshot(f"timeout-{attempt}.png")
                raise
        except WebDriverException:
            raise                       # session-level: let the caller restart

Set page-level timeouts too, or a single hanging resource can stall a worker indefinitely: driver.set_page_load_timeout(30) and driver.set_script_timeout(20).

Headless mode and performance

options.add_argument("--headless=new")     # Chrome's modern headless
options.add_argument("--no-sandbox")                # containers only
options.add_argument("--disable-dev-shm-usage")     # avoid /dev/shm crashes
options.add_argument("--blink-settings=imagesEnabled=false")
options.page_load_strategy = "eager"       # don't wait for images/subresources

page_load_strategy = "eager" returns control at DOMContentLoaded rather than the full load event and is often the single biggest speed win for scraping. "none" returns immediately and leaves all waiting to you.

The costs that actually matter at scale: reuse one driver for many pages rather than launching per page (browser startup dominates otherwise), restart the driver every few hundred pages to keep memory flat, and block images and fonts. And always quit() in a finally — orphaned Chrome processes are what turns a long crawl into an out-of-memory incident.

Parallel scraping and Selenium Grid

Selenium's WebDriver objects aren't thread-safe to share, but running one driver per worker is straightforward. Processes beat threads here because the work is browser-bound:

from concurrent.futures import ProcessPoolExecutor

def scrape_one(url):
    driver = webdriver.Chrome(options=make_options())
    try:
        driver.get(url)
        return driver.find_element(By.TAG_NAME, "h1").text
    finally:
        driver.quit()

with ProcessPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(scrape_one, urls))

Size max_workers by RAM, not CPU count: budget several hundred MB per concurrent browser and leave headroom.

Beyond one machine, Selenium Grid distributes sessions across nodes. The quickest start is the official container, then point a Remote driver at it:

docker run -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome:latest

driver = webdriver.Remote(
    command_executor="http://localhost:4444/wd/hub",
    options=Options())

--shm-size=2g is not optional in Docker — the default 64 MB /dev/shm causes Chrome to crash under load. The same image is what most CI pipelines use as a service container, which makes Grid the natural path when your scraper needs to run in CI alongside tests.

Bot detection: what works and what doesn't

Automated Chrome is detectable, and honesty here saves a lot of wasted effort. Default Selenium sets navigator.webdriver = true and carries other automation tells; the usual first mitigations are:

options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)

undetected-chromedriver (pip install undetected-chromedriver, then uc.Chrome()) patches the driver binary and Chrome's startup to hide more of these signals, and it does get past the softer checks that stop vanilla Selenium.

What none of this reliably beats: Cloudflare's managed challenges, DataDome, PerimeterX, and similar commercial systems. They combine TLS fingerprinting, canvas and WebGL fingerprints, mouse-movement behavior, and IP reputation — headless Chrome loses on several of those regardless of which flags you set, and every Chrome release resets the arms race. If your target sits behind one of these, the realistic options are a well-maintained residential proxy pool plus significant fingerprinting work, or an API that owns that problem for you.

Practical hygiene matters more than tricks: respect robots.txt, rate-limit yourself, randomize timing, and don't hammer a site in parallel from one IP.

Selenium vs Playwright vs Puppeteer

The honest read for 2026: for a new scraping project, Playwright is the better default — auto-waiting removes the largest category of Selenium bugs, and it's meaningfully faster per page. Selenium wins on reach (real Safari, more languages, the W3C standard, the biggest grid and cloud ecosystem) and on inertia: it powers an enormous amount of working code, and rewriting a functioning scraper is rarely the highest-value change. Puppeteer is the choice when you're Node-only and Chrome-focused.

On bot detection, all three are roughly equivalent — none of them wins that fight on its own.

When Selenium isn't enough

Selenium solves rendering and interaction. It doesn't solve blocked IPs, anti-bot challenges, CAPTCHAs, or the operational cost of running a browser fleet. When those become the bottleneck, you can keep your parsing code and let an API handle the browser — WebScraping.AI renders pages in real Chrome behind rotating datacenter or residential proxies and returns the result over plain HTTP:

import requests

# Fully rendered HTML through a managed browser + residential proxies
html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "js": "true",
    "proxy": "residential",
}).text

# Or skip parsing entirely — structured fields from any page
fields = requests.get("https://api.webscraping.ai/ai/fields", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "fields[name]": "Product name",
    "fields[price]": "Price with currency",
}).json()

A common migration path: prototype locally with Selenium, then replace driver.get() + driver.page_source with the /html endpoint in production and keep your Beautiful Soup parsing unchanged. See the AI web scraping overview for the extraction endpoints, or the n8n node and MCP server if the scrape is feeding a workflow or an agent rather than a Python script.

Frequently asked questions

Do I still need to download ChromeDriver? No. Selenium Manager has been built in since Selenium 4.6 and resolves the right driver for your installed browser automatically. Delete webdriver-manager from your requirements and stop pinning driver binaries — a stale pinned driver is now a more common failure than a missing one.

Why does element.text return an empty string? Because text returns only visible text. If the element is hidden, collapsed, or scrolled out of a container with overflow: hidden, you get "" even though the element exists. Use get_attribute("textContent") for the underlying text regardless of visibility.

What's the difference between find_element and find_elements? find_element returns the first match and raises NoSuchElementException when there is none; find_elements returns a list and returns [] instead of raising. Use the plural form for existence checks — it's cleaner than wrapping the singular in try/except.

How do I use an authenticated proxy with Selenium? Not through --proxy-server, which ignores credentials. Either answer Chrome's auth challenge over CDP (Fetch.enable with handleAuthRequests) or load a small proxy-auth extension via options.add_extension(). The extension approach is what most production setups use because it also works with Selenium Grid.

Can Selenium set custom HTTP headers? Not natively — the WebDriver protocol doesn't expose the network layer. The user agent has a launch flag; everything else needs driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", ...), which is Chrome and Edge only.

Can Selenium bypass Cloudflare or CAPTCHAs? Not reliably. undetected-chromedriver and the anti-automation flags handle soft checks, but commercial anti-bot systems fingerprint TLS, canvas, WebGL, and behavior, and detect automated Chrome regardless of language. For protected targets, a scraping API that owns the challenge layer — like WebScraping.AI — is usually cheaper than maintaining an evasion stack.

Is Selenium or Playwright better for web scraping? Playwright for new projects: auto-waiting eliminates the most common source of flaky scrapers, and it's faster per page. Selenium for existing code, unusual browsers like real Safari, languages Playwright doesn't support, or an established Grid. Neither has an advantage at avoiding bot detection.

How many browsers can I run in parallel? Budget 100–400 MB of RAM per concurrent Chrome and size by memory rather than CPU cores — four to eight workers on a typical 8–16 GB machine. Use one driver per process, quit() in a finally, and restart drivers every few hundred pages to keep memory flat.

Is scraping with Selenium legal? Automating a browser is legal in itself; what matters is what you access and how — public versus gated data, terms of service, rate pressure, and privacy law. Our web scraping legality guide covers the current landscape.