---
title: "Selenium Web Scraping: The Complete Python Guide"
description: "How to scrape websites with Selenium in Python: setup, selectors, waits, proxies, profiles, downloads, screenshots, exceptions, Grid, and bot detection."
url: https://webscraping.ai/blog/python-selenium
markdown_index: https://webscraping.ai/llms.txt
---
# Selenium Web Scraping: The Complete Python Guide

Selenium is the oldest and most widely deployed browser automation framework: it drives a real Chrome, Firefox, Edge, or Safari through the W3C WebDriver protocol, so your scraper sees exactly what a user sees — JavaScript executed, AJAX applied, DOM fully built. This guide covers the complete scraping workflow in Python: setup with Selenium Manager, finding elements, waiting correctly, proxies, browser profiles, downloads, screenshots, exception handling, parallel execution with Grid, an honest look at bot detection, and how Selenium compares to Playwright and Puppeteer in 2026.

## Key Takeaways

- `pip install selenium` is the whole install — **Selenium Manager** (built in since 4.6) downloads matching drivers automatically, so `webdriver-manager` and manual chromedriver downloads are obsolete
- Use `find_element(By.CSS_SELECTOR, ...)`; the old `find_element_by_*` methods were removed in Selenium 4.3
- `element.text` returns only _visible_ text — for hidden or collapsed elements use `get_attribute("textContent")`
- Replace every `time.sleep()` with `WebDriverWait` + expected conditions, and never mix implicit and explicit waits
- `--proxy-server` doesn't accept credentials; authenticated proxies need a CDP handler or a proxy extension
- Selenium can't set request headers natively — that's a CDP (`execute_cdp_cmd`) job, and Chrome-only
- Real browsers don't defeat modern anti-bot systems: `undetected-chromedriver` helps on soft checks, but Cloudflare and DataDome still detect automated Chrome

## Why Selenium — and when to skip it

Selenium's advantage is fidelity and reach. It runs a genuine browser, so client-side rendering, lazy loading, and interaction-gated content all work; and it supports more browsers and languages than any alternative, including real Safari and Internet Explorer mode. It's also everywhere: a decade of Stack Overflow answers, corporate test grids, and CI pipelines already speak Selenium.

The cost is speed and weight. Every page means a full browser rendering pipeline — 100–400 MB of RAM and a second or more of CPU-bound work per page, against a few milliseconds for an HTTP request. If the data you need is already in the HTML response, [Requests plus Beautiful Soup](https://webscraping.ai/blog/beautiful-soup-python) is 10–50× cheaper, and you should check that first: open the target page with JavaScript disabled, or `curl` it and search the raw HTML for a value you need. If it's there, you don't need a browser. Our [headless browser guide](https://webscraping.ai/blog/headless-browser-guide) covers that decision in depth, and the [Python scraping libraries roundup](https://webscraping.ai/blog/python-web-scraping-libraries) maps the rest of the toolbox.

For a brand-new browser-automation project in 2026, [Playwright](https://webscraping.ai/blog/playwright-web-scraping) is usually the better default — the comparison section below is honest about why. Selenium remains the right call when you're maintaining existing Selenium code, need a browser or language Playwright doesn't cover, or already run a Selenium Grid.

## Installation and setup

```bash
pip install selenium
```

That's it. Since version 4.6, Selenium ships **Selenium Manager** , which detects your installed browser, downloads the matching driver, and caches it — all automatically on first run. The `webdriver-manager` third-party package and hand-downloaded chromedriver binaries that older tutorials insist on are no longer needed, and pinning a stale driver is now a common source of `SessionNotCreatedException`.

```python
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new") # modern headless mode
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome(options=options) # driver resolved automatically
driver.get("https://example.com")

print(driver.title)
print(driver.page_source[:500]) # rendered HTML

driver.quit() # always quit, not close
```

`driver.quit()` ends the session and kills the browser process; `driver.close()` only closes the current window. Leaking browser processes is the classic Selenium memory bug — use a `try/finally` or a context manager so `quit()` runs even when scraping raises.

Firefox is a one-word change (`webdriver.Firefox` with `selenium.webdriver.firefox.options.Options`), and the rest of this guide applies unchanged except where noted. Selenium 4.49 is current at the time of writing (September 2026); anything 4.6+ has the behavior described here.

## Finding elements

All location goes through the `By` class:

```python
from selenium.webdriver.common.by import By

driver.find_element(By.ID, "search")
driver.find_element(By.CSS_SELECTOR, "div.product > h2.title")
driver.find_element(By.XPATH, "//div[@class='price']")
driver.find_element(By.CLASS_NAME, "product-item")
driver.find_element(By.NAME, "email")
driver.find_element(By.TAG_NAME, "h1")
driver.find_element(By.LINK_TEXT, "Next page")
driver.find_element(By.PARTIAL_LINK_TEXT, "Next")
```

The `find_element_by_id()`-style shortcuts were removed in Selenium 4.3 — if you're porting old code, that's the rename you'll be doing most.

### find\_element vs find\_elements

The difference matters more than the plural suggests:

| | Returns | When nothing matches |
| --- | --- | --- |
| `find_element` | the **first** matching element | raises `NoSuchElementException` |
| `find_elements` | a **list** of all matches | returns `[]` — no exception |

That makes `find_elements` the idiomatic existence check, because it never throws:

```python
# Presence check without try/except
if driver.find_elements(By.CSS_SELECTOR, ".cookie-banner"):
    driver.find_element(By.CSS_SELECTOR, ".cookie-banner button.accept").click()

# Iterate results
for card in driver.find_elements(By.CSS_SELECTOR, ".product"):
    print(card.find_element(By.CSS_SELECTOR, ".name").text)
```

Searches are scoped: calling `find_element` on an element searches only its subtree, which is how you avoid brittle absolute selectors. Prefer stable hooks (IDs, `data-*` attributes) over generated class names, and prefer CSS to XPath unless you need XPath's axes or text matching — see our [CSS selectors cheat sheet](https://webscraping.ai/blog/css-selectors-cheat-sheet) and [XPath cheat sheet](https://webscraping.ai/blog/xpath-cheat-sheet).

## Getting text and attributes

Extracting text is where Selenium surprises people most:

```python
el = driver.find_element(By.CSS_SELECTOR, ".description")

el.text # visible, rendered text only
el.get_attribute("textContent") # all text, including hidden nodes
el.get_attribute("innerHTML") # markup inside the element
el.get_attribute("href") # any HTML attribute or DOM property
el.get_attribute("value") # current value of a form input
```

`element.text` returns what a user could _see_: it respects CSS, so text inside `display: none`, collapsed accordions, or off-screen containers comes back as an empty string, and whitespace is normalized the way the browser renders it. This is the single most common "Selenium returns nothing but the element exists" bug. When you need the underlying text regardless of visibility, use `get_attribute("textContent")`.

`get_attribute` also blurs attributes and properties — it returns the live DOM property when one exists, which is why it gives you the _current_ value of an input rather than the original HTML attribute. For the raw HTML attribute specifically, Selenium 4 added `get_dom_attribute()`.

For bulk extraction, pulling many values in one `execute_script` call is dramatically faster than looping over elements, because each Selenium call is a round trip to the browser:

```python
products = driver.execute_script("""
    return [...document.querySelectorAll('.product')].map(el => ({
        name: el.querySelector('.name')?.textContent.trim(),
        price: el.querySelector('.price')?.textContent.trim(),
        url: el.querySelector('a')?.href,
    }));
""")
```

A common hybrid is to let Selenium render and then hand `driver.page_source` to Beautiful Soup for parsing — you get the browser's rendering with a far more pleasant extraction API.

## Waiting: the part that causes flaky scrapers

Most Selenium bugs are timing bugs. `driver.get()` returns when the load event fires, which says nothing about whether the JavaScript that builds your content has run. The fix is never `time.sleep()` — it's waiting on a condition:

```python
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".results")))
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".price")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more")))
wait.until(EC.text_to_be_present_in_element((By.CSS_SELECTOR, ".status"), "Ready"))
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, ".spinner")))
```

The three conditions worth distinguishing: **presence** means in the DOM (possibly invisible), **visibility** means rendered with non-zero size, **clickable** means visible and enabled. Waiting for presence and then clicking is a frequent cause of `ElementNotInteractableException`.

For anything the built-in conditions don't express, wait on a JavaScript predicate:

```python
wait.until(lambda d: d.execute_script(
    "return document.querySelectorAll('.product').length >= 20"))

wait.until(lambda d: d.execute_script("return document.readyState") == "complete")
```

**Don't mix implicit and explicit waits.** `driver.implicitly_wait(10)` tells the driver to poll for _every_ element lookup, and combining it with `WebDriverWait` produces unpredictable, sometimes multiplied timeouts — a documented Selenium pitfall. Pick explicit waits and leave the implicit wait at zero.

### StaleElementReferenceException

An element reference points at a specific DOM node. If the page re-renders — a framework re-draw, an AJAX refresh, a navigation — your reference dies, even though a visually identical element is now on screen. Re-find the element instead of caching it:

```python
from selenium.common.exceptions import StaleElementReferenceException

def click_safely(driver, selector, attempts=3):
    for _ in range(attempts):
        try:
            driver.find_element(By.CSS_SELECTOR, selector).click()
            return
        except StaleElementReferenceException:
            continue
    raise RuntimeError(f"element {selector} kept going stale")
```

The deeper fix is structural: collect the data you need from each element immediately rather than building a list of element handles and revisiting them after the page has changed.

## Clicking, typing, and forms

```python
driver.find_element(By.CSS_SELECTOR, "button.submit").click()

field = driver.find_element(By.NAME, "q")
field.clear()
field.send_keys("wireless headphones")

from selenium.webdriver.common.keys import Keys
field.send_keys(Keys.ENTER)

from selenium.webdriver.support.ui import Select
Select(driver.find_element(By.ID, "sort")).select_by_value("price_asc")
```

When a normal click fails because an overlay intercepts it (`ElementClickInterceptedException`), the options in order of preference are: dismiss the overlay, scroll the element into view, or fall back to a JavaScript click.

```python
el = driver.find_element(By.CSS_SELECTOR, ".buy-now")
driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", el)
driver.execute_script("arguments[0].click();", el) # bypasses overlays
```

A JavaScript click ignores whether a real user _could_ have clicked, so it will happily "click" hidden elements — useful as an escape hatch, misleading as a default.

For hovers, drags, and key combinations, use `ActionChains`:

```python
from selenium.webdriver.common.action_chains import ActionChains

menu = driver.find_element(By.CSS_SELECTOR, ".menu")
ActionChains(driver).move_to_element(menu).pause(0.3).click(
    driver.find_element(By.CSS_SELECTOR, ".menu .submenu-item")
).perform()

ActionChains(driver).key_down(Keys.CONTROL).send_keys("a").key_up(Keys.CONTROL).perform()
```

## Dynamic content and infinite scroll

For endlessly scrolling listings, scroll and wait for the item count to grow, with a stop condition when it doesn't:

```python
def scroll_until_stable(driver, item_selector, max_rounds=20, pause=1.5):
    count = len(driver.find_elements(By.CSS_SELECTOR, item_selector))
    for _ in range(max_rounds):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        try:
            WebDriverWait(driver, pause * 4).until(
                lambda d: len(d.find_elements(By.CSS_SELECTOR, item_selector)) > count)
        except TimeoutException:
            break # nothing new loaded — we're done
        count = len(driver.find_elements(By.CSS_SELECTOR, item_selector))
    return count
```

Before writing scroll loops, open the Network tab and look for the XHR the page uses to fetch each batch. If it's a plain JSON endpoint, calling it directly with `requests` is faster, more reliable, and easier to paginate than driving a browser — and you can keep Selenium only for obtaining the session cookies it needs. Infinite-scroll listings are the standard shape of [job board](https://webscraping.ai/use-cases/job-listing-aggregation) and [property listing](https://webscraping.ai/use-cases/property-listing-aggregation) scrapes, and both are far cheaper against the JSON than the DOM.

## Windows, tabs, and iframes

Selenium's focus is explicit: it only sees the window and frame it's currently switched to.

```python
original = driver.current_window_handle

driver.switch_to.new_window("tab") # Selenium 4
driver.get("https://example.com/other")

for handle in driver.window_handles: # switch to a popup
    if handle != original:
        driver.switch_to.window(handle)
        break

driver.close() # close the popup
driver.switch_to.window(original) # focus must be restored
```

Content inside an `<iframe>` is invisible to selectors until you switch into it — the reason so many "element not found" reports involve embedded players, checkout widgets, and consent dialogs:

```python
driver.switch_to.frame(driver.find_element(By.CSS_SELECTOR, "iframe#checkout"))
driver.find_element(By.ID, "card-number").send_keys("4242424242424242")
driver.switch_to.default_content() # back to the top-level document
```

## Browser profiles and persistent sessions

A Chrome profile directory holds cookies, local storage, saved logins, extensions, and preferences. Pointing Selenium at a persistent profile means logging in once and reusing that session on every later run:

```python
options = Options()
options.add_argument("--user-data-dir=/home/scraper/chrome-profiles/session-1")
options.add_argument("--profile-directory=Default")
driver = webdriver.Chrome(options=options)
```

Three practical rules. **Never point at the profile of a Chrome you use interactively** — Chrome locks the directory, and you'll get `SessionNotCreatedException` (or corrupt your real profile). Copy it instead. **One profile per concurrent browser** , since two drivers sharing a directory conflict. And **profiles grow** : cache and history accumulate across runs, so prune or recreate them periodically.

Preferences that don't have command-line flags go through `prefs`:

```python
options.add_experimental_option("prefs", {
    "profile.managed_default_content_settings.images": 2, # block images
    "profile.default_content_setting_values.notifications": 2, # block popups
    "credentials_enable_service": False, # no password prompts
})
```

Blocking images alone typically cuts page weight enough to noticeably speed up large crawls.

## Authentication and cookies

For form logins, drive the form once and save the cookies:

```python
import json, os

driver.get("https://example.com/login")
driver.find_element(By.ID, "email").send_keys(os.environ["SCRAPE_USER"])
driver.find_element(By.ID, "password").send_keys(os.environ["SCRAPE_PASS"])
driver.find_element(By.CSS_SELECTOR, "button[type=submit]").click()
WebDriverWait(driver, 15).until(EC.url_contains("/dashboard"))

with open("cookies.json", "w") as f:
    json.dump(driver.get_cookies(), f)

# Later run — restore instead of logging in again
driver.get("https://example.com") # must be on the domain first
for cookie in json.load(open("cookies.json")):
    cookie.pop("sameSite", None) # some drivers reject this key
    driver.add_cookie(cookie)
driver.get("https://example.com/dashboard")
```

You must load a page on the target domain _before_ `add_cookie` — cookies can't be set for a domain the browser isn't currently on.

HTTP Basic auth dialogs are native browser UI, not JavaScript alerts, so `switch_to.alert` won't help. Embedding credentials in the URL (`https://user:pass@host/`) is deprecated, and modern Chrome blocks it for the subresources a page loads. The reliable approach is an `Authorization` header via CDP:

```python
import base64
token = base64.b64encode(b"user:pass").decode()
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders",
                       {"headers": {"Authorization": f"Basic {token}"}})
```

Keep credentials in environment variables or a secrets manager, and only automate logins on accounts you're authorized to use — scraping behind a login is where terms-of-service and legal exposure concentrate. See our [guide to web scraping legality](https://webscraping.ai/blog/is-web-scraping-legal).

## Custom headers and user agents

Selenium has no API for request headers. This is a deliberate consequence of the WebDriver protocol, which automates the browser rather than the network stack. The user agent is the exception, because it's a browser-level setting:

```python
options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                     "AppleWebKit/537.36 (KHTML, like Gecko) "
                     "Chrome/154.0.0.0 Safari/537.36")
```

For arbitrary headers, use the Chrome DevTools Protocol:

```python
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", {"headers": {
    "Accept-Language": "en-US,en;q=0.9",
    "X-API-Key": os.environ["API_KEY"],
}})
```

Two caveats: `execute_cdp_cmd` is Chrome/Edge only — Firefox has no equivalent — and headers set this way apply to every request the page makes, including subresources. The once-standard `selenium-wire` package offered a nicer interface but is no longer actively maintained, so CDP is the safer choice for new code.

Keep your headers internally consistent: a Windows Chrome user agent paired with a Linux platform fingerprint and a mismatched `Accept-Language` is exactly the incoherence anti-bot systems look for. Our [user agent rotation guide](https://webscraping.ai/blog/user-agent-rotation-for-web-scraping) covers doing this well.

## Proxies

A basic proxy is a launch flag:

```python
options.add_argument("--proxy-server=http://proxy.example.com:8080")
options.add_argument("--proxy-server=socks5://proxy.example.com:1080")
```

The complication is authentication: `--proxy-server` accepts no credentials, and `http://user:pass@host:port` is ignored by Chrome, which instead pops a native auth dialog that blocks your scrape. There are two workable answers.

**Handle the challenge over CDP** — no extra files, Chrome only:

```python
driver.execute_cdp_cmd("Fetch.enable", {"handleAuthRequests": True})
# then answer Fetch.authRequired events with Fetch.continueWithAuth
```

**Or ship a tiny proxy extension** , which is what most production Selenium setups do — a two-file `.zip` with a `background.js` that calls `chrome.webRequest.onAuthRequired` and returns the credentials, loaded via `options.add_extension("proxy_auth.zip")`. It's more setup, but it survives across pages and works with Selenium Grid. One catch: since Chrome 137, branded Google Chrome ignores command-line extension loading, which breaks `add_extension` — run it against Chrome for Testing or Chromium instead (`options.binary_location`).

Rotation happens per browser session, not per request: a launched Chrome keeps its `--proxy-server` for its lifetime, so rotating IPs means starting a new driver with a new proxy (or pointing every session at a rotating gateway endpoint that changes the exit IP for you). Our [proxy provider comparison](https://webscraping.ai/blog/best-proxy-providers-for-web-scraping) covers choosing between datacenter and residential pools.

Note that a proxy fixes IP reputation only. It does nothing about the browser fingerprint that marks your Chrome as automated — see the bot detection section.

## File downloads

Downloads need an explicit directory, set through preferences:

```python
download_dir = os.path.abspath("downloads")
os.makedirs(download_dir, exist_ok=True)

options.add_experimental_option("prefs", {
    "download.default_directory": download_dir,
    "download.prompt_for_download": False,
    "plugins.always_open_pdf_externally": True, # download PDFs, don't preview
})
```

Selenium fires no download-complete event, so wait for the file — specifically, for Chrome's `.crdownload` temp file to disappear:

```python
import time, glob

def wait_for_download(directory, timeout=60):
    deadline = time.time() + timeout
    while time.time() < deadline:
        if not glob.glob(os.path.join(directory, "*.crdownload")):
            files = [f for f in glob.glob(os.path.join(directory, "*"))]
            if files:
                return max(files, key=os.path.getctime)
        time.sleep(0.5)
    raise TimeoutError("download did not finish")
```

When the file is at a plain URL, skip the browser: copy the session cookies into `requests` and download it directly. It's faster, streams to disk, and gives you real error handling.

```python
import requests

session = requests.Session()
for c in driver.get_cookies():
    session.cookies.set(c["name"], c["value"], domain=c["domain"])
with session.get(file_url, stream=True) as r:
    r.raise_for_status()
    with open("report.csv", "wb") as f:
        for chunk in r.iter_content(8192):
            f.write(chunk)
```

## Screenshots

```python
driver.save_screenshot("page.png") # viewport only
driver.find_element(By.CSS_SELECTOR, ".chart").screenshot("chart.png")
png_bytes = driver.get_screenshot_as_png() # for in-memory use
```

`save_screenshot` captures the viewport, not the full page — a persistent difference from Puppeteer and Playwright, which offer full-page capture directly. Two workarounds: set a tall window size before capturing, or use CDP:

```python
result = driver.execute_cdp_cmd("Page.captureScreenshot", {
    "captureBeyondViewport": True, "fromSurface": True})
with open("fullpage.png", "wb") as f:
    f.write(base64.b64decode(result["data"]))
```

Screenshots are also the single best debugging tool for headless scrapers: capture one in your exception handler and you'll usually see the cookie wall, CAPTCHA, or empty state that your selector hit.

## SSL certificate errors

Scraping targets with self-signed or expired certificates fails at the browser level. Selenium 4 exposes this as a capability on the options object:

```python
options.set_capability("acceptInsecureCerts", True)
```

(`desired_capabilities`, which older tutorials pass to the constructor, was removed in Selenium 4.10 — configure everything through `options`.) Chrome also accepts `--ignore-certificate-errors`, though the capability is the portable form. Only disable certificate validation for targets you control or trust; on the public internet it removes your protection against interception.

## Browser extensions

```python
options.add_extension("/path/to/extension.crx") # packed
options.add_argument("--load-extension=/path/to/unpacked") # unpacked
```

Extensions are how ad blockers, proxy authenticators, and fingerprint tweaks get injected. Three constraints: since Chrome 137, branded Google Chrome ignores `--load-extension` and Selenium's extension loading, so use Chrome for Testing or Chromium (`options.binary_location`) when you need them; extensions never loaded in the old headless mode (which Chrome 132 removed — `--headless` now means the new mode); and each one adds startup time and memory to every browser you launch.

## Handling exceptions

Selenium's exception names are precise, and knowing which is which shortens debugging enormously:

| Exception | Meaning | Usual fix |
| --- | --- | --- |
| `NoSuchElementException` | selector matched nothing _now_ | wait for it, or use `find_elements` |
| `TimeoutException` | an explicit wait expired | check the condition and the selector |
| `StaleElementReferenceException` | the DOM replaced your element | re-find it after the page changes |
| `ElementNotInteractableException` | present but not clickable/typable | wait for `element_to_be_clickable` |
| `ElementClickInterceptedException` | something is on top of it | dismiss the overlay or scroll into view |
| `SessionNotCreatedException` | driver/browser mismatch or locked profile | upgrade Selenium; don't pin old drivers |
| `WebDriverException` | base class — browser crashed, session died | restart the driver, check resources |

A resilient scrape wraps per-page work so one bad page can't kill the crawl:

```python
from selenium.common.exceptions import TimeoutException, WebDriverException

def scrape_page(driver, url, retries=2):
    for attempt in range(retries + 1):
        try:
            driver.get(url)
            WebDriverWait(driver, 15).until(
                EC.presence_of_element_located((By.CSS_SELECTOR, ".content")))
            return driver.find_element(By.CSS_SELECTOR, ".content").text
        except TimeoutException:
            if attempt == retries:
                driver.save_screenshot(f"timeout-{attempt}.png")
                raise
        except WebDriverException:
            raise # session-level: let the caller restart
```

Set page-level timeouts too, or a single hanging resource can stall a worker indefinitely: `driver.set_page_load_timeout(30)` and `driver.set_script_timeout(20)`.

## Headless mode and performance

```python
options.add_argument("--headless=new") # Chrome's modern headless
options.add_argument("--no-sandbox") # containers only
options.add_argument("--disable-dev-shm-usage") # avoid /dev/shm crashes
options.add_argument("--blink-settings=imagesEnabled=false")
options.page_load_strategy = "eager" # don't wait for images/subresources
```

`page_load_strategy = "eager"` returns control at `DOMContentLoaded` rather than the full load event and is often the single biggest speed win for scraping. `"none"` returns immediately and leaves all waiting to you.

The costs that actually matter at scale: reuse one driver for many pages rather than launching per page (browser startup dominates otherwise), restart the driver every few hundred pages to keep memory flat, and block images and fonts. And always `quit()` in a `finally` — orphaned Chrome processes are what turns a long crawl into an out-of-memory incident.

## Parallel scraping and Selenium Grid

Selenium's WebDriver objects aren't thread-safe to share, but running _one driver per worker_ is straightforward. Processes beat threads here because the work is browser-bound:

```python
from concurrent.futures import ProcessPoolExecutor

def scrape_one(url):
    driver = webdriver.Chrome(options=make_options())
    try:
        driver.get(url)
        return driver.find_element(By.TAG_NAME, "h1").text
    finally:
        driver.quit()

with ProcessPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(scrape_one, urls))
```

Size `max_workers` by RAM, not CPU count: budget several hundred MB per concurrent browser and leave headroom.

Beyond one machine, **Selenium Grid** distributes sessions across nodes. The quickest start is the official container, then point a `Remote` driver at it:

```bash
docker run -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome:latest
```

```python
driver = webdriver.Remote(
    command_executor="http://localhost:4444/wd/hub",
    options=Options())
```

`--shm-size=2g` is not optional in Docker — the default 64 MB `/dev/shm` causes Chrome to crash under load. The same image is what most CI pipelines use as a service container, which makes Grid the natural path when your scraper needs to run in CI alongside tests.

## Bot detection: what works and what doesn't

Automated Chrome is detectable, and honesty here saves a lot of wasted effort. Default Selenium sets `navigator.webdriver = true` and carries other automation tells; the usual first mitigations are:

```python
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)
```

`undetected-chromedriver` (`pip install undetected-chromedriver`, then `uc.Chrome()`) patches the driver binary and Chrome's startup to hide more of these signals, and it does get past the softer checks that stop vanilla Selenium — though its last PyPI release was February 2024, so expect breakage on new Chrome versions.

What none of this reliably beats: Cloudflare's managed challenges, DataDome, PerimeterX, and similar commercial systems. They combine TLS fingerprinting, canvas and WebGL fingerprints, mouse-movement behavior, and IP reputation — headless Chrome loses on several of those regardless of which flags you set, and every Chrome release resets the arms race. If your target sits behind one of these, the realistic options are a well-maintained residential proxy pool plus significant fingerprinting work, or an API that owns that problem for you.

Practical hygiene matters more than tricks: respect `robots.txt`, rate-limit yourself, randomize timing, and don't hammer a site in parallel from one IP.

## Selenium vs Playwright vs Puppeteer

| | Selenium | Playwright | Puppeteer |
| --- | --- | --- | --- |
| Languages | Python, Java, C#, JS, Ruby, more | Python, JS, Java, C# | JavaScript/TypeScript |
| Browsers | Chrome, Firefox, Edge, Safari | Chromium, Firefox, WebKit | Chrome, Firefox (BiDi) |
| Waiting | manual explicit waits | auto-waits on every action | manual, with helpers |
| Protocol | W3C WebDriver (standard) | own driver protocol | DevTools Protocol |
| Ecosystem | largest; grids, cloud vendors, CI | growing fast | Node/Chrome-centric |
| Best fit | existing code, exotic browsers, grids | new projects | Node-only Chrome work |

The honest read for 2026: for a **new** scraping project, [Playwright](https://webscraping.ai/blog/playwright-web-scraping) is the better default — auto-waiting removes the largest category of Selenium bugs, and it's meaningfully faster per page. Selenium wins on reach (real Safari, more languages, the W3C standard, the biggest grid and cloud ecosystem) and on inertia: it powers an enormous amount of working code, and rewriting a functioning scraper is rarely the highest-value change. [Puppeteer](https://webscraping.ai/blog/puppeteer-web-scraping) is the choice when you're Node-only and Chrome-focused.

On bot detection, all three are roughly equivalent — none of them wins that fight on its own.

## When Selenium isn't enough

Selenium solves rendering and interaction. It doesn't solve blocked IPs, anti-bot challenges, CAPTCHAs, or the operational cost of running a browser fleet. When those become the bottleneck, you can keep your parsing code and let an API handle the browser — [WebScraping.AI](https://webscraping.ai/) renders pages in real Chrome behind rotating datacenter or residential proxies and returns the result over plain HTTP:

```python
import requests

# Fully rendered HTML through a managed browser + residential proxies
html = requests.get("https://api.webscraping.ai/html", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "js": "true",
    "proxy": "residential",
}).text

# Or skip parsing entirely — structured fields from any page
fields = requests.get("https://api.webscraping.ai/ai/fields", params={
    "api_key": API_KEY,
    "url": "https://example.com/product/42",
    "fields[name]": "Product name",
    "fields[price]": "Price with currency",
}).json()
```

A common migration path: prototype locally with Selenium, then replace `driver.get()` + `driver.page_source` with the [/html endpoint](https://webscraping.ai/docs) in production and keep your Beautiful Soup parsing unchanged. See the [AI web scraping overview](https://webscraping.ai/ai-web-scraping) for the extraction endpoints, or the [n8n node](https://webscraping.ai/integrations/n8n) and [MCP server](https://webscraping.ai/integrations/mcp-server) if the scrape is feeding a workflow or an agent rather than a Python script.

## Frequently asked questions

**Do I still need to download ChromeDriver?** No. Selenium Manager has been built in since Selenium 4.6 and resolves the right driver for your installed browser automatically. Delete `webdriver-manager` from your requirements and stop pinning driver binaries — a stale pinned driver is now a more common failure than a missing one.

**Why does element.text return an empty string?** Because `text` returns only _visible_ text. If the element is hidden, collapsed, or scrolled out of a container with `overflow: hidden`, you get `""` even though the element exists. Use `get_attribute("textContent")` for the underlying text regardless of visibility.

**What's the difference between find\_element and find\_elements?** `find_element` returns the first match and raises `NoSuchElementException` when there is none; `find_elements` returns a list and returns `[]` instead of raising. Use the plural form for existence checks — it's cleaner than wrapping the singular in try/except.

**How do I use an authenticated proxy with Selenium?** Not through `--proxy-server`, which ignores credentials. Either answer Chrome's auth challenge over CDP (`Fetch.enable` with `handleAuthRequests`) or load a small proxy-auth extension via `options.add_extension()`. The extension approach is what most production setups use because it also works with Selenium Grid — but since Chrome 137 it needs Chrome for Testing or Chromium, because branded Chrome no longer loads extensions this way.

**Can Selenium set custom HTTP headers?** Not natively — the WebDriver protocol doesn't expose the network layer. The user agent has a launch flag; everything else needs `driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", ...)`, which is Chrome and Edge only.

**Can Selenium bypass Cloudflare or CAPTCHAs?** Not reliably. `undetected-chromedriver` and the anti-automation flags handle soft checks, but commercial anti-bot systems fingerprint TLS, canvas, WebGL, and behavior, and detect automated Chrome regardless of language. For protected targets, a scraping API that owns the challenge layer — like [WebScraping.AI](https://webscraping.ai/) — is usually cheaper than maintaining an evasion stack.

**Is Selenium or Playwright better for web scraping?** Playwright for new projects: auto-waiting eliminates the most common source of flaky scrapers, and it's faster per page. Selenium for existing code, unusual browsers like real Safari, languages Playwright doesn't support, or an established Grid. Neither has an advantage at avoiding bot detection.

**How many browsers can I run in parallel?** Budget 100–400 MB of RAM per concurrent Chrome and size by memory rather than CPU cores — four to eight workers on a typical 8–16 GB machine. Use one driver per process, `quit()` in a `finally`, and restart drivers every few hundred pages to keep memory flat.

**Is scraping with Selenium legal?** Automating a browser is legal in itself; what matters is what you access and how — public versus gated data, terms of service, rate pressure, and privacy law. Our [web scraping legality guide](https://webscraping.ai/blog/is-web-scraping-legal) covers the current landscape.