Web Scraping with PHP: The Complete Guide

Web scraping in PHP is two choices, not twenty: an HTTP client to fetch the page, and a parser to pull data out of it. Everything else in the ecosystem is a wrapper over that pair. In 2026 the default answer is Guzzle or native cURL for the fetch and Symfony DomCrawler for the parse, with Symfony Panther as the escape hatch when the data only exists after JavaScript runs.

This guide covers the whole toolbox: cURL and its SSL failures, DOMDocument and DOMXPath, the legacy-but-still-searched Simple HTML DOM, DomCrawler and DiDOM, BrowserKit for logins, Panther for JavaScript-rendered pages, and the anti-blocking ladder — with the honest status of each library, because two of the ones tutorials still recommend are dead. Goutte has been abandoned since 2023, and Simple HTML DOM hasn't shipped a stable release since 2019.

Key Takeaways

• composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector is the entire toolchain for static sites. Omitting symfony/css-selector is the most common beginner error — DomCrawler's filter() throws a LogicException without it, while filterXPath() keeps working.
• DOMDocument::loadHTML() uses libxml's HTML4 parser, which mangles HTML5. On PHP 8.4+, Dom\HTMLDocument parses to the WHATWG spec and adds querySelectorAll with no Composer dependency.
• Never set CURLOPT_SSL_VERIFYPEER => false to make an SSL error go away. cURL error 60 means a missing or stale CA bundle — fix curl.cainfo in php.ini instead.
• Goutte is abandoned. Its last release is v4.0.3 (April 2023) and its own Packagist page tells you to use symfony/browser-kit instead. Replacing Goutte\Client with Symfony\Component\BrowserKit\HttpBrowser is close to a find-and-replace.
• Simple HTML DOM's newest release is 2.0-RC2, from November 2019. If you must use it, use the voku/simple_html_dom fork, which supports PHP 8. Don't start new projects on either.
• Symfony DomCrawler 8.x requires PHP >= 8.4.1. On PHP 8.2 or 8.3, pin symfony/dom-crawler:^7.4 — same API.
• Panther v2.4.0 (January 2026) is the only mainstream pure-PHP route to JavaScript-rendered pages. It drives a real Chrome over WebDriver, so budget a browser binary and hundreds of MB of RAM per concurrent worker.

What do you actually need to scrape a page in PHP?

Split every scraper into two layers and the library choice becomes obvious:

One decision rule tells you whether that's enough. Fetch the page with no browser and search the raw response for a value you need:

curl -s https://example.com/products | grep -i "29.99"

If the value is in there, an HTTP client plus a parser is all you need, and it will be 10–50× cheaper per page than a browser. If it isn't, the page renders client-side — jump to scraping JavaScript-rendered pages below, and see our headless browser guide for when that cost is worth paying.

Which PHP scraping library should you use?

How do you scrape a page with cURL in PHP?

cURL ships with essentially every PHP install, so a dependency-free scraper is a single function:

<?php

function fetchPage(string $url, array $extraHeaders = []): string
{
    $ch = curl_init();

    curl_setopt_array($ch, [
        CURLOPT_URL            => $url,
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_MAXREDIRS      => 5,
        CURLOPT_CONNECTTIMEOUT => 5,
        CURLOPT_TIMEOUT        => 30,
        CURLOPT_ENCODING       => '', // accept every encoding this build supports
        CURLOPT_USERAGENT      => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
            . 'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36',
        CURLOPT_HTTPHEADER     => array_merge([
            'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Language: en-US,en;q=0.9',
        ], $extraHeaders),
    ]);

    $body = curl_exec($ch);

    if ($body === false) {
        $error = curl_error($ch);
        curl_close($ch);
        throw new RuntimeException("Transport error for {$url}: {$error}");
    }

    $status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
    curl_close($ch);

    if ($status >= 400) {
        throw new RuntimeException("HTTP {$status} for {$url}");
    }

    return $body;
}

Keep that user-agent string current — a Chrome build from three years ago is itself a fingerprint. Our guide to user-agent rotation covers how much rotation actually buys you, HTTP headers for web scraping covers the rest of the header set, and the cURL commands reference maps the command-line flags to these CURLOPT_* constants when you're debugging a request in the shell first.

file_get_contents() works too, and you'll see it in older tutorials. It's fine for a throwaway script, but it gives you no status code without parsing $http_response_header, no timeout control without a stream context, and no connection reuse. Reach for cURL as soon as the script matters.

How do you fix SSL and HTTPS errors when scraping with PHP?

Nearly every "PHP scraping HTTPS" problem is the same one:

cURL error 60: SSL certificate problem: unable to get local issuer certificate

That message means your PHP install can't find a CA bundle to verify the server's certificate against — it is almost never a problem with the target site. Fresh XAMPP, WAMP, and some Docker images ship without one.

Every tutorial's answer is to turn verification off:

// Do not do this.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

That doesn't fix the error, it deletes the check that produced it — every request becomes a silent man-in-the-middle opportunity, and on a proxied scraper (where traffic already passes through a third party) that's a live risk rather than a theoretical one. The same goes for file_get_contents() with "verify_peer" => false in its stream context.

The fix takes two minutes. Download the CA bundle from the official cURL site, put it somewhere permanent, and point PHP at it:

; php.ini — applies to both cURL and PHP's OpenSSL streams
curl.cainfo = "/etc/ssl/certs/cacert.pem"
openssl.cafile = "/etc/ssl/certs/cacert.pem"

Restart PHP-FPM or your web server afterwards. If you can't edit php.ini (shared hosting), set it per request instead — still verifying, just with an explicit bundle:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); // 2 = also check the hostname
curl_setopt($ch, CURLOPT_CAINFO, __DIR__ . '/certs/cacert.pem');

CURLOPT_SSL_VERIFYHOST takes 2, not true — the value 1 was removed years ago and passing a boolean silently weakens the check. Two other SSL failures worth recognising:

• Error 35 / handshake failure on an old server usually means a TLS version mismatch. Force one with CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_2 rather than disabling verification.
• "certificate has expired" on a site that loads fine in your browser means your bundle is stale, not their certificate. Re-download it; browsers ship their own trust store and won't tell you yours has rotted.

When is Guzzle worth the extra dependency?

The moment you need more than one page at a time. Pool runs requests over cURL's multi handle with a concurrency cap, which is the difference between a scraper that takes an hour and one that takes two minutes:

<?php

require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
use Psr\Http\Message\ResponseInterface;

$client = new Client(['timeout' => 20, 'connect_timeout' => 5]);
$urls   = ['https://example.com/p/1', 'https://example.com/p/2', 'https://example.com/p/3'];
$pages  = [];

$requests = static function () use ($urls) {
    foreach ($urls as $url) {
        yield new Request('GET', $url);
    }
};

$pool = new Pool($client, $requests(), [
    'concurrency' => 5,
    'fulfilled'   => static function (ResponseInterface $response, int $i) use (&$pages, $urls) {
        $pages[$urls[$i]] = (string) $response->getBody();
    },
    'rejected'    => static function (Throwable $reason, int $i) use ($urls) {
        fwrite(STDERR, "failed {$urls[$i]}: {$reason->getMessage()}\n");
    },
]);

$pool->promise()->wait();

Guzzle sets no request timeout by default and has no built-in retry option — both are things you configure yourself. Our PHP Guzzle guide covers the exception hierarchy, retry middleware, and proxy options in depth; this post stays on the scraping side.

How do you parse HTML with DOMDocument in PHP?

DOMDocument is bundled with PHP through ext-dom, so it's the parser you reach for when you don't want a Composer dependency at all. Load the HTML, wrap it in a DOMXPath, and query:

<?php

$html = fetchPage('https://books.toscrape.com/');

$dom = new DOMDocument();

// Real-world HTML is never valid. Collect libxml's complaints instead of
// printing a warning per unclosed tag.
libxml_use_internal_errors(true);
$dom->loadHTML($html);
libxml_clear_errors();

$xpath = new DOMXPath($dom);

$products = [];

foreach ($xpath->query("//article[contains(@class, 'product_pod')]") as $node) {
    $title = $xpath->query('.//h3/a/@title', $node)->item(0);
    $price = $xpath->query(".//p[contains(@class, 'price_color')]", $node)->item(0);

    $products[] = [
        'title' => $title?->nodeValue,
        'price' => $price ? trim($price->textContent) : null,
    ];
}

print_r($products);

Four things account for most DOMDocument bug reports:

The @ and libxml_use_internal_errors() question. loadHTML() emits a PHP warning for every malformed-HTML complaint libxml raises, which on a real page can be hundreds. libxml_use_internal_errors(true) routes them into a buffer you can inspect with libxml_get_errors(); @$dom->loadHTML() just discards them, including the ones that would have told you the fetch returned an error page. Prefer the former.

The context node in query(). $xpath->query('//h3', $node) ignores $node entirely — a leading // searches from the document root no matter what context you pass. Relative queries need .//. This is the single most common reason a scraper returns the first product's title for all fifty products.

Encoding. loadHTML() assumes ISO-8859-1 unless the markup says otherwise, so UTF-8 pages come back with mojibake. The old fix, mb_convert_encoding($html, 'HTML-ENTITIES', 'UTF-8'), is deprecated as of PHP 8.2 and will emit warnings. Prepend a content-type hint instead:

$dom->loadHTML(
    '<meta http-equiv="Content-Type" content="text/html; charset=utf-8">' . $html,
    LIBXML_NOERROR
);

It's an HTML4 parser. libxml predates HTML5, so <section>, <template>, and unquoted or namespaced attributes can end up in the wrong place in the tree. Usually it degrades gracefully. When it doesn't, that's the reason.

getElementsByTagName() and getAttribute() cover simple cases without XPath, but XPath is what makes DOMDocument competitive with the Composer options — it handles text matching, axis traversal, and positional predicates that CSS selectors can't express. Our XPath cheat sheet has the expressions worth memorising.

Dom\HTMLDocument on PHP 8.4+

PHP 8.4 added a genuinely new parser in the Dom namespace that implements the WHATWG HTML5 spec, plus the querySelector/querySelectorAll methods people have been installing packages to get:

<?php

$doc = Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);

foreach ($doc->querySelectorAll('article.product_pod h3 a') as $link) {
    echo $link->getAttribute('title'), PHP_EOL;
}

This is the right default for new code on 8.4 or newer. It parses modern markup correctly where DOMDocument::loadHTML() doesn't, needs no dependency, and the CSS-selector API means you can copy selectors straight out of devtools. The old DOMDocument class still exists and still works — nothing breaks — but there's no reason to start there anymore.

How do you parse HTML with Symfony DomCrawler?

DomCrawler is the default for anything larger than a script. Install it with the CSS selector bridge or filter() will throw:

composer require symfony/dom-crawler symfony/css-selector

<?php

require 'vendor/autoload.php';

use Symfony\Component\DomCrawler\Crawler;

$crawler = new Crawler(fetchPage('https://books.toscrape.com/'));

$books = $crawler->filter('article.product_pod')->each(static function (Crawler $node) {
    return [
        'title' => $node->filter('h3 a')->attr('title'),
        // passing a default keeps a missing node from throwing
        'price' => $node->filter('.price_color')->text(''),
        'url'   => $node->filter('h3 a')->link()->getUri(),
    ];
});

print_r($books);

Two gotchas:

• ->text() and ->attr() throw an InvalidArgumentException when the node list is empty. Pass a default (->text('')) or guard with ->count(). A scraper that dies on the one product missing a price is worse than one that records null.
• filter() needs symfony/css-selector; filterXPath() never does. That LogicException about a missing CSS selector converter means exactly one thing: run composer require symfony/css-selector.

DiDOM (imangazaliev/didom) is the lighter alternative in the same shape — CSS selectors and XPath over ext-dom, find() returning arrays of elements, and a smaller install than pulling in a Symfony component. It's a reasonable pick if DomCrawler feels heavy for what you're doing; DomCrawler wins on ecosystem, since BrowserKit and Panther both return Crawler objects and reuse the same extraction code.

Is Simple HTML DOM still safe to use?

Short answer: it works, and you shouldn't start new projects with it.

The original library's most recent release on Packagist is 2.0-RC2 from November 2019 — a release candidate that never became stable, now approaching seven years old and predating PHP 8 entirely. If you're maintaining code that already depends on it, switch to the maintained fork rather than the original:

# maintained fork, PHP 8 compatible
composer require voku/simple_html_dom

# original — legacy projects only
composer require simplehtmldom/simplehtmldom

The fork namespaces everything under voku\helper\HtmlDomParser, so it's a drop-in only after you update the entry points:

<?php

require 'vendor/autoload.php';

use voku\helper\HtmlDomParser;

$html = HtmlDomParser::str_get_html(fetchPage('https://example.com/products'));

foreach ($html->find('article.product') as $product) {
    echo $product->find('h3', 0)->plaintext, PHP_EOL;
}

Two things to know before you write any of this. file_get_html() fetches the URL for you with PHP's stream wrappers, which means no timeout, no custom headers, no proxy, and no status code — fetch with cURL or Guzzle and pass the string to str_get_html() instead. And the parser holds circular references, so long-running scripts leak memory unless you call $html->clear(); unset($html); after each page. The classic Call to undefined function file_get_html() error is just a missing require of the autoloader or the standalone simple_html_dom.php.

The sections below cover the tasks people actually hit with it, with the modern equivalent alongside — because on PHP 8.4 you can do all of them with no dependency at all.

Handling malformed HTML

Simple HTML DOM's selling point was tolerance: it's a regex-and-string-based parser, so it happily walks markup with unclosed <div>s, stray <, and mismatched nesting that a strict XML parser would reject outright. That tolerance is also why it's slow and memory-hungry on large documents.

You get the same tolerance from ext-dom — libxml's HTML parser is a recovering parser by design — as long as you handle its complaints instead of suppressing them:

<?php

libxml_use_internal_errors(true);

$dom = new DOMDocument();
$loaded = $dom->loadHTML($html, LIBXML_NOERROR | LIBXML_NOWARNING);

$fatal = array_filter(
    libxml_get_errors(),
    static fn (LibXMLError $e) => $e->level === LIBXML_ERR_FATAL
);
libxml_clear_errors();

if (!$loaded || $fatal !== []) {
    // A page this broken usually means you got an error page, not a product page.
    throw new RuntimeException('Unparseable response');
}

Two practical points. First, check what you fetched before blaming the parser: truncated HTML is far more often a broken response, a WAF challenge page, or a gzip body you forgot to decode than a genuinely malformed document. Second, if the markup really is hostile, Dom\HTMLDocument on PHP 8.4+ follows the HTML5 spec's error-recovery rules — the same ones your browser uses — which is a strictly better answer than either libxml's HTML4 recovery or Simple HTML DOM's string matching.

Parsing HTML fragments instead of full documents

AJAX endpoints often return a bare <li> list or a <tr> block with no <html> wrapper. DOMDocument::loadHTML() "helpfully" wraps a fragment in <html><body>, which shifts your XPath expressions and breaks selectors written against the real page.

Simple HTML DOM's str_get_html() handles fragments without wrapping. With ext-dom, use LIBXML_HTML_NOIMPLIED | LIBXML_HTML_NODEFDTD:

<?php

$fragment = '<li class="item">First</li><li class="item">Second</li>';

$dom = new DOMDocument();
libxml_use_internal_errors(true);
$dom->loadHTML($fragment, LIBXML_HTML_NOIMPLIED | LIBXML_HTML_NODEFDTD | LIBXML_NOERROR);
libxml_clear_errors();

$xpath = new DOMXPath($dom);

foreach ($xpath->query("//li[@class='item']") as $item) {
    echo trim($item->textContent), PHP_EOL;
}

Watch out for one trap: LIBXML_HTML_NOIMPLIED on a fragment with multiple top-level siblings can behave inconsistently across libxml versions. If results look wrong, wrap the fragment in a single container element yourself ('<div id="frag">' . $fragment . '</div>') and query inside it — explicit and version-proof. DomCrawler does this for you: new Crawler($fragment) treats the input as a fragment already.

One more caveat specific to fragments returned by XHR: a <tr> or <td> fragment parsed outside a <table> gets dropped by HTML5 parsing rules, because those elements are only valid inside table context. Wrap it in <table><tbody>…</tbody></table> before parsing.

Extracting data from HTML tables with complex structures

Flat tables are three lines. The problems start with colspan and rowspan, where the grid position of a cell no longer matches its index in the row.

The fix is to build a grid and fill every position a merged cell occupies, so later rows land in the right columns:

<?php

function tableToGrid(DOMElement $table, DOMXPath $xpath): array
{
    $grid = [];
    $rowIndex = 0;

    foreach ($xpath->query('.//tr', $table) as $tr) {
        $colIndex = 0;

        foreach ($xpath->query('./th | ./td', $tr) as $cell) {
            // step over positions already claimed by a rowspan above
            while (isset($grid[$rowIndex][$colIndex])) {
                $colIndex++;
            }

            $text    = trim($cell->textContent);
            $colspan = max(1, (int) $cell->getAttribute('colspan'));
            $rowspan = max(1, (int) $cell->getAttribute('rowspan'));

            for ($r = 0; $r < $rowspan; $r++) {
                for ($c = 0; $c < $colspan; $c++) {
                    $grid[$rowIndex + $r][$colIndex + $c] = $text;
                }
            }

            $colIndex += $colspan;
        }

        $rowIndex++;
    }

    ksort($grid);

    return array_map(static function (array $row) {
        ksort($row);
        return $row;
    }, $grid);
}

Three details that bite in production:

• Nested tables. .//tr matches rows of inner tables too. Scope to the direct children (./tbody/tr | ./tr) or filter out rows whose nearest ancestor table isn't the one you're processing.
• <tbody> is implied. Browsers insert a <tbody> that isn't in the source, so an XPath of /table/tr written from devtools fails against the parsed markup. Always use .//tr, never a rigid absolute path.
• Multi-level headers. When a table has two header rows, treat every row containing <th> as header and combine them into composite column names, rather than assuming row zero is the header.

Extracting dropdown and select options

Select elements are the easiest structured data on a page — value/label pairs, already normalised — which is why they're a common scraping target for country lists, size charts, and filter taxonomies:

<?php

$crawler = new Crawler($html);

$options = $crawler->filter('select[name="country"] option')->each(
    static fn (Crawler $option) => [
        'value'    => $option->attr('value') ?? '',
        'label'    => trim($option->text('')),
        'selected' => $option->attr('selected') !== null,
    ]
);

Three things the naive version misses. An option with no value attribute submits its text content as the value, so $option->attr('value') ?? '' needs an explicit fallback to the label if you're reproducing form behaviour. <optgroup> labels carry meaning — a "Europe" grouping is data you're throwing away if you flatten straight to options. And a placeholder first option (<option value="">Select a country</option>) is not a real choice; filter empty values out.

The big one: dropdowns are frequently populated by JavaScript after page load, or by an XHR triggered when a previous dropdown changes (country → state → city). If the option list comes back with one placeholder entry, the data isn't in the HTML — open the network tab, find the request that returns the list, and call that JSON endpoint directly. Our API scraping guide covers finding and using those endpoints, and it's almost always faster than rendering the page.

Extracting image URLs

Two problems, both easy to miss. Image src attributes are usually relative, and lazy-loaded images don't put the real URL in src at all:

<?php

$base    = 'https://example.com/catalog/';
$crawler = new Crawler($html);

$images = $crawler->filter('img')->each(static function (Crawler $img) use ($base) {
    // lazy loaders park the real URL in a data- attribute and leave a
    // placeholder or an inline SVG in src
    $src = $img->attr('data-src')
        ?? $img->attr('data-original')
        ?? $img->attr('src');

    if ($src === null || str_starts_with($src, 'data:')) {
        return null;
    }

    // srcset holds the highest-resolution variants: "url 1x, url 2x"
    $srcset = $img->attr('srcset');

    return [
        'url' => (string) Symfony\Component\DomCrawler\UriResolver::resolve($src, $base),
        'alt' => $img->attr('alt') ?? '',
        'srcset' => $srcset,
    ];
});

$images = array_values(array_filter($images));

UriResolver::resolve() ships with DomCrawler and handles protocol-relative (//cdn.example.com/…), root-relative, and ../ paths correctly — resolving them with string concatenation gets ../ wrong every time. If the page has a <base href>, resolve against that instead of the page URL. Background images set in CSS (background-image: url(...)) aren't in any img tag; you'd need to parse inline styles or the stylesheet, and at that point a browser is usually the cheaper answer.

Goutte is abandoned — what replaced it?

Goutte's Packagist page carries a plain abandonment notice recommending symfony/browser-kit, and that's not a demotion: from v4 onward Goutte was already a thin proxy over Symfony's HttpBrowser. The migration is mostly imports.

composer require symfony/browser-kit symfony/http-client symfony/css-selector

<?php

require 'vendor/autoload.php';

use Symfony\Component\BrowserKit\HttpBrowser;
use Symfony\Component\DomCrawler\Crawler;
use Symfony\Component\HttpClient\HttpClient;

$browser = new HttpBrowser(HttpClient::create());

// log in, then scrape the pages behind the session
$crawler = $browser->request('GET', 'https://quotes.toscrape.com/login');

$form = $crawler->selectButton('Login')->form([
    'username' => 'admin',
    'password' => 'admin',
]);

$crawler = $browser->submit($form);

$quotes = $crawler->filter('.quote .text')->each(
    static fn (Crawler $node) => $node->text('')
);

// follow a link the same way a browser would
$crawler = $browser->click($crawler->selectLink('Next')->link());

HttpBrowser keeps cookies across requests, so a login is just the first request in the sequence, and selectButton(...)->form() reads the hidden CSRF token straight out of the markup — which is why this beats hand-rolling the POST. It does not execute JavaScript: if the login posts through a JS handler rather than a plain <form>, you need a real browser.

How do you scrape JavaScript-rendered pages in PHP?

This is where PHP is genuinely weaker than Node or Python, and it's worth being direct about the options.

file_get_contents() and cURL fetch the server's HTML and stop. A React, Vue, or Angular page ships a near-empty <div id="root"> and fills it in the browser, so those tools return a skeleton no matter how good your selectors are. You have four ways out, in rough order of how often they're the right call:

1. Find the underlying API. Client-side rendering means the data arrives over XHR as JSON. Open devtools, filter to Fetch/XHR, reload, and look for the request that returns your data. Calling that endpoint from Guzzle is faster, more stable, and returns parsed data instead of markup you have to scrape. Try this before anything below — see the API scraping guide.

2. Symfony Panther. The pure-PHP route: it speaks the W3C WebDriver protocol to a real Chrome or Firefox, and exposes the same Crawler API as DomCrawler, so extraction code you already wrote keeps working. Covered in detail below.

3. A rendering API. Offload the browser to a service and keep your PHP process a lightweight HTTP client. This is the option when you need JS rendering across many pages and don't want browser infrastructure in your deploy.

4. Shelling out to Node. shell_exec('node scrape.js ' . escapeshellarg($url)) with a Puppeteer or Playwright script works, and PHP wrappers around Puppeteer exist to make it feel native. Both add a Node runtime to every machine that runs the scraper — the dependency you were presumably using PHP to avoid — and shell_exec with user-controlled input is a command-injection hazard unless every argument goes through escapeshellarg(). There's also chrome-php/chrome, which talks the Chrome DevTools Protocol from PHP directly and skips the Node runtime, though it's a smaller project than Panther with less documentation.

Scraping with Symfony Panther

composer require --dev symfony/panther dbrekelmans/bdi
vendor/bin/bdi detect drivers   # downloads a matching chromedriver

<?php

require 'vendor/autoload.php';

use Symfony\Component\DomCrawler\Crawler;
use Symfony\Component\Panther\Client;

$client = Client::createChromeClient(null, ['--headless=new', '--disable-gpu', '--window-size=1920,1080']);

$client->request('GET', 'https://example.com/spa/products');

// block until the client-rendered nodes exist, then extract
$crawler = $client->waitFor('.product-grid .product', 10);

$products = $crawler->filter('.product-grid .product')->each(
    static fn (Crawler $node) => [
        'name'  => $node->filter('h2')->text(''),
        'price' => $node->filter('[data-price]')->attr('data-price'),
    ]
);

$client->quit();

The costs are real: a browser binary and a matching driver on every machine that runs the scraper, hundreds of MB of RAM per concurrent instance, and startup time measured in seconds rather than milliseconds. Always quit() — orphaned chromedriver processes are the classic way to fill a scraping box's memory overnight, and try/finally is the only way to guarantee it runs when extraction throws.

How do you wait for elements to load in Panther?

The single most common Panther bug is extracting before the page has finished rendering, and the single most common bad fix is sleep(5) — which is both too slow on fast responses and too short on slow ones. Panther has purpose-built waits:

<?php

// wait until a node exists in the DOM (default timeout: 30s)
$client->waitFor('.product-grid .product');

// wait until it exists *and* is visible — a node hidden by CSS satisfies
// waitFor() but has no text to extract
$client->waitForVisibility('#price', 15);

// wait for a loading state to go away; usually more reliable than waiting
// for content, because the spinner is removed only when the render is done
$client->waitForInvisibility('.loading-spinner', 20);

// wait for specific text, for pages that render placeholders first
$client->waitForElementToContain('#status', 'In stock', 10);

The rules that make waits reliable:

• Wait for the thing you're about to read, not for a generic container. A grid element can exist while its children are still being appended.
• Prefer waiting for the spinner to disappear over waiting for content to appear. Content selectors often match a skeleton placeholder that has the same class as the real thing.
• Second argument is the timeout in seconds, and its default is generous. Lower it for optional elements so a missing one costs you 3 seconds rather than 30.
• A wait that times out throws. Facebook\WebDriver\Exception\TimeoutException (Panther runs on php-webdriver) is what you catch for a genuinely optional element:

try {
    $client->waitFor('.promo-banner', 3);
    $promo = $client->getCrawler()->filter('.promo-banner')->text('');
} catch (\Facebook\WebDriver\Exception\TimeoutException) {
    $promo = null;   // not every product has one
}

• Don't poll in a loop with usleep(). If none of the built-in waits express your condition, $client->wait() returns a WebDriverWait you can drive with a custom callback — it uses the driver's own polling rather than blocking your process.

How do you debug a Panther scraper?

Headless failures are opaque: the selector didn't match, and you can't see the page. Panther's debugging surface is mostly environment variables, set before the client is created:

PANTHER_NO_HEADLESS=1          # run with a visible browser window
PANTHER_ERROR_SCREENSHOT_DIR=./var/error-screenshots
PANTHER_NO_SANDBOX=1           # required in most Docker images
PANTHER_DEVTOOLS=0             # suppress the devtools window when not headless

PANTHER_NO_HEADLESS=1 answers most questions in one run — you watch the page load and see the cookie wall, the CAPTCHA, or the empty grid for yourself. When that isn't possible (CI, a remote box), capture the state at the moment of failure:

<?php

try {
    $crawler = $client->waitFor('.product-grid .product', 10);
} catch (\Throwable $e) {
    // what the browser actually rendered
    $client->takeScreenshot('var/debug/failure.png');
    file_put_contents('var/debug/failure.html', $client->getPageSource());

    // JS errors and failed requests the page logged
    foreach ($client->getWebDriver()->manage()->getLog('browser') as $entry) {
        fwrite(STDERR, "[{$entry['level']}] {$entry['message']}\n");
    }

    throw $e;
} finally {
    $client->quit();
}

Those three artefacts — screenshot, page source, console log — settle almost every case. The screenshot shows a block page or consent dialog instantly. The saved HTML lets you test selectors offline. The console log catches the JS error that stopped rendering before your element existed.

Two more things worth knowing. $client->getPageSource() returns the rendered DOM, not the server's response — diffing it against a plain cURL fetch of the same URL tells you exactly what JavaScript added, and therefore whether you needed a browser at all. And a selector that works in your browser's devtools but not in Panther usually means content inside an <iframe>: you have to $client->switchTo()->frame(...) first, since selectors don't cross document boundaries.

What about Roach PHP and PHPScraper?

Two useful libraries a level up from the fetch/parse pair:

• Roach PHP (v3.2.1, March 2025) is a Scrapy-style crawling framework: spiders, request scheduling, downloader middleware, item processing pipelines. Worth it when you're crawling thousands of pages with dedupe and multi-stage processing, overkill for a nightly price check.
• PHPScraper (v3.0.0, April 2024) wraps BrowserKit and DomCrawler behind properties like $web->title, $web->links, $web->contentKeywords. Fast for metadata and content extraction; you'll drop to DomCrawler as soon as you need a specific selector.

How do you avoid getting blocked?

The escalation ladder, cheapest first:

1. Send real headers. A request with no Accept-Language and PHP's default user agent is trivially identifiable. The cURL example above is the floor.
2. Slow down. One request per second per domain stops most rate limiters. usleep(random_int(800_000, 1_500_000)) between requests, not a fixed sleep(1) — perfectly periodic traffic is itself a signal.
3. Retry properly. Exponential backoff on 429 and 5xx, and honour Retry-After when the response includes it. Never retry a 403 immediately; it means something about your request, not your timing.
4. Rotate IPs. Datacenter proxies until you start seeing 403s and CAPTCHAs, then residential. Both ext-curl (CURLOPT_PROXY) and Guzzle (the proxy option) support them directly; our proxy types guide covers the difference.
5. Render only when you have to. A headless browser also fixes some blocks, because it produces a real TLS and JS fingerprint — but at 10–50× the cost per page.

None of this settles whether you should scrape a given target. Terms of service, copyright, and personal-data rules (GDPR, CCPA) all apply independently of the technique; our guide to web scraping legality covers where the lines actually sit.

Scraping with the WebScraping.AI PHP SDK

When maintaining proxies and headless browsers stops being the interesting part of the job, the fetch layer can be an API call. Our PHP SDK is PSR-18 based (it works with Guzzle, Symfony HttpClient, or anything else you already have) and returns strings and arrays you hand straight to DomCrawler:

<?php
// composer require webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use Symfony\Component\DomCrawler\Crawler;
use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');

// JS-rendered HTML through a residential IP in Germany — no browser on your box
$html = $client->html(
    'https://example.com/products',
    js: true,
    waitFor: '.product-grid .product',
    proxy: 'residential',
    country: 'de',
);

$products = (new Crawler($html))->filter('.product-grid .product')->each(
    static fn (Crawler $node) => $node->filter('h2')->text('')
);

If you'd rather not add a dependency, it's a plain GET — the same fetchPage() from the top of this guide works against the API, and the wait_for parameter replaces every Panther wait above:

$html = fetchPage('https://api.webscraping.ai/html?' . http_build_query([
    'api_key'  => 'YOUR_API_KEY',
    'url'      => 'https://example.com/products',
    'js'       => 'true',
    'wait_for' => '.product-grid .product',
    'proxy'    => 'residential',
]));

Or skip the selectors entirely. AI field extraction takes plain-English field descriptions and returns structured data, which survives the markup changes that break CSS selectors:

$product = $client->fields('https://example.com/product/1', [
    'name'  => 'Product name',
    'price' => 'Price with currency symbol',
    'stock' => 'Whether the item is in stock',
]);

print_r($product);
// ['name' => '...', 'price' => '$29.99', 'stock' => 'In stock']

The pricing is a published multiplier table, not a mystery: a datacenter fetch costs 1 credit without JS and 5 with it, residential is 10 and 25, stealth is 50, and AI extraction adds 5. Failed requests are never charged. The free tier is 2,000 credits a month with no credit card, which is enough to see whether the pages you care about come back clean. The full parameter list — timeout, device, js_script, custom_proxy, error_on_404 — is in the API docs, and the same endpoints back use cases like price monitoring and product data aggregation.

Frequently asked questions

Is PHP good for web scraping? For static pages, yes — cURL and DomCrawler are as capable as Python's requests and Beautiful Soup, and if your application is already PHP, scraping in the same runtime saves an entire deployment target. Where PHP lags is the JavaScript-heavy end: Panther is the only mainstream option, versus Playwright, Puppeteer, and Selenium in Node and Python.

Should I use Goutte in 2026? No. It's marked abandoned on Packagist, its last release was April 2023, and the maintainer points at symfony/browser-kit. Any tutorial still recommending Goutte was written before that and hasn't been checked since — treat the rest of its advice with the same suspicion.

Is Simple HTML DOM still maintained? The original isn't — its newest release is 2.0-RC2 from November 2019. The voku/simple_html_dom fork is maintained and PHP 8 compatible, and it's the right target if you're keeping existing code alive. For new work, DomCrawler or Dom\HTMLDocument does everything it does without the memory-leak footguns.

What's the difference between DomCrawler and BrowserKit? DomCrawler parses HTML you already have. BrowserKit fetches it and keeps browser-like state — cookies, history, form submission, link clicking — returning a DomCrawler Crawler for each response. Use DomCrawler alone when you fetch with Guzzle or cURL; add BrowserKit when you need a session or have to submit forms.

Can PHP scrape JavaScript-rendered pages? Only by driving a real browser or calling one over the network. Symfony Panther over WebDriver is the pure-PHP route; the alternatives are a rendering API or shelling out to a Node script. There is no PHP-embeddable JS engine that will run a modern SPA.

Why does DomCrawler's filter() throw a LogicException? You're missing symfony/css-selector, which translates CSS to XPath. Run composer require symfony/css-selector. Until you do, filterXPath() still works — the XPath path never needed the bridge.

How do I fix "SSL certificate problem: unable to get local issuer certificate" in PHP? Your PHP install has no usable CA bundle. Download cacert.pem from curl.se, point curl.cainfo and openssl.cafile at it in php.ini, and restart PHP. Don't set CURLOPT_SSL_VERIFYPEER to false — that removes the check rather than fixing it.

Why does my XPath return the same value for every row? Your context-relative query starts with //, which searches from the document root regardless of the context node you passed to DOMXPath::query(). Use .// for relative queries.

Which PHP version should I be on for scraping? PHP 8.4 or 8.5. PHP 8.0 and 8.1 are past end of life, and 8.2 and 8.3 are in security-only support. 8.4 also brings Dom\HTMLDocument and its spec-compliant HTML5 parser, which is a real upgrade over DOMDocument::loadHTML() for anything scraping-related.