---
title: "API Documentation"
description: "Complete API reference for WebScraping.AI. Learn how to scrape websites, extract data with AI, and integrate with your applications."
url: https://webscraping.ai/docs
markdown_index: https://webscraping.ai/llms.txt
---
# API Documentation

Everything you need to integrate WebScraping.AI into your applications.

Updated July 20, 2026

[Quick Start](https://webscraping.ai/docs/quick-start.md)[OpenAPI Spec](https://webscraping.ai/openapi.yml)[Markdown](https://webscraping.ai/docs.md "These docs as Markdown, for AI assistants and coding agents")

## Quick Start

Get started with WebScraping.AI in under 5 minutes. Here's a simple example to extract data from any webpage.

1

##### Get your API key

Sign up at [webscraping.ai](https://webscraping.ai/auth/sign_up) to get your free API key with 2,000 credits.

2

##### Make your first request

Try the AI question endpoint to ask a question about any webpage:

**cURL**

```bash
curl -G "https://api.webscraping.ai/ai/question" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "question=What is this page about?"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
answer = client.question("https://example.com", question="What is this page about?")
print(answer)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const answer = await client.question({
  url: 'https://example.com',
  question: 'What is this page about?',
});
console.log(answer);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$answer = $client->question('https://example.com', 'What is this page about?');
echo $answer;
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
answer = client.question('https://example.com', question: 'What is this page about?')
puts answer
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    answer, _ := client.Question(context.Background(), &webscrapingai.QuestionOptions{
        URL: "https://example.com",
        Question: "What is this page about?",
    })
    fmt.Println(answer)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.QuestionOptions;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
String answer = client.question(QuestionOptions.builder()
    .url("https://example.com")
    .question("What is this page about?")
    .build());
System.out.println(answer);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var answer = await client.QuestionAsync(new QuestionRequest {
    Url = "https://example.com",
    Question = "What is this page about?",
});
Console.WriteLine(answer);
```

3

##### Get your response

The API returns the AI-generated answer as plain text:

```text
This page is the example domain maintained by IANA for illustrative purposes in documents and tutorials.
```

**That's it!** You've made your first API call. Requests cost from 1 credit depending on JS rendering and proxy type (AI extraction adds +5) — see [Rules & Pricing](https://webscraping.ai/docs/rules.md). Failed requests are free.

## Authentication

All API requests require an API key. Pass your key as a query parameter:

**HTTP**

```http
GET https://api.webscraping.ai/html?url=https://example.com&api_key=YOUR_API_KEY
```

**Keep your API key secure!** Don't expose it in client-side code. Use environment variables or a backend proxy for production applications.

## Base URL

All API endpoints use this base URL:

 https://api.webscraping.ai

## Rules & Pricing

- Requests start at 1 credit (no JS, datacenter proxy). JS rendering, residential and stealth proxies cost more, and AI extraction adds +5 credits. Google search results (`/serp`) are a flat 15 credits per request and structured data (`/data`) is 15 credits per request (50 for Reddit) — see the table below.
- Requests can take up to 30 seconds. Use the `timeout` parameter to control duration.
- Failed requests are free. You're only charged for successful responses. For `/data`, pages that come back `parse_failed` or `not_found` count as successful and are charged; only failed fetches are free.
- Credits come from a monthly subscription plan, pay-as-you-go top-ups (from $20, $1 = 5,000 credits, valid 12 months), or both — pay-as-you-go credits are used after your plan quota runs out. See [pricing](https://webscraping.ai/#pricing).

##### Credit Cost Summary

| Configuration | Credits | Notes |
| --- | --- | --- |
| Basic (no JS, datacenter proxy) | **1** | Fastest, for static sites |
| With JS rendering | **5** | Default setting, headless Chrome |
| Residential proxy (no JS) | **10** | For anti-bot protected sites |
| Residential proxy + JS | **25** | Maximum compatibility |
| Stealth proxy | **50** | For sites with the strongest anti-bot protection |
| AI endpoints (/ai/question, /ai/fields) | **+5** | Added on top of the proxy/JS cost above |
| Search results ([/serp](https://webscraping.ai/docs/serp.md)) | **15** | Flat per search; proxy/JS settings don't apply |
| Structured data ([/data](https://webscraping.ai/docs/data.md)) | **15** (Reddit: **50** ) | Per page, including pages that parse empty or don't exist; proxy/JS settings don't apply |

All costs are per successful request. Failed requests are free.

## Success Rates & Retrying

Most failed requests succeed on a retry or with a different configuration. If you encounter failures:

- **Retry the request** - Many failures are temporary due to network issues or website load
- **Increase timeout** - Set `timeout` to 20000-25000ms for slow-loading websites
- **Use residential proxies** - Set `proxy=residential` if datacenter proxies are blocked
- **Adjust js\_timeout** - Increase `js_timeout` for pages with slow-loading dynamic content

## Best Practices

**JavaScript Rendering**

JS rendering is enabled by default (`js=true`) using headless Chrome. Keep enabled for SPAs (React, Vue, Angular), AJAX content, and dynamic pages. Disable (`js=false`) for static sites, faster responses, or server-rendered content.

**Proxy Strategy**

Start with datacenter proxies (default) for speed and cost. Switch to residential proxies if: website blocks datacenter IPs, getting 403 errors, need to bypass anti-bot protection, or scraping geo-restricted content.

## Sending Cookies

Send cookies to the target website using the `headers` parameter. Cookies should be formatted as a standard Cookie header value with semicolon-separated key-value pairs.

##### Cookie Format

**Format**

```text
"Cookie": "key1=value1; key2=value2; key3=value3"
```

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/dashboard" \
  --data-urlencode 'headers[Cookie]=session_id=abc123; user_pref=dark_mode; lang=en'
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
html = client.html(
    "https://example.com/dashboard",
    headers={"Cookie": "session_id=abc123; user_pref=dark_mode; lang=en"},
)
print(html)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const html = await client.html({
  url: 'https://example.com/dashboard',
  headers: { Cookie: 'session_id=abc123; user_pref=dark_mode; lang=en' },
});
console.log(html);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$html = $client->html(
    'https://example.com/dashboard',
    headers: ['Cookie' => 'session_id=abc123; user_pref=dark_mode; lang=en'],
);
echo $html;
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
html = client.html(
  'https://example.com/dashboard',
  headers: { 'Cookie' => 'session_id=abc123; user_pref=dark_mode; lang=en' }
)
puts html
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    html, _ := client.HTML(context.Background(), &webscrapingai.HTMLOptions{
        URL: "https://example.com/dashboard",
        Headers: map[string]string{
            "Cookie": "session_id=abc123; user_pref=dark_mode; lang=en",
        },
    })
    fmt.Println(html)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.HtmlOptions;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
String html = client.html(HtmlOptions.builder()
    .url("https://example.com/dashboard")
    .addHeader("Cookie", "session_id=abc123; user_pref=dark_mode; lang=en")
    .build());
System.out.println(html);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var html = await client.HtmlAsync(new HtmlRequest {
    Url = "https://example.com/dashboard",
    Headers = new Dictionary<string, string> {
        ["Cookie"] = "session_id=abc123; user_pref=dark_mode; lang=en",
    },
});
Console.WriteLine(html);
```

**Common use cases:** Session authentication, user preferences, A/B testing variants, geo-location settings, and accessing logged-in content.

## Official SDKs

Use our official SDKs for easier integration in your preferred language.

[Python](https://github.com/webscraping-ai/webscraping-ai-python)

`pip install webscraping_ai`

[JavaScript](https://github.com/webscraping-ai/webscraping-ai-js)

`npm install webscraping-ai`

[PHP](https://github.com/webscraping-ai/webscraping-ai-php)

`composer require webscraping-ai/webscraping-ai-php`

[Ruby](https://github.com/webscraping-ai/webscraping-ai-ruby)

`gem install webscraping_ai`

[Go](https://github.com/webscraping-ai/webscraping-ai-go)

`go get github.com/webscraping-ai/webscraping-ai-go/v4`

[Java](https://github.com/webscraping-ai/webscraping-ai-java)

`ai.webscraping:webscraping-ai`

[C# / .NET](https://github.com/webscraping-ai/webscraping-ai-dotnet)

`dotnet add package WebScrapingAI`

[MCP Server](https://webscraping.ai/integrations/mcp-server)

`https://mcp.webscraping.ai/mcp`

[CLI](https://github.com/webscraping-ai/webscraping-ai-cli)

`npm install -g webscraping-ai-cli`

[n8n Node](https://github.com/webscraping-ai/webscraping-ai-n8n)

`n8n-nodes-webscraping-ai`

Need an SDK for another language? [Let us know!](mailto:support@webscraping.ai)

## Using with AI Tools

Integrate WebScraping.AI with AI assistants and LLM platforms.

##### MCP Server

Our hosted [MCP server](https://webscraping.ai/integrations/mcp-server) integrates WebScraping.AI directly with AI assistants that support the Model Context Protocol — Claude, Claude Desktop, Claude Code, Cursor, Codex, Windsurf, and any MCP-compatible platform. Add the URL to your client and sign in with your WebScraping.AI account; no API key needed:

```bash
# Remote MCP server URL (Streamable HTTP, OAuth login)
https://mcp.webscraping.ai/mcp

# Example: Claude Code
claude mcp add --transport http webscraping-ai https://mcp.webscraping.ai/mcp
```

Prefer a self-hosted or customizable server? The previous [open-source npm version](https://github.com/webscraping-ai/webscraping-ai-mcp-server) runs locally over stdio with your API key (`npx -y webscraping-ai-mcp`) and exposes the same 9 tools, including `webscraping_ai_serp` for Google search results and `webscraping_ai_data` for structured data from supported sites. See the [setup guide](https://webscraping.ai/integrations/mcp-server) for both options.

##### CLI with AI Skill

Our [command-line tool](https://github.com/webscraping-ai/webscraping-ai-cli) wraps every endpoint for use in scripts and terminals, and ships an AI agent skill your coding assistant can install:

```bash
# Install the CLI
npm install -g webscraping-ai-cli

# Google search results for a query (alias: search)
webscraping-ai serp coffee machines --gl us --hl en --page 1

# Structured data for a page on a supported site
webscraping-ai data 'https://www.youtube.com/watch?v=dQw4w9WgXcQ'

# Install the agent skill into your editor(s)
webscraping-ai setup skill --all
```

The skill teaches AI coding assistants — Claude Code, Cursor, Windsurf, Kiro, OpenCode, Gemini CLI, GitHub Copilot, Augment, and Factory — when and how to fetch live page content through the CLI.

##### OpenAPI Specification

Use our [OpenAPI specification](https://webscraping.ai/openapi.yml) (also available as [JSON](https://webscraping.ai/openapi.json)) to integrate with AI tools that support API schemas, such as GPT Actions or custom agents.

##### Docs for AI Agents

Point your coding agent or AI assistant at these instead of the HTML pages:

- [llms.txt](https://webscraping.ai/llms.txt): an index of the whole site in Markdown, and [llms-full.txt](https://webscraping.ai/llms-full.txt): these docs and the main product pages in one file
- Every page as Markdown: add `.md` to its path ([/docs.md](https://webscraping.ai/docs.md)) or request it with `Accept: text/markdown`. Each docs section has its own file, e.g. [/docs/ai-fields.md](https://webscraping.ai/docs/ai-fields.md).
- An [agent skill for the HTTP API](https://webscraping.ai/.well-known/agent-skills/webscraping-ai-api/SKILL.md), listed in [/.well-known/agent-skills/index.json](https://webscraping.ai/.well-known/agent-skills/index.json)

## Proxy Mode

Use WebScraping.AI as a proxy server for your existing tools. Route requests through our infrastructure without changing your code.

##### Proxy Settings

| Host | proxy.webscraping.ai |
| --- | --- |
| Port | 8888 |
| Username | Your API key |
| Password | Parameters (e.g., js=true&proxy=residential) |

##### Plug into your existing scraping tools

Proxy Mode is designed for tools you already use — HTTP clients, headless browsers, scraping frameworks. The snippets below show the most common integrations. For a more idiomatic, code-first integration, see our [official SDKs](https://webscraping.ai/docs/sdks.md) instead.

**cURL**

```bash
# https://curl.se/docs/manpage.html#-x
curl -x "http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888" \
     -k "https://example.com"
```

**Python (requests)**

```python
# pip install requests
# https://pypi.org/project/requests/
import requests

proxy_url = "http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888"

response = requests.get(
    "https://example.com",
    proxies={"http": proxy_url, "https": proxy_url},
    verify=False, # proxy uses a self-signed cert
)
print(response.text)
```

**Scrapy**

```python
# pip install scrapy
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpproxy
import scrapy

PROXY = "http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888"

class ExampleSpider(scrapy.Spider):
    name = "example"
    # Scrapy doesn't verify TLS certificates by default,
    # so the proxy's self-signed cert needs no extra setting

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={"proxy": PROXY},
            callback=self.parse,
        )

    def parse(self, response):
        yield {"title": response.css("title::text").get()}
```

**Selenium**

```python
# pip install selenium-wire "blinker<1.8"
# https://pypi.org/project/selenium-wire/ (vanilla Selenium can't pass proxy credentials;
# selenium-wire is archived and breaks with blinker 1.8+, hence the pin)
from seleniumwire import webdriver

PROXY = "http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888"

seleniumwire_options = {
    "proxy": {"http": PROXY, "https": PROXY, "no_proxy": "localhost,127.0.0.1"},
    "verify_ssl": False,
}

driver = webdriver.Chrome(seleniumwire_options=seleniumwire_options)
driver.get("https://example.com")
print(driver.page_source)
driver.quit()
```

**Playwright**

```javascript
// npm install playwright
// https://playwright.dev/docs/network#http-proxy
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({
    proxy: {
      server: 'http://proxy.webscraping.ai:8888',
      username: 'YOUR_API_KEY',
      password: 'js=true&proxy=residential',
    },
  });
  const context = await browser.newContext({ ignoreHTTPSErrors: true });
  const page = await context.newPage();
  await page.goto('https://example.com');
  console.log(await page.content());
  await browser.close();
})();
```

**Puppeteer**

```javascript
// npm install puppeteer
// https://pptr.dev/api/puppeteer.page.authenticate
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    args: [
      '--proxy-server=proxy.webscraping.ai:8888',
      '--ignore-certificate-errors',
    ],
  });
  const page = await browser.newPage();
  await page.authenticate({
    username: 'YOUR_API_KEY',
    password: 'js=true&proxy=residential',
  });
  await page.goto('https://example.com');
  console.log(await page.content());
  await browser.close();
})();
```

**Note:** Proxy mode uses a self-signed SSL certificate. Tell your client to accept it: `-k` in cURL, `verify=False` in requests, `verify_ssl: False` in selenium-wire, `ignoreHTTPSErrors: true` in Playwright, `--ignore-certificate-errors` in Puppeteer (Scrapy doesn't verify certificates by default).

* * *

### AI Endpoints

## Ask Questions About a Page

Use AI to answer questions about any webpage. Perfect for extracting specific information without parsing HTML.

`GET /ai/question`

##### Parameters [→ All parameters reference](https://webscraping.ai/docs/parameters.md)

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of the webpage to analyze |
| question required | string | Question to ask about the page content |
| api\_key required | string | Your API key |

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/ai/question" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://news.ycombinator.com" \
  --data-urlencode "question=What are the top 3 stories on this page?"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
answer = client.question(
    "https://news.ycombinator.com",
    question="What are the top 3 stories on this page?",
)
print(answer)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const answer = await client.question({
  url: 'https://news.ycombinator.com',
  question: 'What are the top 3 stories on this page?',
});
console.log(answer);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$answer = $client->question(
    'https://news.ycombinator.com',
    'What are the top 3 stories on this page?',
);
echo $answer;
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
answer = client.question(
  'https://news.ycombinator.com',
  question: 'What are the top 3 stories on this page?'
)
puts answer
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    answer, _ := client.Question(context.Background(), &webscrapingai.QuestionOptions{
        URL: "https://news.ycombinator.com",
        Question: "What are the top 3 stories on this page?",
    })
    fmt.Println(answer)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.QuestionOptions;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
String answer = client.question(QuestionOptions.builder()
    .url("https://news.ycombinator.com")
    .question("What are the top 3 stories on this page?")
    .build());
System.out.println(answer);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var answer = await client.QuestionAsync(new QuestionRequest {
    Url = "https://news.ycombinator.com",
    Question = "What are the top 3 stories on this page?",
});
Console.WriteLine(answer);
```

##### Response

```text
The top 3 stories on Hacker News are:
1. "Show HN: I built an AI-powered code review tool"
2. "The future of web development"
3. "Why Rust is taking over systems programming"
```

## Extract Structured Fields

Extract specific data fields from any webpage as structured JSON. Ideal for scraping product details, articles, profiles, and more.

`GET /ai/fields`

##### Parameters [→ All parameters reference](https://webscraping.ai/docs/parameters.md)

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of the webpage to extract from |
| fields required | object | Object with field names as keys and extraction instructions as values |
| api\_key required | string | Your API key |

##### Example - Extract Product Data

**cURL**

```bash
curl -G "https://api.webscraping.ai/ai/fields" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://amazon.com/dp/B08N5WRWNW" \
  --data-urlencode "fields[title]=Product title" \
  --data-urlencode "fields[price]=Current price with currency" \
  --data-urlencode "fields[rating]=Average star rating"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
result = client.fields(
    "https://amazon.com/dp/B08N5WRWNW",
    fields={
        "title": "Product title",
        "price": "Current price with currency",
        "rating": "Average star rating",
    },
)
print(result)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const result = await client.fields({
  url: 'https://amazon.com/dp/B08N5WRWNW',
  fields: {
    title: 'Product title',
    price: 'Current price with currency',
    rating: 'Average star rating',
  },
});
console.log(result);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$result = $client->fields('https://amazon.com/dp/B08N5WRWNW', [
    'title' => 'Product title',
    'price' => 'Current price with currency',
    'rating' => 'Average star rating',
]);
print_r($result);
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
result = client.fields(
  'https://amazon.com/dp/B08N5WRWNW',
  fields: {
    title: 'Product title',
    price: 'Current price with currency',
    rating: 'Average star rating'
  }
)
puts result.inspect
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    result, _ := client.Fields(context.Background(), &webscrapingai.FieldsOptions{
        URL: "https://amazon.com/dp/B08N5WRWNW",
        Fields: map[string]string{
            "title": "Product title",
            "price": "Current price with currency",
            "rating": "Average star rating",
        },
    })
    fmt.Println(result.Result)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.FieldsOptions;
import ai.webscraping.result.FieldsResult;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
FieldsResult result = client.fields(FieldsOptions.builder()
    .url("https://amazon.com/dp/B08N5WRWNW")
    .addField("title", "Product title")
    .addField("price", "Current price with currency")
    .addField("rating", "Average star rating")
    .build());
System.out.println(result.getResult());
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var result = await client.FieldsAsync(new FieldsRequest {
    Url = "https://amazon.com/dp/B08N5WRWNW",
    Fields = new Dictionary<string, string> {
        ["title"] = "Product title",
        ["price"] = "Current price with currency",
        ["rating"] = "Average star rating",
    },
});
Console.WriteLine(result.Result);
```

##### Response

The extracted fields come back under a `result` key; a field the page doesn't contain is `null`.

```json
{
  "result": {
    "title": "Apple AirPods Pro (2nd Generation)",
    "price": "$249.00",
    "rating": "4.7 out of 5 stars"
  }
}
```

**Pro tip:** Be specific with your field descriptions. Instead of "price", use "Current sale price including currency symbol" for better accuracy.

* * *

### Scraping Endpoints

## Get Page HTML

Fetch the full HTML content of any webpage. Includes JavaScript rendering via headless Chrome and automatic proxy rotation.

`GET /html`

##### Parameters [→ All parameters reference](https://webscraping.ai/docs/parameters.md)

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of the webpage to fetch |
| api\_key required | string | Your API key |
| js optional | boolean | Enable JavaScript rendering (default: true) |
| proxy optional | string | Proxy type: "datacenter" (default), "residential", "stealth" or "auto" |

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "js=true"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
html = client.html("https://example.com", js=True)
print(html)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const html = await client.html({ url: 'https://example.com', js: true });
console.log(html);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$html = $client->html('https://example.com', js: true);
echo $html;
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
html = client.html('https://example.com', js: true)
puts html
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    js := true
    html, _ := client.HTML(context.Background(), &webscrapingai.HTMLOptions{
        URL: "https://example.com",
        CommonOptions: webscrapingai.CommonOptions{JS: &js},
    })
    fmt.Println(html)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.HtmlOptions;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
String html = client.html(HtmlOptions.builder()
    .url("https://example.com")
    .js(true)
    .build());
System.out.println(html);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var html = await client.HtmlAsync(new HtmlRequest {
    Url = "https://example.com",
    Js = true,
});
Console.WriteLine(html);
```

`POST /html`

Use POST requests to send data to the target page (e.g., form submissions, API calls). The `/text`, `/selected` and `/selected-multiple` endpoints accept POST the same way.

##### How POST works

| query string | params | `api_key`, `url` and all other parameters go in the query string, as with GET |
| --- | --- | --- |
| request body | raw | The body of your POST request is sent to the target URL, with your request's `Content-Type`. JSON and `text/plain` bodies pass through as-is; to send a form-encoded body, post it as `text/plain` and set `headers[Content-Type]=application/x-www-form-urlencoded` |

##### POST Example

**cURL**

```bash
# Send a JSON body to a target page
curl -X POST "https://api.webscraping.ai/html?api_key=YOUR_API_KEY&url=https%3A%2F%2Fhttpbin.org%2Fpost" \
  -H "Content-Type: application/json" \
  -d '{"username": "test", "password": "demo"}'
```

**Python**

```python
import requests

# Send a JSON body to a target page
response = requests.post(
    "https://api.webscraping.ai/html",
    params={"api_key": "YOUR_API_KEY", "url": "https://httpbin.org/post"},
    json={"username": "test", "password": "demo"},
)
print(response.text)
```

**JavaScript**

```javascript
// Send a JSON body to a target page
const query = new URLSearchParams({
  api_key: 'YOUR_API_KEY',
  url: 'https://httpbin.org/post',
});
const response = await fetch(`https://api.webscraping.ai/html?${query}`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ username: 'test', password: 'demo' }),
});
console.log(await response.text());
```

**Use cases:** Login to websites, submit search forms, interact with APIs that require POST requests, or scrape pages that need form submissions.

## Get Page Text

Convert a webpage to clean Markdown — boilerplate stripped, headings, lists, and links preserved (table cells come through as plain text). Perfect for feeding content to LLMs and RAG pipelines. [Try it on any URL](https://webscraping.ai/tools/html-to-markdown).

`GET /text`

##### Parameters [→ All parameters reference](https://webscraping.ai/docs/parameters.md)

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of the webpage |
| text\_format optional | string | "plain" (default) returns raw Markdown; "json"/"xml" wrap it with title and description |
| return\_links optional | boolean | Include links in JSON response (default: false) |

##### Example with JSON format

**cURL**

```bash
curl -G "https://api.webscraping.ai/text" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "text_format=json"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
result = client.text("https://example.com", text_format="json")
print(result)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const result = await client.text({
  url: 'https://example.com',
  text_format: 'json',
});
console.log(result);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$result = $client->text('https://example.com', textFormat: 'json');
print_r($result);
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
result = client.text('https://example.com', text_format: 'json')
puts result.inspect
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    text, _ := client.Text(context.Background(), &webscrapingai.TextOptions{
        URL: "https://example.com",
        TextFormat: "json",
    })
    fmt.Println(text)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.TextOptions;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
String text = client.text(TextOptions.builder()
    .url("https://example.com")
    .textFormat("json")
    .build());
System.out.println(text);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var text = await client.TextAsync(new TextRequest {
    Url = "https://example.com",
    TextFormat = "json",
});
Console.WriteLine(text);
```

##### Response

```json
{
  "title": "Example Domain",
  "description": "This domain is for use in illustrative examples...",
  "content": "Example Domain\n\nThis domain is for use in illustrative examples in documents..."
}
```

## Get Selected HTML

Extract HTML from specific page elements using CSS selectors. Useful when you only need a portion of the page.

`GET /selected`

##### Parameters [→ All parameters reference](https://webscraping.ai/docs/parameters.md)

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of the webpage |
| selector required | string | CSS selector (e.g., "h1", ".price", "#main") |

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/selected" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "selector=h1"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
html = client.selected("https://example.com", selector="h1")
print(html)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const html = await client.selected({
  url: 'https://example.com',
  selector: 'h1',
});
console.log(html);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$html = $client->selected('https://example.com', 'h1');
echo $html;
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
html = client.selected('https://example.com', selector: 'h1')
puts html
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    html, _ := client.Selected(context.Background(), &webscrapingai.SelectedOptions{
        URL: "https://example.com",
        Selector: "h1",
    })
    fmt.Println(html)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.SelectedOptions;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
String html = client.selected(SelectedOptions.builder()
    .url("https://example.com")
    .selector("h1")
    .build());
System.out.println(html);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var html = await client.SelectedAsync(new SelectedRequest {
    Url = "https://example.com",
    Selector = "h1",
});
Console.WriteLine(html);
```

##### Response

```text
<h1>Example Domain</h1>
```

**Multiple selectors?** Use the [`/selected-multiple`](https://webscraping.ai/docs/selected-multiple.md) endpoint to extract several page areas in one request.

## Get Multiple Selected Areas

Extract several page areas in one request: pass one CSS selector per area and get back, for each selector, the HTML of every element it matches. Costs the same as a single `/selected` request.

`GET /selected-multiple`

##### Parameters [→ All parameters reference](https://webscraping.ai/docs/parameters.md)

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of the webpage |
| selectors required | string[] | CSS selectors. Repeat the parameter once per selector (`?selectors=h1&selectors=p`), not `selectors[]=`, which is ignored. |

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/selected-multiple" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "selectors=h1" \
  --data-urlencode "selectors=p"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
areas = client.selected_multiple("https://example.com", selectors=["h1", "p"])
print(areas)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const areas = await client.selectedMultiple({
  url: 'https://example.com',
  selectors: ['h1', 'p'],
});
console.log(areas);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$areas = $client->selectedMultiple('https://example.com', ['h1', 'p']);
print_r($areas);
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
areas = client.selected_multiple('https://example.com', selectors: ['h1', 'p'])
p areas
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    areas, _ := client.SelectedMultiple(context.Background(), &webscrapingai.SelectedMultipleOptions{
        URL: "https://example.com",
        Selectors: []string{"h1", "p"},
    })
    fmt.Println(areas)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.SelectedMultipleOptions;
import ai.webscraping.result.SelectedMultipleResult;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
SelectedMultipleResult areas = client.selectedMultiple(SelectedMultipleOptions.builder()
    .url("https://example.com")
    .selectors("h1", "p")
    .build());
System.out.println(areas.getResults());
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var areas = await client.SelectedMultipleAsync(new SelectedMultipleRequest {
    Url = "https://example.com",
    Selectors = new[] { "h1", "p" },
});
foreach (var matches in areas.Results) Console.WriteLine(string.Join(" | ", matches));
```

##### Response

One array per selector, in request order, holding the inner HTML of every element that matches it. A selector that matches nothing returns an empty array.

```json
[
  ["Example Domain"],
  ["This domain is for use in documentation examples without needing permission. Avoid use in operations.", "<a href=\"https://iana.org/domains/example\">Learn more</a>"]
]
```

* * *

### Search & Data Endpoints

## Search Results (SERP)

Get parsed Google search results for a query as JSON: organic results (position, title, link, domain, displayed link, snippet, date), related searches, spelling-correction info, and pagination. Google is the only engine today.

Unlike the scraping endpoints, `/serp` is query-shaped, not URL-shaped: you pass the search query in `q` instead of a `url`, and WebScraping.AI handles proxy routing and parsing. The page-scraping parameters (`js`, `proxy`, `country`, `headers`, `timeout`, …) don't apply. Each successful search costs a flat **15 credits** ; failed searches are not charged.

`GET /serp`

##### Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| q required | string | Search query, e.g. `coffee machines` |
| api\_key required | string | Your API key |
| engine | string | Search engine to query. Only `google` (the default) is supported |
| gl | string | Two-letter country code for the search location (Google `gl`). Default: `us` |
| hl | string | Two-letter language code for the results (Google `hl`). Default: `en` |
| page | integer | Results page number, 10 results per page. A whole number from 1 to 100. Default: `1`. Any other value (0, a fraction, a non-number, or above 100) is rejected with a `400` before billing |

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/serp" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "q=coffee machines" \
  --data-urlencode "gl=us" \
  --data-urlencode "hl=en" \
  --data-urlencode "page=1"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
results = client.serp("coffee machines", gl="us", hl="en", page=1)
for r in results["organic_results"]:
    print(r["position"], r["title"], r["link"])
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const serp = await client.serp({ q: 'coffee machines', gl: 'us', hl: 'en', page: 1 });
for (const r of serp.organic_results) {
  console.log(`${r.position}. ${r.title} — ${r.link}`);
}
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$serp = $client->serp(q: 'coffee machines', gl: 'us', hl: 'en', page: 1);
foreach ($serp['organic_results'] as $result) {
    printf("%d. %s — %s\n", $result['position'], $result['title'], $result['link']);
}
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
results = client.serp(q: 'coffee machines', gl: 'us', hl: 'en', page: 1)
results['organic_results'].each do |r|
  puts "#{r['position']}. #{r['title']} — #{r['link']}"
end
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    page := 1
    serp, _ := client.Serp(context.Background(), &webscrapingai.SerpOptions{
        Q: "coffee machines",
        GL: "us",
        HL: "en",
        Page: &page,
    })
    for _, r := range serp.OrganicResults {
        fmt.Println(r.Position, r.Title, r.Link)
    }
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.SerpOptions;
import ai.webscraping.result.SerpResult;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
SerpResult serp = client.serp(SerpOptions.builder()
    .q("coffee machines")
    .gl("us")
    .hl("en")
    .page(1)
    .build());
for (SerpResult.OrganicResult r : serp.getOrganicResults()) {
    System.out.println(r.getPosition() + ". " + r.getTitle() + " — " + r.getLink());
}
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var serp = await client.SerpAsync(new SerpRequest {
    Q = "coffee machines",
    Gl = "us",
    Hl = "en",
    Page = 1,
});
foreach (var r in serp.OrganicResults) {
    Console.WriteLine($"{r.Position}. {r.Title} — {r.Link}");
}
```

##### Response

```json
{
  "search_parameters": {
    "engine": "google",
    "q": "coffee machines",
    "gl": "us",
    "hl": "en",
    "page": 1
  },
  "search_information": {
    "query_displayed": "coffee machines",
    "organic_results_state": "Results for exact spelling"
  },
  "organic_results": [
    {
      "position": 1,
      "title": "Best Coffee Machines of 2026",
      "link": "https://www.example.com/best-coffee-machines",
      "domain": "example.com",
      "displayed_link": "www.example.com › Reviews › Coffee Machines",
      "snippet": "We tested 20 coffee machines to find the best ones for every budget...",
      "date": "Apr 13, 2026"
    },
    {
      "position": 2,
      "title": "Coffee Machines | Example Store",
      "link": "https://shop.example.org/coffee-machines",
      "domain": "shop.example.org",
      "displayed_link": "shop.example.org › coffee-machines"
    }
  ],
  "related_searches": [
    { "query": "best espresso machine" }
  ],
  "pagination": {
    "current": 1,
    "next": 2
  }
}
```

Trimmed to two results; a full page has up to 10. `snippet`, `date`, `related_searches` and `search_information.showing_results_for` (the auto-corrected query) appear only when Google shows them, and `pagination.next` is absent on the last page. People Also Ask, ads, knowledge graph and local results are not included.

**Paging:** `position` restarts at 1 on every page. For an absolute rank, compute `(page - 1) * 10 + position`. A search that genuinely returns no results (`organic_results_state` is `Fully empty`) is a successful, billed search; only failed searches are free.

* * *

## Structured Data

Get structured JSON for a public page on a supported site from its normal URL: for example a YouTube video, channel or playlist, a TikTok video or profile, an X post or profile, a LinkedIn company, job or profile, an Instagram post, reel or profile, or a Reddit post, subreddit or user. The site and page type are detected from the URL, and WebScraping.AI handles fetching, proxies and parsing.

More sites are added over time, so don't validate URLs on your side: an unsupported URL or page type returns a `400` that is not charged, whose message lists what is supported. For other sites, use [`/ai/fields`](https://webscraping.ai/docs/ai-fields.md). The page-scraping parameters (`js`, `proxy`, `headers`, `timeout`, …) don't apply. Each request costs **15 credits** ( **50** for Reddit, whose pages are loaded in a real browser); requests that fail to fetch are not charged, while pages that parse empty or don't exist are.

**Site guides** (supported page types, fields and limits per site): [Social Media Scraper API](https://webscraping.ai/social-media-scraper-api) (overview of all sites), [YouTube Scraper API](https://webscraping.ai/youtube-scraper-api), [TikTok Scraper API](https://webscraping.ai/tiktok-scraper-api), [Twitter (X) Scraper API](https://webscraping.ai/twitter-scraper-api), [LinkedIn Scraper API](https://webscraping.ai/linkedin-scraper-api), [Instagram Scraper API](https://webscraping.ai/instagram-scraper-api), [Reddit Scraper API](https://webscraping.ai/reddit-scraper-api).

`GET /data`

##### Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| url required | string | URL of a page on a supported site, e.g. `https://www.youtube.com/watch?v=dQw4w9WgXcQ` |
| api\_key required | string | Your API key |
| country | string | Country of the proxy used to fetch the page, one of the [proxy countries](https://webscraping.ai/docs#param-country). Default: `us` |
| transcript | boolean | YouTube videos only. Also fetch the transcript into `data.transcript` (null when no matching captions are available). If the transcript fetch fails, the whole request fails with a 500 and is not charged |
| transcript\_language | string | YouTube videos only, with `transcript=true`. Caption language to pick, e.g. `de`. Without it, English is preferred, then the first available track |

##### Example

**cURL**

```bash
curl -G "https://api.webscraping.ai/data" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://www.youtube.com/watch?v=dQw4w9WgXcQ"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
result = client.data("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(result["request_parameters"]["provider"], result["parse_status"])
if result["data"]:
    print(result["data"]["title"])
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const result = await client.data({ url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' });
console.log(result.request_parameters.provider, result.parse_status);
console.log(result.data?.title);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$result = $client->data(url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ');
echo $result['request_parameters']['provider'], ' ', $result['parse_status'], "\n";
echo $result['data']['title'] ?? '', "\n";
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
result = client.data('https://www.youtube.com/watch?v=dQw4w9WgXcQ')
puts "#{result['request_parameters']['provider']} #{result['parse_status']}"
puts result['data']&.dig('title')
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "encoding/json"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    res, _ := client.Data(context.Background(), &webscrapingai.DataOptions{
        URL: "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    })
    fmt.Println(res.RequestParameters.Provider, res.ParseStatus)

    // Data is raw JSON whose shape depends on the provider and page type
    var video struct {
        Title string `json:"title"`
    }
    if res.Data != nil {
        json.Unmarshal(res.Data, &video)
    }
    fmt.Println(video.Title)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.DataOptions;
import ai.webscraping.result.DataResult;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
DataResult result = client.data(DataOptions.builder()
    .url("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
    .build());
System.out.println(result.getRequestParameters().getProvider() + " " + result.getParseStatus());
if (result.getData() != null) {
    System.out.println(result.getData().path("title").asText());
}
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var result = await client.DataAsync(new DataRequest {
    Url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
});
Console.WriteLine($"{result.RequestParameters.Provider} {result.ParseStatus}");
if (result.Data is { } data) {
    Console.WriteLine(data.GetProperty("title").GetString());
}
```

##### Response

```json
{
  "request_parameters": {
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "provider": "youtube",
    "type": "video"
  },
  "parse_status": "ok",
  "data": {
    "video_id": "dQw4w9WgXcQ",
    "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
    "link": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    ...
  }
}
```

Trimmed. The fields in `data` depend on `provider` and `type`; they use snake\_case, and fields a page doesn't expose are null (some flags come back `false`, and lists come back empty).

**parse\_status:** `ok` when the page was parsed, `parse_failed` when it was fetched but couldn't be parsed (`data` may be null or partial), `not_found` when the page doesn't exist. All three are successful, billed requests; only failed fetches are free.

* * *

## Account Information

Check your remaining API credits and account status.

`GET /account`

##### Example

**cURL**

```bash
curl "https://api.webscraping.ai/account?api_key=YOUR_API_KEY"
```

**Python**

```python
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")
info = client.account()
print(info)
```

**JavaScript**

```javascript
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';

const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const info = await client.account();
console.log(info);
```

**PHP**

```php
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';

use WebScrapingAI\Client;

$client = new Client('YOUR_API_KEY');
$info = $client->account();
print_r($info);
```

**Ruby**

```ruby
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'

client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
info = client.account
puts info.inspect
```

**Go**

```go
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main

import (
    "context"
    "fmt"

    webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)

func main() {
    client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
    info, _ := client.Account(context.Background())
    fmt.Printf("%s - %d calls left\n", info.Email, info.RemainingAPICalls)
}
```

**Java**

```java
// Maven: ai.webscraping:webscraping-ai:4.2.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.result.AccountInfo;

Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
AccountInfo info = client.account();
System.out.println(info);
```

**C#**

```csharp
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;

var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var info = await client.AccountAsync();
Console.WriteLine($"{info.Email} - {info.RemainingApiCalls} calls left");
```

##### Response

```json
{
  "email": "user@example.com",
  "remaining_api_calls": 1850,
  "remaining_monthly_credits": 1850,
  "remaining_payg_credits": 100000,
  "remaining_total_credits": 101850,
  "resets_at": 1704067200,
  "remaining_concurrency": 10
}
```

## Error Codes

Understanding API error responses and how to handle them.

| Code | Description | Solution |
| --- | --- | --- |
| 400 | Invalid parameters | Check parameter values and format |
| 402 | Insufficient credits | Upgrade your plan, top up pay-as-you-go credits, or wait for credit reset |
| 403 | Invalid API key | Verify your API key is correct |
| 429 | Too many concurrent requests | Reduce request rate or upgrade for higher concurrency |
| 500 | Target page could not be scraped (blocked, anti-bot challenge, non-2xx status, DNS failure), or an unexpected error (`error_code` `internal_error`) | Read `error_code` and follow `next_step` (see below); retry an `internal_error` and contact support if it persists |
| 504 | Request timeout | Increase timeout parameter value |

**Good news:** Failed requests don't count against your credit quota. You only pay for successful responses.

### Target errors are structured

When the target website blocks the request or returns an error, the `500` response body tells you what happened and what to try next, with the real credit cost of that retry. Example for a site that blocks datacenter IPs:

```
{
  "message": "The target website returned HTTP 403 (Forbidden) through a datacenter proxy: it is blocking this IP pool (anti-bot protection). Retry with proxy=residential (10 credits per request). Failed requests are not billed.",
  "error_code": "target_blocked",
  "status_code": 403,
  "status_message": "Forbidden",
  "body": "<html><body>Access denied</body></html>",
  "request_parameters": { "proxy": "datacenter", "js": false },
  "next_step": {
    "params": { "proxy": "residential" },
    "cost": 10,
    "message": "Retry with proxy=residential (10 credits per request)."
  }
}
```

The escalation ladder is **datacenter → residential → stealth**. Anti-bot challenge pages go straight to `proxy=stealth`, because they need the hardened browser, not just a better IP. When the request already ran on the stealth tier (or a dedicated route), `next_step` is omitted and the message says the target is currently not reachable; Google search URLs are pointed at the [`/serp` endpoint](https://webscraping.ai/docs/serp.md) instead. Automating the retry is safe: `next_step.params` are ordinary request parameters to set, `next_step.remove` (present only when you used `custom_proxy`, which overrides any proxy setting) lists parameters to drop, and `next_step.cost` is the price per request on current plans if the retry succeeds. If you would rather not automate it yourself, `proxy=auto` walks the same ladder inside one request (see the [proxy parameter](https://webscraping.ai/docs#param-proxy)); its error bodies carry `request_parameters.auto: true` and `request_parameters.tiers_tried`, and `request_parameters.proxy` is the tier the walk ended on.

| `error_code` | Meaning | Next step |
| --- | --- | --- |
| `target_blocked` | Target returned 403, 429, 503 or 999: an IP-reputation block or rate limit | Better proxy tier from `next_step` |
| `target_challenge` | Target served an anti-bot challenge page (Cloudflare, Akamai, PerimeterX, Datadome) instead of content, or closed the browser tab | `proxy=stealth`; a Cloudflare JavaScript challenge hit with `js=false` suggests `js=true` instead (with `proxy=residential` on datacenter) |
| `target_error` | Other non-2xx status from the target (its own 4xx or 5xx) | Check the URL; for 5xx a proxy upgrade is suggested as a hedge |
| `target_redirect` | Redirect returned because `error_on_redirect=true` | Billed as a success; the redirect target is in `message` |
| `target_unreachable` | Hostname does not resolve, or the target dropped the connection without an HTTP response (range-level blocking) | Check the URL, or the proxy tier from `next_step` |
| `wait_for_timeout` | `wait_for` selector never appeared | Check the selector or increase `timeout` |
| `invalid_url`, `forbidden_target`, `custom_proxy_error` | Malformed URL, a domain/body the API does not serve, or your `custom_proxy` rejected our credentials (407) | Fix the request |
| `response_too_large` | Response over the size limit | Use `/selected` or `/selected-multiple` |
| `timeout` (504), `internal_error` | Page did not load in time, or an error on our side | Increase `timeout`; for internal errors retry or contact support |

* * *

## Parameters Reference

Complete documentation for all API parameters of the scraping and AI endpoints. Click on any parameter to see detailed information and examples. The [search results endpoint](https://webscraping.ai/docs/serp.md) takes its own query parameters (`q`, `engine`, `gl`, `hl`, `page`), documented in its section, and so does the [structured data endpoint](https://webscraping.ai/docs/data.md) (`url`, `country`, `transcript`, `transcript_language`).

##### Quick Reference

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| [url](https://webscraping.ai/docs#param-url) | string | required | Target webpage URL |
| [api\_key](https://webscraping.ai/docs#param-api_key) | string | required | Your API key for authentication |
| [js](https://webscraping.ai/docs#param-js) | boolean | true | Enable JavaScript rendering |
| [js\_timeout](https://webscraping.ai/docs#param-js_timeout) | integer | 2000 | JavaScript rendering timeout (ms) |
| [timeout](https://webscraping.ai/docs#param-timeout) | integer | 10000 | Total request timeout (ms) |
| [wait\_for](https://webscraping.ai/docs#param-wait_for) | string | - | CSS selector to wait for |
| [proxy](https://webscraping.ai/docs#param-proxy) | string | datacenter | Proxy type |
| [country](https://webscraping.ai/docs#param-country) | string | us | Proxy country |
| [device](https://webscraping.ai/docs#param-device) | string | desktop | Device emulation |
| [headers](https://webscraping.ai/docs#param-headers) | object | - | Custom HTTP headers |
| [js\_script](https://webscraping.ai/docs#param-js_script) | string | - | Custom JavaScript to execute |
| [custom\_proxy](https://webscraping.ai/docs#param-custom_proxy) | string | - | Your own proxy URL |
| [error\_on\_404](https://webscraping.ai/docs#param-error_on_404) | boolean | false | Return error for 404 pages |
| [error\_on\_redirect](https://webscraping.ai/docs#param-error_on_redirect) | boolean | false | Return error on redirects |

urlstringrequired

The URL of the target webpage to scrape or analyze. Must be a valid HTTP or HTTPS URL.

**Details**
- Must include protocol (http:// or https://)
- URL encoding is handled automatically
- Redirects are followed by default
- Query parameters in URL are preserved

**Example** `url=https://example.com/page?id=123`

api\_keystringrequired

Your unique API key for authentication. Get your key from the [dashboard](https://webscraping.ai/dashboard).

**Security:** Never expose your API key in client-side code. Use environment variables or a backend proxy for production.

jsbooleandefault: true

Enable JavaScript rendering using a headless Chromium browser. Required for SPAs, dynamic content, and modern web applications.

**When to use js=true**
- Single Page Applications (React, Vue, Angular)
- Content loaded via AJAX/fetch
- Lazy-loaded images and content
- Interactive elements that need to render

**When to use js=false**
- Static HTML pages
- Faster response times needed
- Lower credit cost (no JS = 1 credit vs 5 credits)
- Server-rendered content

**Example**

```bash
# With JS rendering (default)
curl "https://api.webscraping.ai/html?api_key=KEY&url=https://spa-app.com&js=true"

# Without JS rendering (faster, cheaper)
curl "https://api.webscraping.ai/html?api_key=KEY&url=https://static-site.com&js=false"
```

js\_timeoutintegerdefault: 2000

Maximum time in milliseconds to wait for JavaScript execution after the page loads. Increase this value if you see loading indicators instead of actual content.

**Details**
- **Minimum:** 1 ms
- **Maximum:** 20000 ms (20 seconds)
- **Default:** 2000 ms (2 seconds)
- Only applies when `js=true`

**Recommendations**
- **Fast sites:** 1000-2000 ms
- **Medium sites:** 3000-5000 ms
- **Slow/heavy sites:** 5000-10000 ms
- Use `wait_for` for more precision

timeoutintegerdefault: 10000

Maximum total time in milliseconds for the entire request, including page retrieval, JavaScript rendering, and processing.

**Details**
- **Minimum:** 1 ms
- **Maximum:** 25000 ms (25 seconds); larger values are capped
- **Default:** 10000 ms (10 seconds)
- Increase if you get 504 timeout errors

**Example** `timeout=15000`

Sets a 15-second timeout for slow-loading pages

wait\_forstringoptional

CSS selector to wait for before returning the page content. The request will wait until this element appears in the DOM, then return the content. This overrides `js_timeout`.

If the element never appears within the request `timeout`, the request fails with a `500` error naming the selector, along with the target page's HTTP status code (`status_code`) and a preview of the page body (`body`) so you can see what the page actually contained. Failed requests are not billed. If you see this error on a page that should contain the element, check the selector, increase the `timeout` parameter, or inspect the `body` preview for anti-bot challenges.

**Use cases**
- Wait for product data to load
- Wait for search results
- Wait for lazy-loaded content
- Wait for specific components to render

**Examples**
- `wait_for=.product-price`
- `wait_for=#search-results`
- `wait_for=[data-loaded="true"]`
- `wait_for=.reviews-container`

**Example**

```bash
# Wait for product grid to load before scraping
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://shop.com/products" \
  --data-urlencode "wait_for=.product-grid"
```

proxystringdefault: datacenter

Type of proxy to use for the request. Choose based on your target website's anti-bot measures.

| Value | Description | Best for | Cost |
| --- | --- | --- | --- |
| `datacenter` | Fast datacenter proxies with rotating IPs | Most websites, APIs, general scraping | 1 credit (5 with JS) |
| `residential` | Real residential IPs from ISPs | Anti-bot protected sites, sneaker sites, social media | 10 credits (25 with JS) |
| `stealth` | Premium proxies with advanced anti-bot bypass | The most heavily protected sites where residential is not enough | 50 credits |
| `auto` | Tries datacenter, then residential, then stealth inside one request and returns the first result that isn't blocked | Unknown or mixed targets, first runs against a new site | The tier that worked (1–50 credits); failed attempts are free |

**Tip:** Start with datacenter proxies. Switch to residential if you encounter blocks or CAPTCHAs, and use stealth as a last resort for sites with the strongest anti-bot protection. `proxy=auto` does this walk for you: anti-bot challenge pages skip straight to stealth, and the API remembers the cheapest tier that worked for each domain for 30 days, so later requests to a protected site start there instead of re-trying the blocked tiers. Each extra tier costs response time, so once you know a site needs residential, setting it explicitly is faster. `auto` cannot be combined with `custom_proxy`.

countrystringdefault: us

Country code for geo-targeting. The request will be made from a proxy in the specified country.

| Code | Country | Code | Country |
| --- | --- | --- | --- |
| `us` | United States | `ru` | Russia |
| `gb` | United Kingdom | `jp` | Japan |
| `de` | Germany | `kr` | South Korea |
| `fr` | France | `in` | India |
| `ca` | Canada | `it` | Italy |
| `es` | Spain | `hk` | Hong Kong |
| `tr` | Turkey | | |

**Example**

```bash
# Scrape from a UK IP address
curl "https://api.webscraping.ai/html?api_key=KEY&url=https://uk-shop.com&country=gb"
```

devicestringdefault: desktop

Device type emulation. Affects viewport size, user agent, and touch capabilities.

| Value | Viewport | Use case |
| --- | --- | --- |
| `desktop` | 1920x940 (1080p screen) | Desktop websites, full layouts |
| `mobile` | 414x715 (iPhone 11) | Mobile sites, responsive layouts, AMP pages |
| `tablet` | 810x1080 (iPad 7th gen) | Tablet-optimized layouts |

headersobjectoptional

Custom HTTP headers to send with the request. Useful for authentication, cookies, or custom user agents.

**Format options**
- JSON object: `headers={"Cookie":"session=abc"}`
- Nested params: `headers[Cookie]=session=abc`

**Common headers**
- `Cookie` - Session cookies
- `Authorization` - Bearer tokens
- `Referer` - Referrer URL

**Example**

```bash
# Pass custom headers as JSON
curl -G "https://api.webscraping.ai/html" \
  --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode 'headers={"Cookie":"session=abc123","Authorization":"Bearer token"}'
```

js\_scriptstringoptional

Custom JavaScript code to execute on the page after it loads. Useful for clicking buttons, scrolling, filling forms, or extracting data. Only the [`/html`](https://webscraping.ai/docs/html.md) endpoint runs it; the other endpoints ignore it. The script's result is the value of its last expression, so don't use a top-level `return` (it is a syntax error).

**Use cases**
- Click "Load More" buttons
- Scroll to load lazy content
- Accept cookie banners
- Fill and submit forms
- Extract data from JavaScript variables

**Tip:** Use `return_script_result=true` to get the return value of your script instead of the page HTML.

**Examples**

```bash
# Click a button
js_script=document.querySelector('.load-more-btn').click()

# Scroll to bottom
js_script=window.scrollTo(0, document.body.scrollHeight)

# Extract data and return it (with return_script_result=true)
js_script=JSON.stringify(window. __INITIAL_DATA__ )
```

custom\_proxystringoptional

Use your own proxy server instead of our built-in proxy pool. Useful if you have specific proxy requirements or existing proxy subscriptions.

**Format** `http://username:password@host:port` **Example** `custom_proxy=http://user:pass@proxy.example.com:8080`
**Recommended providers:** [Decodo (formerly Smartproxy)](https://smartproxy.pxf.io/free_resi), [Bright Data](https://get.brightdata.com/webscrapingai)

error\_on\_404booleandefault: false

Return an error response when the target page returns a 404 status code, instead of returning the 404 page content.

- **false (default):** Returns the 404 page HTML content. Useful if you need to scrape the error page.
- **true:** Returns a 500 error. Useful for validation or when you want failed pages to trigger error handling.

error\_on\_redirectbooleandefault: false

Return an error when the target page redirects, instead of following the redirect.

- **false (default):** Automatically follows redirects and returns the final page content.
- **true:** Returns an error on redirect. Useful for detecting URL changes or validating canonical URLs.

## Full API Reference

For the complete machine-readable API specification — every endpoint, parameter, and response schema — download our OpenAPI 3.1 spec.

[Download OpenAPI Spec](https://webscraping.ai/openapi.yml)