Table of contents

How can I handle AJAX calls in Playwright?

AJAX (Asynchronous JavaScript and XML) calls allow web pages to update content dynamically without full page reloads. Playwright provides several powerful methods to handle and wait for AJAX requests during browser automation.

Core Methods for AJAX Handling

1. Waiting for Network Requests

waitForRequest() - Waits for a specific network request to be made:

# Python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto('https://example.com')

    # Wait for specific API request
    request = page.wait_for_request("**/api/users")
    print(f"Request URL: {request.url}")

    # With pattern matching
    request = page.wait_for_request(lambda request: "api" in request.url)
// JavaScript
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');

  // Wait for specific API request
  const request = await page.waitForRequest('**/api/users');
  console.log(`Request URL: ${request.url()}`);

  // With function predicate
  const request2 = await page.waitForRequest(request => 
    request.url().includes('api')
  );
})();

waitForResponse() - Waits for a specific network response:

# Python
# Wait for response and check status
response = page.wait_for_response("**/api/data")
print(f"Status: {response.status}")
print(f"Response body: {response.text()}")

# Wait for successful response only
response = page.wait_for_response(
    lambda response: response.url.endswith('/api/data') and response.status == 200
)
// JavaScript
// Wait for response and check status
const response = await page.waitForResponse('**/api/data');
console.log(`Status: ${response.status()}`);
console.log(`Response body: ${await response.text()}`);

// Wait for successful response only
const response2 = await page.waitForResponse((response) =>
  response.url().endsWith('/api/data') && response.status() === 200
);

2. Combining Actions with AJAX Waiting

Use Promise.all() to perform actions and wait for AJAX calls simultaneously:

# Python
# Trigger action and wait for AJAX response simultaneously
with page.expect_response("**/api/submit") as response_info:
    page.click("#submit-button")
response = response_info.value
print(f"Form submitted with status: {response.status}")
// JavaScript
// Trigger action and wait for AJAX response simultaneously
const [response] = await Promise.all([
  page.waitForResponse('**/api/submit'),
  page.click('#submit-button')
]);
console.log(`Form submitted with status: ${response.status()}`);

3. Route Interception and Modification

Intercept and modify AJAX requests before they're sent:

# Python
def handle_route(route, request):
    # Modify request headers
    headers = request.headers
    headers["Authorization"] = "Bearer token123"
    route.continue_(headers=headers)

# Intercept all API calls
page.route("**/api/**", handle_route)

# Mock API response
def mock_api(route, request):
    route.fulfill(
        status=200,
        content_type="application/json",
        body='{"users": [{"id": 1, "name": "Test User"}]}'
    )

page.route("**/api/users", mock_api)
// JavaScript
// Intercept and modify requests
await page.route('**/api/**', (route, request) => {
  const headers = {
    ...request.headers(),
    'Authorization': 'Bearer token123'
  };
  route.continue({ headers });
});

// Mock API response
await page.route('**/api/users', (route) => {
  route.fulfill({
    status: 200,
    contentType: 'application/json',
    body: JSON.stringify({ users: [{ id: 1, name: 'Test User' }] })
  });
});

4. Complete Example: Handling Dynamic Content Loading

# Python - Complete example
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    # Navigate to page
    page.goto('https://jsonplaceholder.typicode.com')

    # Click to load posts and wait for API response
    with page.expect_response("**/posts") as response_info:
        page.click("button:has-text('Load Posts')")

    response = response_info.value
    posts_data = response.json()
    print(f"Loaded {len(posts_data)} posts")

    # Wait for content to be rendered
    page.wait_for_selector(".post-item")

    # Extract loaded content
    posts = page.query_selector_all(".post-item")
    print(f"Found {len(posts)} post elements")

    browser.close()
// JavaScript - Complete example
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: false });
  const page = await browser.newPage();

  // Navigate to page
  await page.goto('https://jsonplaceholder.typicode.com');

  // Click to load posts and wait for API response
  const [response] = await Promise.all([
    page.waitForResponse('**/posts'),
    page.click('button:has-text("Load Posts")')
  ]);

  const postsData = await response.json();
  console.log(`Loaded ${postsData.length} posts`);

  // Wait for content to be rendered
  await page.waitForSelector('.post-item');

  // Extract loaded content
  const posts = await page.$$('.post-item');
  console.log(`Found ${posts.length} post elements`);

  await browser.close();
})();

Advanced Techniques

Waiting for Multiple AJAX Calls

# Python - Wait for multiple API calls
async def wait_for_multiple_requests():
    # Wait for all required API calls to complete
    with page.expect_response("**/api/users") as users_response:
        with page.expect_response("**/api/posts") as posts_response:
            page.click("#load-all-data")

    users = users_response.value.json()
    posts = posts_response.value.json()
    return users, posts
// JavaScript - Wait for multiple API calls
const [usersResponse, postsResponse] = await Promise.all([
  page.waitForResponse('**/api/users'),
  page.waitForResponse('**/api/posts'),
  page.click('#load-all-data')
]);

const users = await usersResponse.json();
const posts = await postsResponse.json();

Timeout Handling

# Python - Custom timeout handling
try:
    response = page.wait_for_response("**/api/slow-endpoint", timeout=10000)
    print("Response received within 10 seconds")
except TimeoutError:
    print("Request timed out after 10 seconds")
// JavaScript - Custom timeout handling
try {
  const response = await page.waitForResponse('**/api/slow-endpoint', { 
    timeout: 10000 
  });
  console.log('Response received within 10 seconds');
} catch (error) {
  console.log('Request timed out after 10 seconds');
}

Best Practices

  1. Use URL patterns (**/api/**) instead of full URLs for flexibility
  2. Combine actions with waits using Promise.all() or expect_response()
  3. Set appropriate timeouts for slow AJAX calls
  4. Validate response data before proceeding with tests
  5. Handle failed requests with try-catch blocks
  6. Use route interception for testing error scenarios

These methods provide comprehensive control over AJAX handling in Playwright, enabling robust automation of dynamic web applications.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon