Table of contents

How do I simulate different network conditions in Headless Chromium?

Network condition simulation in Headless Chromium is essential for testing web applications under various network scenarios, including slow connections, limited bandwidth, high latency, and offline conditions. This capability helps developers optimize their applications for different user environments and test edge cases that might not be apparent with fast local connections.

Understanding Network Throttling

Headless Chromium provides built-in network throttling capabilities through the Chrome DevTools Protocol (CDP). This allows you to simulate various network conditions including:

  • Download/Upload throughput (bytes per second)
  • Latency (milliseconds)
  • Connection type (WiFi, 3G, 4G, etc.)
  • Offline mode

Implementation with Puppeteer

Puppeteer provides the most straightforward way to control network conditions in Headless Chromium. Here's how to implement network throttling:

Basic Network Throttling Setup

const puppeteer = require('puppeteer');

async function simulateNetworkConditions() {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  // Enable network domain
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');

  // Simulate slow 3G connection
  await client.send('Network.emulateNetworkConditions', {
    offline: false,
    downloadThroughput: 500 * 1024 / 8, // 500 Kbps in bytes/second
    uploadThroughput: 500 * 1024 / 8,   // 500 Kbps in bytes/second
    latency: 400 // 400ms latency
  });

  await page.goto('https://example.com');

  // Your scraping logic here

  await browser.close();
}

simulateNetworkConditions();

Predefined Network Profiles

Create reusable network profiles for common scenarios:

const networkProfiles = {
  'Slow 3G': {
    offline: false,
    downloadThroughput: 500 * 1024 / 8,
    uploadThroughput: 500 * 1024 / 8,
    latency: 400
  },
  'Fast 3G': {
    offline: false,
    downloadThroughput: 1.6 * 1024 * 1024 / 8,
    uploadThroughput: 750 * 1024 / 8,
    latency: 150
  },
  'WiFi': {
    offline: false,
    downloadThroughput: 30 * 1024 * 1024 / 8,
    uploadThroughput: 15 * 1024 * 1024 / 8,
    latency: 28
  },
  'Offline': {
    offline: true,
    downloadThroughput: 0,
    uploadThroughput: 0,
    latency: 0
  }
};

async function applyNetworkProfile(page, profileName) {
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');

  const profile = networkProfiles[profileName];
  if (!profile) {
    throw new Error(`Network profile '${profileName}' not found`);
  }

  await client.send('Network.emulateNetworkConditions', profile);
  console.log(`Applied network profile: ${profileName}`);
}

// Usage
const browser = await puppeteer.launch();
const page = await browser.newPage();
await applyNetworkProfile(page, 'Slow 3G');

Advanced Network Configuration

For more sophisticated network simulation, you can implement dynamic throttling:

class NetworkThrottler {
  constructor(page) {
    this.page = page;
    this.client = null;
  }

  async initialize() {
    this.client = await this.page.target().createCDPSession();
    await this.client.send('Network.enable');
  }

  async setConditions(conditions) {
    if (!this.client) {
      await this.initialize();
    }

    await this.client.send('Network.emulateNetworkConditions', {
      offline: conditions.offline || false,
      downloadThroughput: conditions.downloadThroughput || 0,
      uploadThroughput: conditions.uploadThroughput || 0,
      latency: conditions.latency || 0
    });
  }

  async simulateIntermittentConnection() {
    // Simulate unstable connection
    const conditions = [
      { downloadThroughput: 1000 * 1024 / 8, latency: 100 },
      { offline: true },
      { downloadThroughput: 500 * 1024 / 8, latency: 300 },
      { downloadThroughput: 2000 * 1024 / 8, latency: 50 }
    ];

    for (let i = 0; i < conditions.length; i++) {
      await this.setConditions(conditions[i]);
      await new Promise(resolve => setTimeout(resolve, 5000)); // Wait 5 seconds
    }
  }

  async disable() {
    await this.client.send('Network.emulateNetworkConditions', {
      offline: false,
      downloadThroughput: 0,
      uploadThroughput: 0,
      latency: 0
    });
  }
}

// Usage example
const throttler = new NetworkThrottler(page);
await throttler.setConditions({
  downloadThroughput: 1.5 * 1024 * 1024 / 8, // 1.5 Mbps
  uploadThroughput: 750 * 1024 / 8,           // 750 Kbps
  latency: 200                                // 200ms
});

Python Implementation with Selenium

For Python developers using Selenium with ChromeDriver, network throttling requires additional setup:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import json

def setup_network_throttling():
    # Chrome options for headless mode
    chrome_options = Options()
    chrome_options.add_argument('--headless')
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument('--disable-dev-shm-usage')

    # Enable performance logging
    caps = DesiredCapabilities.CHROME
    caps['goog:loggingPrefs'] = {'performance': 'ALL'}

    driver = webdriver.Chrome(options=chrome_options, desired_capabilities=caps)

    return driver

def apply_network_conditions(driver, conditions):
    """Apply network throttling conditions"""
    command_result = driver.execute_cdp_cmd('Network.enable', {})

    driver.execute_cdp_cmd('Network.emulateNetworkConditions', {
        'offline': conditions.get('offline', False),
        'downloadThroughput': conditions.get('downloadThroughput', 0),
        'uploadThroughput': conditions.get('uploadThroughput', 0),
        'latency': conditions.get('latency', 0)
    })

# Network profiles
NETWORK_PROFILES = {
    'slow_3g': {
        'downloadThroughput': 500 * 1024 / 8,  # 500 Kbps
        'uploadThroughput': 500 * 1024 / 8,    # 500 Kbps
        'latency': 400                         # 400ms
    },
    'fast_3g': {
        'downloadThroughput': 1.6 * 1024 * 1024 / 8,  # 1.6 Mbps
        'uploadThroughput': 750 * 1024 / 8,            # 750 Kbps
        'latency': 150                                 # 150ms
    },
    'offline': {
        'offline': True
    }
}

# Usage example
driver = setup_network_throttling()

# Apply slow 3G conditions
apply_network_conditions(driver, NETWORK_PROFILES['slow_3g'])

# Navigate and test
driver.get('https://example.com')

# Measure performance
navigation_start = driver.execute_script("return window.performance.timing.navigationStart")
dom_complete = driver.execute_script("return window.performance.timing.domComplete")
load_time = dom_complete - navigation_start

print(f"Page load time under slow 3G: {load_time}ms")

driver.quit()

Command Line Interface

You can also control network conditions directly through Chrome's command line arguments:

# Launch Chrome with network throttling
google-chrome --headless \
  --enable-features=NetworkService \
  --force-fieldtrials="NetworkService/Enabled" \
  --force-fieldtrial-params="NetworkService.Enabled:throttling_profile/Slow-3G" \
  --dump-dom https://example.com

# Available throttling profiles:
# - Slow-3G
# - Fast-3G  
# - WiFi
# - Offline

Testing Performance Under Different Conditions

Combine network throttling with performance monitoring to gather meaningful metrics:

async function testPagePerformance(url, networkProfile) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Enable performance metrics
  await page.tracing.start({ path: 'trace.json' });

  // Apply network conditions
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');
  await client.send('Network.emulateNetworkConditions', networkProfile);

  // Measure navigation timing
  const startTime = Date.now();

  await page.goto(url, { waitUntil: 'networkidle2' });

  const endTime = Date.now();
  const loadTime = endTime - startTime;

  // Get performance metrics
  const performanceMetrics = await page.evaluate(() => {
    const timing = performance.timing;
    return {
      domContentLoaded: timing.domContentLoadedEventEnd - timing.navigationStart,
      firstPaint: performance.getEntriesByType('paint')[0]?.startTime || 0,
      firstContentfulPaint: performance.getEntriesByType('paint')[1]?.startTime || 0
    };
  });

  await page.tracing.stop();

  console.log(`Network Profile: ${JSON.stringify(networkProfile)}`);
  console.log(`Total Load Time: ${loadTime}ms`);
  console.log(`Performance Metrics:`, performanceMetrics);

  await browser.close();

  return { loadTime, performanceMetrics };
}

// Test multiple network conditions
const testConditions = [
  { name: 'WiFi', ...networkProfiles.WiFi },
  { name: 'Fast 3G', ...networkProfiles['Fast 3G'] },
  { name: 'Slow 3G', ...networkProfiles['Slow 3G'] }
];

for (const condition of testConditions) {
  await testPagePerformance('https://example.com', condition);
}

Monitoring Network Activity

When simulating network conditions, it's valuable to monitor network requests in Puppeteer to understand how your application behaves:

async function monitorNetworkWithThrottling(page, networkProfile) {
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');

  // Apply throttling
  await client.send('Network.emulateNetworkConditions', networkProfile);

  // Monitor network events
  const networkEvents = [];

  client.on('Network.requestWillBeSent', (params) => {
    networkEvents.push({
      type: 'request',
      url: params.request.url,
      method: params.request.method,
      timestamp: params.timestamp
    });
  });

  client.on('Network.responseReceived', (params) => {
    networkEvents.push({
      type: 'response',
      url: params.response.url,
      status: params.response.status,
      timestamp: params.timestamp
    });
  });

  client.on('Network.loadingFailed', (params) => {
    networkEvents.push({
      type: 'failed',
      url: params.request?.url || 'unknown',
      errorText: params.errorText,
      timestamp: params.timestamp
    });
  });

  return networkEvents;
}

Best Practices and Considerations

1. Realistic Network Profiles

Use network profiles that reflect real-world conditions:

const realisticProfiles = {
  'Mobile 2G': {
    downloadThroughput: 256 * 1024 / 8,  // 256 Kbps
    uploadThroughput: 256 * 1024 / 8,    // 256 Kbps
    latency: 800                         // 800ms
  },
  'Mobile 3G': {
    downloadThroughput: 1.6 * 1024 * 1024 / 8,  // 1.6 Mbps
    uploadThroughput: 768 * 1024 / 8,            // 768 Kbps
    latency: 300                                 // 300ms
  },
  'Broadband': {
    downloadThroughput: 10 * 1024 * 1024 / 8,   // 10 Mbps
    uploadThroughput: 2 * 1024 * 1024 / 8,      // 2 Mbps
    latency: 40                                  // 40ms
  }
};

2. Testing Edge Cases

Test how your application handles network failures and recovery:

async function testNetworkFailureRecovery(page) {
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');

  // Start with normal connection
  await page.goto('https://example.com');

  // Simulate network failure
  await client.send('Network.emulateNetworkConditions', { offline: true });

  // Wait and then restore connection
  await new Promise(resolve => setTimeout(resolve, 5000));

  await client.send('Network.emulateNetworkConditions', {
    offline: false,
    downloadThroughput: 1000 * 1024 / 8,
    uploadThroughput: 1000 * 1024 / 8,
    latency: 100
  });

  // Test if application recovers properly
}

3. Performance Budgets

Set performance budgets for different network conditions and fail tests when exceeded:

const performanceBudgets = {
  'Fast 3G': { maxLoadTime: 5000 },
  'Slow 3G': { maxLoadTime: 15000 },
  'WiFi': { maxLoadTime: 3000 }
};

async function validatePerformanceBudget(url, networkProfile, budget) {
  const result = await testPagePerformance(url, networkProfile);

  if (result.loadTime > budget.maxLoadTime) {
    throw new Error(`Performance budget exceeded: ${result.loadTime}ms > ${budget.maxLoadTime}ms`);
  }

  return result;
}

Troubleshooting Common Issues

Network Throttling Not Working

Ensure you're enabling the Network domain before setting conditions:

// Always enable Network domain first
await client.send('Network.enable');
await client.send('Network.emulateNetworkConditions', conditions);

Inconsistent Results

Network throttling can be affected by system resources. Consider using Puppeteer with Docker for consistent testing environments.

Handling Timeouts

When testing slow network conditions, adjust your timeout handling in Puppeteer accordingly:

await page.goto(url, { 
  waitUntil: 'networkidle2', 
  timeout: 30000 // Increase timeout for slow networks
});

Real-World Testing Scenarios

Mobile Performance Testing

async function testMobilePerformance(url) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Set mobile viewport
  await page.setViewport({ width: 375, height: 667, isMobile: true });

  // Apply mobile 3G conditions
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');
  await client.send('Network.emulateNetworkConditions', realisticProfiles['Mobile 3G']);

  // Test page load performance
  const startTime = Date.now();
  await page.goto(url, { waitUntil: 'networkidle2' });
  const loadTime = Date.now() - startTime;

  console.log(`Mobile 3G load time: ${loadTime}ms`);

  await browser.close();
  return loadTime;
}

Progressive Web App Testing

async function testPWAOfflineCapabilities(url) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // First visit to cache resources
  await page.goto(url);
  await page.waitForTimeout(2000);

  // Go offline
  const client = await page.target().createCDPSession();
  await client.send('Network.enable');
  await client.send('Network.emulateNetworkConditions', { offline: true });

  // Test offline functionality
  try {
    await page.reload({ waitUntil: 'networkidle2' });
    console.log('PWA works offline successfully');
  } catch (error) {
    console.log('PWA failed offline test:', error.message);
  }

  await browser.close();
}

Conclusion

Simulating different network conditions in Headless Chromium is crucial for building robust web applications that perform well across various network scenarios. By implementing proper network throttling, monitoring network activity, and testing edge cases, you can ensure your applications provide a good user experience regardless of network conditions.

The combination of realistic network profiles, performance monitoring, and systematic testing helps identify performance bottlenecks and optimization opportunities that might not be apparent under ideal network conditions. This approach is particularly valuable for mobile-first applications and global web services that serve users with diverse connectivity scenarios.

Whether you're testing progressive web apps, optimizing for mobile users, or ensuring your scraping scripts work reliably under poor network conditions, network simulation in Headless Chromium provides the tools needed to build resilient and performant applications.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What are the best practices for running Headless Chromium in production?
How do I handle page navigation and redirects in Headless Chromium?
Can I use Headless Chromium to generate PDFs from web pages?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon