Is there a way to capture network traffic with Headless Chromium?

Yes, it is indeed possible to capture network traffic using headless Chromium. You can do this by controlling the browser through tools like Puppeteer (for Node.js) or Selenium with a language like Python. These tools allow you to automate browser interaction, including capturing network requests and responses.

Here's how you can capture network traffic using Puppeteer (Node.js) and Selenium (Python) with headless Chromium:

Using Puppeteer with Node.js

First, ensure you have Puppeteer installed:

npm install puppeteer

Then, use the following Node.js script to capture network traffic:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  // Listen for all network requests
  page.on('request', request => {
    console.log('Request URL:', request.url());
  });

  // Listen for all network responses
  page.on('response', response => {
    console.log('Response URL:', response.url());
  });

  await page.goto('https://example.com');

  // Other actions...

  await browser.close();
})();

Using Selenium with Python

First, you need to install Selenium and the Chrome WebDriver:

pip install selenium

You can download the Chrome WebDriver from the Chromium website and ensure it is in your PATH.

Then use the following Python script with Selenium to capture network traffic:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

# Enable performance logging
caps = DesiredCapabilities.CHROME
caps['goog:loggingPrefs'] = {'performance': 'ALL'}

# Set Chrome options to run headless
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')  # Required on some Windows systems

# Initialize WebDriver
driver = webdriver.Chrome(desired_capabilities=caps, options=options)

# Navigate to a page
driver.get('https://example.com')

# Retrieve and process the performance logs
logs = driver.get_log('performance')
for entry in logs:
    print(entry)

# Other actions...

# Close the browser
driver.quit()

The performance logs will contain network traffic information, including requests and responses in a JSON format. You may need to parse this JSON to extract the information you're interested in.

Please note that browser versions and dependencies change over time, so you may need to update the code or dependencies accordingly. Always refer to the latest documentation for Puppeteer and Selenium for the most up-to-date methods and best practices.