Can Selenium handle websites with complex JavaScript?

Yes, Selenium can handle websites with complex JavaScript. Selenium WebDriver is one of the most popular tools for Web UI Automation. It interacts with the web page just like a human would - clicking, typing, and extracting text.

While many scraping tools, like Beautiful Soup, can't interact with JavaScript, Selenium can. It works by automating a real web browser, allowing it to interact with all aspects of a web page - including the execution of JavaScript code.

Here's an example of how you can use Selenium to interact with JavaScript on a webpage:

from selenium import webdriver

# Open a browser and navigate to the page
driver = webdriver.Firefox()
driver.get('http://www.example.com')

# Execute some JavaScript
driver.execute_script('return document.title')

# Close the browser
driver.quit()

In this example, the execute_script function is used to run JavaScript code. It's like typing JavaScript code directly into the browser's developer console.

Selenium WebDriver supports multiple programming languages, and it works the same way in each. Here's the same example in JavaScript:

const {Builder, By, Key, until} = require('selenium-webdriver');

(async function example() {
  let driver = await new Builder().forBrowser('firefox').build();
  try {
    await driver.get('http://www.example.com');
    let title = await driver.executeScript('return document.title');
    console.log(title);
  } finally {
    await driver.quit();
  }
})();

In this JavaScript example, we use the executeScript function in the same way as in the Python example. This function returns a Promise, so we use the await keyword to wait for the Promise to resolve.

In conclusion, Selenium WebDriver provides a powerful tool for automating web browsers, which allows it to interact with JavaScript on a webpage. By using the executeScript function, we can execute JavaScript code directly in the browser, allowing us to interact with even the most complex JavaScript websites.