How do I execute custom JavaScript on a webpage using Nightmare?

Nightmare is a high-level browser automation library for Node.js, which uses Electron under the hood. It allows you to interact with web pages through an API to perform tasks such as navigating, filling out forms, clicking buttons, and even executing custom JavaScript code within the context of the webpage.

To execute custom JavaScript on a webpage using Nightmare, you can use the .evaluate() function. This function allows you to run any JavaScript code within the page context. It can also return values back to the Node.js context.

Here's a simple example of how to use Nightmare to navigate to a page and execute custom JavaScript:

const Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true }); // set `show` to false if you don't need to show the browser window

nightmare
  .goto('https://example.com') // navigate to the webpage
  .evaluate(() => {
    // This code runs in the context of the browser
    const title = document.title;
    // You can manipulate the DOM or run any JavaScript code
    const headings = Array.from(document.querySelectorAll('h1')).map(h => h.textContent);
    // Return what you need
    return { title, headings };
  })
  .end() // end the Nightmare instance
  .then((result) => {
    // Output the result from the evaluate function
    console.log(result);
  })
  .catch((error) => {
    console.error('Search failed:', error);
  });

In the example above, the .evaluate() function is used to get the document's title and all h1 headings text content. Then, it packages those into an object which is returned to the .then() function in the Node.js context.

Please note that the .evaluate() function is sandboxed, meaning that it cannot access variables or functions defined outside of its scope. To pass arguments to the .evaluate() function, you can add them as additional parameters after the function definition:

nightmare
  .goto('https://example.com')
  .evaluate((arg1, arg2) => {
    // Custom JavaScript that uses arg1 and arg2
    return { arg1, arg2 };
  }, 'First Argument', 'Second Argument') // passing arguments to the evaluate function
  .then((result) => {
    console.log(result);
  })
  .catch((error) => {
    console.error('Search failed:', error);
  });

To use Nightmare, make sure you have it installed via npm:

npm install nightmare

Always be mindful of the legal and ethical aspects of web scraping. Make sure you are allowed to scrape and execute JavaScript on the target website by checking the site's robots.txt file and terms of service.