Table of contents

How can I use Selenium WebDriver with C# for browser automation?

Selenium WebDriver is a powerful browser automation framework that allows you to programmatically control web browsers using C#. It's widely used for web scraping, automated testing, and browser-based tasks that require JavaScript rendering or user interaction simulation.

Installing Selenium WebDriver for C

To get started with Selenium WebDriver in C#, you'll need to install the necessary NuGet packages. The most common approach is to use the Selenium.WebDriver package along with a browser-specific driver.

Installation via NuGet Package Manager Console

# Install Selenium WebDriver
Install-Package Selenium.WebDriver

# Install Chrome driver (choose one based on your browser)
Install-Package Selenium.WebDriver.ChromeDriver

# Alternative: Install Firefox driver
Install-Package Selenium.WebDriver.GeckoDriver.Win64

# Alternative: Install Edge driver
Install-Package Selenium.WebDriver.MSEdgeDriver

Installation via .NET CLI

dotnet add package Selenium.WebDriver
dotnet add package Selenium.WebDriver.ChromeDriver

Basic Selenium WebDriver Setup

Here's a simple example to get started with Selenium WebDriver in C#:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;

class Program
{
    static void Main()
    {
        // Create a new instance of ChromeDriver
        IWebDriver driver = new ChromeDriver();

        try
        {
            // Navigate to a website
            driver.Navigate().GoToUrl("https://example.com");

            // Get page title
            Console.WriteLine($"Page title: {driver.Title}");

            // Get current URL
            Console.WriteLine($"Current URL: {driver.Url}");
        }
        finally
        {
            // Always close the browser
            driver.Quit();
        }
    }
}

Configuring Browser Options

For more control over browser behavior, you can configure various options before launching the browser:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

public class SeleniumConfig
{
    public static IWebDriver CreateHeadlessBrowser()
    {
        ChromeOptions options = new ChromeOptions();

        // Run in headless mode (no GUI)
        options.AddArgument("--headless");

        // Disable GPU acceleration
        options.AddArgument("--disable-gpu");

        // Set window size
        options.AddArgument("--window-size=1920,1080");

        // Disable images for faster loading
        options.AddArgument("--blink-settings=imagesEnabled=false");

        // User agent customization
        options.AddArgument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36");

        // Disable automation detection
        options.AddExcludedArgument("enable-automation");
        options.AddAdditionalOption("useAutomationExtension", false);

        return new ChromeDriver(options);
    }
}

Finding and Interacting with Elements

Selenium provides multiple strategies for locating elements on a page:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;
using System;

public class ElementInteraction
{
    public static void InteractWithElements()
    {
        IWebDriver driver = new ChromeDriver();
        driver.Navigate().GoToUrl("https://example.com/form");

        try
        {
            // Find element by ID
            IWebElement emailInput = driver.FindElement(By.Id("email"));
            emailInput.SendKeys("user@example.com");

            // Find element by Name
            IWebElement passwordInput = driver.FindElement(By.Name("password"));
            passwordInput.SendKeys("securePassword123");

            // Find element by CSS Selector
            IWebElement submitButton = driver.FindElement(By.CssSelector("button[type='submit']"));
            submitButton.Click();

            // Find element by XPath
            IWebElement heading = driver.FindElement(By.XPath("//h1[@class='title']"));
            Console.WriteLine($"Heading text: {heading.Text}");

            // Find multiple elements
            var links = driver.FindElements(By.TagName("a"));
            Console.WriteLine($"Total links: {links.Count}");

            // Find element by link text
            IWebElement aboutLink = driver.FindElement(By.LinkText("About Us"));
            aboutLink.Click();

            // Find element by partial link text
            IWebElement contactLink = driver.FindElement(By.PartialLinkText("Contact"));
            contactLink.Click();
        }
        finally
        {
            driver.Quit();
        }
    }
}

Handling Waits and Synchronization

One of the most critical aspects of browser automation is properly handling timing and synchronization. Similar to how you would handle timeouts in Puppeteer, Selenium offers multiple wait strategies:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;
using System;

public class WaitStrategies
{
    public static void DemonstrateWaits()
    {
        IWebDriver driver = new ChromeDriver();

        try
        {
            // Implicit wait (applies to all FindElement calls)
            driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);

            driver.Navigate().GoToUrl("https://example.com");

            // Explicit wait for a specific condition
            WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(15));

            // Wait until element is visible
            IWebElement element = wait.Until(driver =>
            {
                var el = driver.FindElement(By.Id("dynamicContent"));
                return el.Displayed ? el : null;
            });

            // Wait until element is clickable
            wait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.ElementToBeClickable(By.Id("submitBtn")));

            // Wait until title contains specific text
            wait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.TitleContains("Welcome"));

            // Wait until URL changes
            wait.Until(driver => driver.Url.Contains("dashboard"));

            // Custom wait condition
            wait.Until(driver =>
            {
                var readyState = ((IJavaScriptExecutor)driver)
                    .ExecuteScript("return document.readyState").ToString();
                return readyState == "complete";
            });
        }
        finally
        {
            driver.Quit();
        }
    }
}

Note: To use ExpectedConditions, install the DotNetSeleniumExtras.WaitHelpers package:

Install-Package DotNetSeleniumExtras.WaitHelpers

Executing JavaScript

Selenium allows you to execute custom JavaScript code within the browser context:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;

public class JavaScriptExecution
{
    public static void ExecuteJavaScript()
    {
        IWebDriver driver = new ChromeDriver();
        IJavaScriptExecutor js = (IJavaScriptExecutor)driver;

        try
        {
            driver.Navigate().GoToUrl("https://example.com");

            // Scroll to bottom of page
            js.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");

            // Click an element using JavaScript
            IWebElement button = driver.FindElement(By.Id("submitBtn"));
            js.ExecuteScript("arguments[0].click();", button);

            // Get page height
            long pageHeight = (long)js.ExecuteScript("return document.body.scrollHeight;");
            Console.WriteLine($"Page height: {pageHeight}px");

            // Modify element properties
            IWebElement input = driver.FindElement(By.Id("email"));
            js.ExecuteScript("arguments[0].value = 'test@example.com';", input);

            // Extract data from page
            string jsonData = (string)js.ExecuteScript("return JSON.stringify(window.pageData);");
            Console.WriteLine($"Page data: {jsonData}");

            // Wait for AJAX to complete
            js.ExecuteScript("return jQuery.active == 0");
        }
        finally
        {
            driver.Quit();
        }
    }
}

Handling Multiple Windows and Tabs

When working with pop-ups or multiple browser windows, you'll need to switch between them:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;
using System.Linq;

public class WindowHandling
{
    public static void HandleMultipleWindows()
    {
        IWebDriver driver = new ChromeDriver();

        try
        {
            driver.Navigate().GoToUrl("https://example.com");

            // Store the main window handle
            string mainWindow = driver.CurrentWindowHandle;

            // Click a link that opens a new window
            driver.FindElement(By.LinkText("Open New Window")).Click();

            // Get all window handles
            var windowHandles = driver.WindowHandles;

            // Switch to the new window
            foreach (var handle in windowHandles)
            {
                if (handle != mainWindow)
                {
                    driver.SwitchTo().Window(handle);
                    break;
                }
            }

            // Perform actions in the new window
            Console.WriteLine($"New window title: {driver.Title}");

            // Close the current window
            driver.Close();

            // Switch back to main window
            driver.SwitchTo().Window(mainWindow);

            // Open a new tab using JavaScript
            ((IJavaScriptExecutor)driver).ExecuteScript("window.open('https://example.org', '_blank');");

            // Switch to the latest tab
            driver.SwitchTo().Window(driver.WindowHandles.Last());
        }
        finally
        {
            driver.Quit();
        }
    }
}

Handling Alerts and Dialogs

Managing JavaScript alerts, confirms, and prompts requires special handling:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;

public class AlertHandling
{
    public static void HandleAlerts()
    {
        IWebDriver driver = new ChromeDriver();

        try
        {
            driver.Navigate().GoToUrl("https://example.com");

            // Trigger an alert
            driver.FindElement(By.Id("alertButton")).Click();

            // Switch to alert
            IAlert alert = driver.SwitchTo().Alert();

            // Get alert text
            Console.WriteLine($"Alert text: {alert.Text}");

            // Accept alert (click OK)
            alert.Accept();

            // For confirmation dialogs
            driver.FindElement(By.Id("confirmButton")).Click();
            IAlert confirmDialog = driver.SwitchTo().Alert();
            confirmDialog.Dismiss(); // Click Cancel

            // For prompt dialogs
            driver.FindElement(By.Id("promptButton")).Click();
            IAlert promptDialog = driver.SwitchTo().Alert();
            promptDialog.SendKeys("User input");
            promptDialog.Accept();
        }
        finally
        {
            driver.Quit();
        }
    }
}

Working with Frames and iFrames

When you need to interact with content inside frames, similar to handling iframes in Puppeteer:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

public class FrameHandling
{
    public static void HandleFrames()
    {
        IWebDriver driver = new ChromeDriver();

        try
        {
            driver.Navigate().GoToUrl("https://example.com/page-with-iframe");

            // Switch to frame by index
            driver.SwitchTo().Frame(0);

            // Switch to frame by name or ID
            driver.SwitchTo().Frame("frameName");

            // Switch to frame by WebElement
            IWebElement frameElement = driver.FindElement(By.Id("myFrame"));
            driver.SwitchTo().Frame(frameElement);

            // Interact with elements inside the frame
            IWebElement elementInFrame = driver.FindElement(By.Id("frameContent"));
            Console.WriteLine(elementInFrame.Text);

            // Switch back to main content
            driver.SwitchTo().DefaultContent();

            // Switch to parent frame (for nested frames)
            driver.SwitchTo().ParentFrame();
        }
        finally
        {
            driver.Quit();
        }
    }
}

Complete Web Scraping Example

Here's a practical example that demonstrates scraping product information from an e-commerce site:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;
using System;
using System.Collections.Generic;
using System.Linq;

public class Product
{
    public string Title { get; set; }
    public decimal Price { get; set; }
    public string Rating { get; set; }
    public string Url { get; set; }
}

public class WebScraper
{
    public static List<Product> ScrapeProducts(string url)
    {
        var products = new List<Product>();

        ChromeOptions options = new ChromeOptions();
        options.AddArgument("--headless");
        options.AddArgument("--disable-gpu");

        IWebDriver driver = new ChromeDriver(options);

        try
        {
            driver.Navigate().GoToUrl(url);

            // Wait for products to load
            WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
            wait.Until(driver => driver.FindElements(By.CssSelector(".product-item")).Count > 0);

            // Scroll to load all products (lazy loading)
            IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
            js.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");

            System.Threading.Thread.Sleep(2000); // Wait for lazy-loaded content

            // Find all product elements
            var productElements = driver.FindElements(By.CssSelector(".product-item"));

            foreach (var productElement in productElements)
            {
                try
                {
                    var product = new Product
                    {
                        Title = productElement.FindElement(By.CssSelector(".product-title")).Text,
                        Price = decimal.Parse(
                            productElement.FindElement(By.CssSelector(".product-price"))
                                .Text.Replace("$", "").Trim()
                        ),
                        Rating = productElement.FindElement(By.CssSelector(".product-rating")).Text,
                        Url = productElement.FindElement(By.CssSelector("a")).GetAttribute("href")
                    };

                    products.Add(product);
                }
                catch (NoSuchElementException ex)
                {
                    Console.WriteLine($"Error parsing product: {ex.Message}");
                    continue;
                }
            }

            Console.WriteLine($"Successfully scraped {products.Count} products");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error during scraping: {ex.Message}");
        }
        finally
        {
            driver.Quit();
        }

        return products;
    }

    public static void Main()
    {
        var products = ScrapeProducts("https://example.com/products");

        foreach (var product in products)
        {
            Console.WriteLine($"{product.Title} - ${product.Price} - Rating: {product.Rating}");
        }
    }
}

Error Handling and Best Practices

Robust error handling is essential for reliable browser automation:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;

public class RobustScraper
{
    public static void ScrapeSafely()
    {
        IWebDriver driver = null;

        try
        {
            ChromeOptions options = new ChromeOptions();
            options.AddArgument("--headless");
            options.PageLoadTimeout = TimeSpan.FromSeconds(30);

            driver = new ChromeDriver(options);
            driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);

            driver.Navigate().GoToUrl("https://example.com");

            try
            {
                var element = driver.FindElement(By.Id("targetElement"));
                Console.WriteLine(element.Text);
            }
            catch (NoSuchElementException)
            {
                Console.WriteLine("Element not found, trying alternative selector");
                var altElement = driver.FindElement(By.CssSelector(".alternative-class"));
                Console.WriteLine(altElement.Text);
            }
            catch (StaleElementReferenceException)
            {
                Console.WriteLine("Element became stale, refinding element");
                var element = driver.FindElement(By.Id("targetElement"));
                Console.WriteLine(element.Text);
            }
        }
        catch (WebDriverTimeoutException ex)
        {
            Console.WriteLine($"Timeout occurred: {ex.Message}");
        }
        catch (WebDriverException ex)
        {
            Console.WriteLine($"WebDriver error: {ex.Message}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Unexpected error: {ex.Message}");
        }
        finally
        {
            driver?.Quit();
        }
    }
}

Performance Optimization Tips

  1. Use headless mode for faster execution when you don't need to see the browser
  2. Disable images and CSS when you only need HTML content
  3. Use implicit waits sparingly - prefer explicit waits for specific conditions
  4. Reuse browser instances when scraping multiple pages from the same domain
  5. Close unused tabs and windows to free up memory
  6. Use network request monitoring techniques to wait for specific API calls to complete

When to Use an API Instead

While Selenium WebDriver is powerful, it's resource-intensive and slower than HTTP-based scraping. For production web scraping at scale, consider using specialized APIs like WebScraping.AI that handle browser automation, proxy rotation, and CAPTCHA solving for you. This is particularly important when you need to scrape thousands of pages or when dealing with complex anti-bot systems.

Conclusion

Selenium WebDriver provides comprehensive browser automation capabilities for C# developers. It's ideal for scraping JavaScript-heavy websites, automating complex user interactions, and testing web applications. By following the patterns and best practices outlined in this guide, you can build robust and reliable browser automation solutions for your web scraping and testing needs.

Remember to always respect websites' robots.txt files, terms of service, and implement appropriate rate limiting to avoid overloading servers.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What is the best way to parse dates and times in C# from scraped content?
How do I make HTTP POST requests in C# for web scraping?
How do I make HTTP GET requests in C# for web scraping?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon