Table of contents

How can I scrape data from mobile-responsive websites using Java?

Scraping mobile-responsive websites requires special considerations since these sites adapt their layout, content, and functionality based on the device's screen size and capabilities. Java provides powerful tools like Selenium WebDriver and headless browsers to effectively scrape mobile-responsive content by emulating mobile devices and handling dynamic layouts.

Understanding Mobile-Responsive Challenges

Mobile-responsive websites present unique challenges for web scraping:

  • Dynamic content loading: Elements may load differently based on viewport size
  • Touch-optimized interfaces: Different interaction patterns compared to desktop
  • Conditional content: Some content may only appear on mobile or desktop versions
  • Performance optimizations: Lazy loading and progressive enhancement techniques
  • Different DOM structures: Mobile layouts often use different HTML structures

Setting Up Selenium WebDriver for Mobile Scraping

Basic Mobile Chrome Configuration

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import java.util.HashMap;
import java.util.Map;

public class MobileScraper {

    public static WebDriver createMobileDriver() {
        ChromeOptions options = new ChromeOptions();

        // Enable mobile emulation
        Map<String, String> mobileEmulation = new HashMap<>();
        mobileEmulation.put("deviceName", "iPhone 12 Pro");
        options.setExperimentalOption("mobileEmulation", mobileEmulation);

        // Additional mobile-optimized settings
        options.addArguments("--user-agent=Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15");
        options.addArguments("--disable-blink-features=AutomationControlled");
        options.addArguments("--no-sandbox");
        options.addArguments("--disable-dev-shm-usage");

        return new ChromeDriver(options);
    }
}

Custom Mobile Device Emulation

public static WebDriver createCustomMobileDriver(int width, int height, double pixelRatio) {
    ChromeOptions options = new ChromeOptions();

    // Custom device metrics
    Map<String, Object> deviceMetrics = new HashMap<>();
    deviceMetrics.put("width", width);
    deviceMetrics.put("height", height);
    deviceMetrics.put("pixelRatio", pixelRatio);

    Map<String, Object> mobileEmulation = new HashMap<>();
    mobileEmulation.put("deviceMetrics", deviceMetrics);
    mobileEmulation.put("userAgent", "Mozilla/5.0 (Linux; Android 11; SM-G991B) AppleWebKit/537.36");

    options.setExperimentalOption("mobileEmulation", mobileEmulation);

    return new ChromeDriver(options);
}

Handling Mobile-Specific Elements

Responsive Element Detection

import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;
import java.time.Duration;
import java.util.List;

public class ResponsiveElementHandler {
    private WebDriver driver;
    private WebDriverWait wait;

    public ResponsiveElementHandler(WebDriver driver) {
        this.driver = driver;
        this.wait = new WebDriverWait(driver, Duration.ofSeconds(10));
    }

    public WebElement findResponsiveElement(String mobileSelector, String desktopSelector) {
        // Try mobile selector first
        List<WebElement> mobileElements = driver.findElements(By.cssSelector(mobileSelector));
        if (!mobileElements.isEmpty() && mobileElements.get(0).isDisplayed()) {
            return mobileElements.get(0);
        }

        // Fallback to desktop selector
        return wait.until(ExpectedConditions.elementToBeClickable(By.cssSelector(desktopSelector)));
    }

    public void handleMobileNavigation() {
        try {
            // Look for mobile hamburger menu
            WebElement hamburgerMenu = driver.findElement(By.cssSelector(".mobile-menu-toggle, .hamburger, [aria-label='Menu']"));
            if (hamburgerMenu.isDisplayed()) {
                hamburgerMenu.click();
                Thread.sleep(500); // Wait for menu animation
            }
        } catch (Exception e) {
            // Desktop navigation is likely visible
            System.out.println("Mobile menu not found, using desktop navigation");
        }
    }
}

Touch and Swipe Gestures

import org.openqa.selenium.interactions.Actions;
import org.openqa.selenium.interactions.PointerInput;
import org.openqa.selenium.interactions.Sequence;
import java.time.Duration;
import java.util.Arrays;

public class MobileGestureHandler {
    private WebDriver driver;

    public MobileGestureHandler(WebDriver driver) {
        this.driver = driver;
    }

    public void swipeDown(int startX, int startY, int endY) {
        PointerInput finger = new PointerInput(PointerInput.Kind.TOUCH, "finger");
        Sequence swipe = new Sequence(finger, 1);

        swipe.addAction(finger.createPointerMove(Duration.ofMillis(0), 
            PointerInput.Origin.viewport(), startX, startY));
        swipe.addAction(finger.createPointerDown(PointerInput.MouseButton.LEFT.asArg()));
        swipe.addAction(finger.createPointerMove(Duration.ofMillis(600), 
            PointerInput.Origin.viewport(), startX, endY));
        swipe.addAction(finger.createPointerUp(PointerInput.MouseButton.LEFT.asArg()));

        driver.perform(Arrays.asList(swipe));
    }

    public void handleInfiniteScroll() {
        long lastHeight = (Long) ((JavascriptExecutor) driver)
            .executeScript("return document.body.scrollHeight");

        while (true) {
            // Scroll to bottom
            ((JavascriptExecutor) driver)
                .executeScript("window.scrollTo(0, document.body.scrollHeight);");

            // Wait for new content to load
            try {
                Thread.sleep(2000);
            } catch (InterruptedException e) {
                break;
            }

            // Check if new content loaded
            long newHeight = (Long) ((JavascriptExecutor) driver)
                .executeScript("return document.body.scrollHeight");

            if (newHeight == lastHeight) {
                break; // No more content to load
            }
            lastHeight = newHeight;
        }
    }
}

Complete Mobile Scraping Example

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;
import java.time.Duration;
import java.util.List;
import java.util.ArrayList;

public class MobileResponsiveScraper {
    private WebDriver driver;
    private WebDriverWait wait;

    public static void main(String[] args) {
        MobileResponsiveScraper scraper = new MobileResponsiveScraper();
        scraper.scrapeEcommerceProducts();
    }

    public void scrapeEcommerceProducts() {
        // Initialize mobile driver
        driver = createMobileDriver();
        wait = new WebDriverWait(driver, Duration.ofSeconds(15));

        try {
            driver.get("https://example-ecommerce.com");

            // Handle mobile-specific cookie banner
            handleMobileCookieBanner();

            // Navigate using mobile interface
            navigateToProductCategory();

            // Wait for mobile-optimized product grid
            waitForMobileContent();

            // Scrape products with mobile layout considerations
            List<Product> products = scrapeProductList();

            // Print results
            products.forEach(System.out::println);

        } finally {
            if (driver != null) {
                driver.quit();
            }
        }
    }

    private void handleMobileCookieBanner() {
        try {
            // Mobile cookie banners often have different selectors
            WebElement cookieButton = wait.until(ExpectedConditions.elementToBeClickable(
                By.cssSelector(".cookie-accept-mobile, .accept-cookies-btn, [data-mobile-cookie='accept']")));
            cookieButton.click();
        } catch (Exception e) {
            System.out.println("No mobile cookie banner found");
        }
    }

    private void navigateToProductCategory() {
        try {
            // Handle mobile hamburger menu
            WebElement mobileMenuToggle = driver.findElement(
                By.cssSelector(".mobile-menu-toggle, .hamburger-menu, [aria-label='Open menu']"));

            if (mobileMenuToggle.isDisplayed()) {
                mobileMenuToggle.click();
                Thread.sleep(500);

                // Find category in mobile menu
                WebElement categoryLink = wait.until(ExpectedConditions.elementToBeClickable(
                    By.xpath("//a[contains(text(), 'Electronics') or contains(@data-category, 'electronics')]")));
                categoryLink.click();
            }
        } catch (Exception e) {
            // Try desktop navigation as fallback
            WebElement desktopCategory = driver.findElement(By.linkText("Electronics"));
            desktopCategory.click();
        }
    }

    private void waitForMobileContent() {
        // Wait for mobile-specific content indicators
        wait.until(ExpectedConditions.or(
            ExpectedConditions.presenceOfElementLocated(By.cssSelector(".mobile-product-grid")),
            ExpectedConditions.presenceOfElementLocated(By.cssSelector(".product-list-mobile")),
            ExpectedConditions.presenceOfElementLocated(By.cssSelector("[data-mobile-products]"))
        ));

        // Additional wait for lazy loading
        ((JavascriptExecutor) driver).executeScript(
            "return document.readyState === 'complete' && " +
            "typeof jQuery !== 'undefined' ? jQuery.active === 0 : true;"
        );
    }

    private List<Product> scrapeProductList() {
        List<Product> products = new ArrayList<>();

        // Mobile-optimized product selectors
        List<WebElement> productCards = driver.findElements(By.cssSelector(
            ".mobile-product-card, .product-item-mobile, .mobile-grid-item, .product-card"
        ));

        for (WebElement card : productCards) {
            try {
                Product product = extractProductFromMobileCard(card);
                if (product != null) {
                    products.add(product);
                }
            } catch (Exception e) {
                System.out.println("Error extracting product: " + e.getMessage());
            }
        }

        return products;
    }

    private Product extractProductFromMobileCard(WebElement card) {
        try {
            // Mobile layouts often stack information vertically
            String name = extractText(card, ".product-name, .mobile-product-title, h3, h4");
            String price = extractText(card, ".price, .mobile-price, .cost, [data-price]");
            String imageUrl = extractImageUrl(card, ".product-image img, .mobile-product-img");
            String rating = extractText(card, ".rating, .stars, .mobile-rating, [data-rating]");

            return new Product(name, price, imageUrl, rating);
        } catch (Exception e) {
            return null;
        }
    }

    private String extractText(WebElement parent, String selector) {
        try {
            WebElement element = parent.findElement(By.cssSelector(selector));
            return element.getText().trim();
        } catch (Exception e) {
            return "";
        }
    }

    private String extractImageUrl(WebElement parent, String selector) {
        try {
            WebElement img = parent.findElement(By.cssSelector(selector));
            return img.getAttribute("src");
        } catch (Exception e) {
            return "";
        }
    }

    // Product data class
    private static class Product {
        private String name, price, imageUrl, rating;

        public Product(String name, String price, String imageUrl, String rating) {
            this.name = name;
            this.price = price;
            this.imageUrl = imageUrl;
            this.rating = rating;
        }

        @Override
        public String toString() {
            return String.format("Product{name='%s', price='%s', rating='%s'}", 
                name, price, rating);
        }
    }
}

Advanced Mobile Scraping Techniques

Viewport-Based Content Detection

public boolean isMobileViewport() {
    long viewportWidth = (Long) ((JavascriptExecutor) driver)
        .executeScript("return Math.max(document.documentElement.clientWidth, window.innerWidth || 0)");
    return viewportWidth <= 768; // Common mobile breakpoint
}

public void adaptToViewport() {
    if (isMobileViewport()) {
        // Use mobile-specific selectors and strategies
        handleMobileLayout();
    } else {
        // Use desktop selectors and strategies
        handleDesktopLayout();
    }
}

Progressive Web App (PWA) Support

public void handlePWAContent() {
    // Wait for service worker registration
    ((JavascriptExecutor) driver).executeScript(
        "return 'serviceWorker' in navigator && " +
        "navigator.serviceWorker.ready.then(() => true)"
    );

    // Handle app-like navigation
    WebElement backButton = driver.findElement(By.cssSelector("[role='button'][aria-label='Back']"));
    if (backButton.isDisplayed()) {
        // PWA-style navigation detected
        System.out.println("PWA interface detected");
    }
}

Performance Optimization for Mobile Scraping

Efficient Resource Management

public static ChromeOptions getOptimizedMobileOptions() {
    ChromeOptions options = new ChromeOptions();

    // Mobile emulation
    Map<String, String> mobileEmulation = new HashMap<>();
    mobileEmulation.put("deviceName", "iPhone 12 Pro");
    options.setExperimentalOption("mobileEmulation", mobileEmulation);

    // Performance optimizations
    options.addArguments("--disable-images"); // Faster loading
    options.addArguments("--disable-javascript"); // If JS not needed
    options.addArguments("--disable-plugins");
    options.addArguments("--disable-extensions");
    options.addArguments("--no-sandbox");
    options.addArguments("--disable-dev-shm-usage");
    options.addArguments("--disable-gpu");
    options.addArguments("--headless"); // Headless mode

    return options;
}

Best Practices for Mobile Web Scraping

  1. Test Multiple Device Emulations: Different devices may show different content
  2. Handle Touch Interactions: Use appropriate gesture handlers for mobile interfaces
  3. Wait for Mobile-Specific Loading: Mobile sites often have different loading patterns
  4. Respect Mobile Bandwidth: Be mindful of data usage and loading times
  5. Consider Mobile-First Design: Many sites prioritize mobile layouts

Similar to how Puppeteer handles viewport configuration for different screen sizes, Java's Selenium WebDriver provides comprehensive mobile emulation capabilities for responsive web scraping.

Handling Different Mobile Platforms

Android Emulation

public static WebDriver createAndroidDriver() {
    ChromeOptions options = new ChromeOptions();

    Map<String, String> mobileEmulation = new HashMap<>();
    mobileEmulation.put("deviceName", "Samsung Galaxy S21");
    options.setExperimentalOption("mobileEmulation", mobileEmulation);

    // Android-specific user agent
    options.addArguments("--user-agent=Mozilla/5.0 (Linux; Android 11; SM-G991B) AppleWebKit/537.36");

    return new ChromeDriver(options);
}

iOS Emulation

public static WebDriver createiOSDriver() {
    ChromeOptions options = new ChromeOptions();

    Map<String, String> mobileEmulation = new HashMap<>();
    mobileEmulation.put("deviceName", "iPhone 13 Pro Max");
    options.setExperimentalOption("mobileEmulation", mobileEmulation);

    // iOS-specific user agent
    options.addArguments("--user-agent=Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15");

    return new ChromeDriver(options);
}

When implementing mobile scraping solutions, it's important to understand how different frameworks handle mobile-responsive content. For instance, handling AJAX requests in mobile environments requires special consideration for touch-based interactions and mobile network conditions.

Conclusion

Scraping mobile-responsive websites with Java requires careful consideration of device emulation, responsive design patterns, and mobile-specific user interactions. By using Selenium WebDriver's mobile emulation capabilities and implementing proper mobile detection strategies, you can effectively extract data from sites that adapt their content based on device characteristics.

Key success factors include: - Proper mobile device emulation setup - Understanding responsive design breakpoints - Handling mobile-specific UI elements and navigation patterns - Implementing appropriate wait strategies for mobile content loading - Testing across multiple device configurations

For complex mobile scraping scenarios involving dynamic content loading, consider exploring advanced waiting strategies that can be adapted to Java Selenium implementations.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I handle anti-bot measures and detection avoidance in Java?
What is the best way to parse JSON responses in Java web scraping?
How can I implement user-agent rotation in Java web scraping?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon