Table of contents

How can I scrape data from websites that use GraphQL APIs in Java?

Scraping data from websites that use GraphQL APIs requires a different approach than traditional REST API scraping. GraphQL is a query language and runtime for APIs that allows clients to request exactly the data they need. In Java, you can effectively scrape GraphQL endpoints using HTTP clients and proper query construction.

Understanding GraphQL Basics

GraphQL APIs expose a single endpoint that accepts POST requests with query payloads. Unlike REST APIs with multiple endpoints, GraphQL uses a unified endpoint where you specify the exact data structure you want in your query.

Key GraphQL Concepts

  • Query: Read operations to fetch data
  • Mutation: Write operations to modify data
  • Schema: Defines the structure and types available
  • Introspection: Ability to query the schema itself

Setting Up Java Dependencies

First, add the necessary dependencies to your pom.xml for Maven projects:

<dependencies>
    <!-- HTTP Client -->
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.14</version>
    </dependency>

    <!-- JSON Processing -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.15.2</version>
    </dependency>

    <!-- Optional: GraphQL Java Client -->
    <dependency>
        <groupId>com.graphql-java</groupId>
        <artifactId>graphql-java-client</artifactId>
        <version>2023.03.29</version>
    </dependency>
</dependencies>

Basic GraphQL Query Implementation

Here's a fundamental example of scraping a GraphQL API using Apache HttpClient:

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;
import java.util.HashMap;
import java.util.Map;

public class GraphQLScraper {
    private static final String GRAPHQL_ENDPOINT = "https://api.example.com/graphql";
    private final CloseableHttpClient httpClient;
    private final ObjectMapper objectMapper;

    public GraphQLScraper() {
        this.httpClient = HttpClients.createDefault();
        this.objectMapper = new ObjectMapper();
    }

    public JsonNode executeQuery(String query, Map<String, Object> variables) throws Exception {
        HttpPost httpPost = new HttpPost(GRAPHQL_ENDPOINT);

        // Set headers
        httpPost.setHeader("Content-Type", "application/json");
        httpPost.setHeader("Accept", "application/json");

        // Create request body
        Map<String, Object> requestBody = new HashMap<>();
        requestBody.put("query", query);
        if (variables != null) {
            requestBody.put("variables", variables);
        }

        String jsonBody = objectMapper.writeValueAsString(requestBody);
        httpPost.setEntity(new StringEntity(jsonBody, "UTF-8"));

        try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
            String responseBody = EntityUtils.toString(response.getEntity());
            return objectMapper.readTree(responseBody);
        }
    }

    public void close() throws Exception {
        httpClient.close();
    }
}

Advanced GraphQL Scraping Examples

Scraping User Data with Variables

import java.util.ArrayList;
import java.util.List;

public class UserDataScraper {
    private final GraphQLScraper scraper;

    public UserDataScraper() {
        this.scraper = new GraphQLScraper();
    }

    public List<User> scrapeUsers(int limit, String cursor) throws Exception {
        String query = """
            query GetUsers($limit: Int!, $cursor: String) {
                users(first: $limit, after: $cursor) {
                    edges {
                        node {
                            id
                            name
                            email
                            createdAt
                            profile {
                                bio
                                avatar
                            }
                        }
                        cursor
                    }
                    pageInfo {
                        hasNextPage
                        endCursor
                    }
                }
            }
        """;

        Map<String, Object> variables = new HashMap<>();
        variables.put("limit", limit);
        if (cursor != null) {
            variables.put("cursor", cursor);
        }

        JsonNode response = scraper.executeQuery(query, variables);
        return parseUsers(response);
    }

    private List<User> parseUsers(JsonNode response) {
        List<User> users = new ArrayList<>();
        JsonNode edges = response.get("data").get("users").get("edges");

        for (JsonNode edge : edges) {
            JsonNode node = edge.get("node");
            User user = new User();
            user.setId(node.get("id").asText());
            user.setName(node.get("name").asText());
            user.setEmail(node.get("email").asText());
            // Parse additional fields...
            users.add(user);
        }

        return users;
    }
}

Handling Authentication

Many GraphQL APIs require authentication. Here's how to handle different authentication methods:

public class AuthenticatedGraphQLScraper extends GraphQLScraper {
    private String authToken;

    public void setAuthToken(String token) {
        this.authToken = token;
    }

    @Override
    public JsonNode executeQuery(String query, Map<String, Object> variables) throws Exception {
        HttpPost httpPost = new HttpPost(GRAPHQL_ENDPOINT);

        // Set authentication headers
        httpPost.setHeader("Authorization", "Bearer " + authToken);
        httpPost.setHeader("Content-Type", "application/json");

        // Create request body
        Map<String, Object> requestBody = new HashMap<>();
        requestBody.put("query", query);
        if (variables != null) {
            requestBody.put("variables", variables);
        }

        String jsonBody = objectMapper.writeValueAsString(requestBody);
        httpPost.setEntity(new StringEntity(jsonBody, "UTF-8"));

        try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
            String responseBody = EntityUtils.toString(response.getEntity());
            return objectMapper.readTree(responseBody);
        }
    }

    // Method for API key authentication
    public void setApiKey(String apiKey) {
        // Implementation for API key auth
    }

    // Method for session-based authentication
    public void setCookies(String cookies) {
        // Implementation for cookie-based auth
    }
}

Schema Introspection

Before scraping, you often need to understand the API structure. GraphQL's introspection feature allows you to query the schema:

public class SchemaIntrospector {
    private final GraphQLScraper scraper;

    public SchemaIntrospector() {
        this.scraper = new GraphQLScraper();
    }

    public JsonNode getSchema() throws Exception {
        String introspectionQuery = """
            query IntrospectionQuery {
                __schema {
                    types {
                        name
                        kind
                        description
                        fields {
                            name
                            type {
                                name
                                kind
                            }
                        }
                    }
                }
            }
        """;

        return scraper.executeQuery(introspectionQuery, null);
    }

    public List<String> getAvailableTypes() throws Exception {
        JsonNode schema = getSchema();
        List<String> types = new ArrayList<>();

        JsonNode typesNode = schema.get("data").get("__schema").get("types");
        for (JsonNode type : typesNode) {
            types.add(type.get("name").asText());
        }

        return types;
    }
}

Handling Pagination and Large Datasets

GraphQL APIs often use cursor-based pagination. Here's how to handle it effectively:

public class PaginatedScraper {
    private final GraphQLScraper scraper;
    private static final int PAGE_SIZE = 100;

    public PaginatedScraper() {
        this.scraper = new GraphQLScraper();
    }

    public List<JsonNode> scrapeAllPages(String baseQuery) throws Exception {
        List<JsonNode> allResults = new ArrayList<>();
        String cursor = null;
        boolean hasNextPage = true;

        while (hasNextPage) {
            Map<String, Object> variables = new HashMap<>();
            variables.put("first", PAGE_SIZE);
            if (cursor != null) {
                variables.put("after", cursor);
            }

            JsonNode response = scraper.executeQuery(baseQuery, variables);
            JsonNode pageInfo = response.get("data").get("connection").get("pageInfo");

            // Extract data from current page
            JsonNode edges = response.get("data").get("connection").get("edges");
            allResults.add(edges);

            // Check for next page
            hasNextPage = pageInfo.get("hasNextPage").asBoolean();
            if (hasNextPage) {
                cursor = pageInfo.get("endCursor").asText();
            }

            // Add delay to respect rate limits
            Thread.sleep(1000);
        }

        return allResults;
    }
}

Error Handling and Rate Limiting

Implement robust error handling and rate limiting for production scraping:

public class RobustGraphQLScraper {
    private final GraphQLScraper scraper;
    private final int maxRetries = 3;
    private final long baseDelay = 1000; // 1 second

    public RobustGraphQLScraper() {
        this.scraper = new GraphQLScraper();
    }

    public JsonNode executeWithRetry(String query, Map<String, Object> variables) throws Exception {
        Exception lastException = null;

        for (int attempt = 0; attempt < maxRetries; attempt++) {
            try {
                JsonNode response = scraper.executeQuery(query, variables);

                // Check for GraphQL errors
                if (response.has("errors")) {
                    JsonNode errors = response.get("errors");
                    throw new GraphQLException("GraphQL errors: " + errors.toString());
                }

                return response;

            } catch (Exception e) {
                lastException = e;

                if (attempt < maxRetries - 1) {
                    long delay = baseDelay * (long) Math.pow(2, attempt); // Exponential backoff
                    Thread.sleep(delay);
                }
            }
        }

        throw new Exception("Failed after " + maxRetries + " attempts", lastException);
    }
}

class GraphQLException extends Exception {
    public GraphQLException(String message) {
        super(message);
    }
}

Working with Complex Data Types

Handle nested objects and arrays in GraphQL responses:

public class ComplexDataProcessor {

    public void processNestedData(JsonNode data) {
        JsonNode posts = data.get("data").get("user").get("posts").get("edges");

        for (JsonNode postEdge : posts) {
            JsonNode post = postEdge.get("node");

            // Extract post data
            String title = post.get("title").asText();
            String content = post.get("content").asText();

            // Handle nested comments
            JsonNode comments = post.get("comments").get("edges");
            for (JsonNode commentEdge : comments) {
                JsonNode comment = commentEdge.get("node");
                String commentText = comment.get("text").asText();
                String authorName = comment.get("author").get("name").asText();

                // Process comment data...
                System.out.println("Comment: " + commentText + " by " + authorName);
            }
        }
    }
}

Best Practices for GraphQL Scraping

1. Optimize Query Efficiency

Request only the fields you need to minimize bandwidth and processing time:

// Good: Specific fields only
String efficientQuery = """
    query {
        users {
            id
            name
            email
        }
    }
""";

// Avoid: Requesting unnecessary nested data
String inefficientQuery = """
    query {
        users {
            id
            name
            email
            posts {
                title
                content
                comments {
                    text
                    author {
                        name
                        profile {
                            bio
                            avatar
                        }
                    }
                }
            }
        }
    }
""";

2. Implement Connection Pooling

import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;

public class PooledGraphQLScraper {
    private final PoolingHttpClientConnectionManager connectionManager;
    private final CloseableHttpClient httpClient;

    public PooledGraphQLScraper() {
        connectionManager = new PoolingHttpClientConnectionManager();
        connectionManager.setMaxTotal(20);
        connectionManager.setDefaultMaxPerRoute(10);

        httpClient = HttpClients.custom()
            .setConnectionManager(connectionManager)
            .build();
    }
}

3. Monitor Rate Limits

public class RateLimitedScraper {
    private final GraphQLScraper scraper;
    private long lastRequestTime = 0;
    private final long minInterval = 1000; // 1 second between requests

    public RateLimitedScraper() {
        this.scraper = new GraphQLScraper();
    }

    public JsonNode executeQuery(String query, Map<String, Object> variables) throws Exception {
        long currentTime = System.currentTimeMillis();
        long timeSinceLastRequest = currentTime - lastRequestTime;

        if (timeSinceLastRequest < minInterval) {
            Thread.sleep(minInterval - timeSinceLastRequest);
        }

        JsonNode response = scraper.executeQuery(query, variables);
        lastRequestTime = System.currentTimeMillis();

        return response;
    }
}

Integration with Modern Java Tools

When working with complex GraphQL scraping scenarios, you might also need to handle JavaScript-heavy applications. For such cases, consider integrating your Java scraper with tools that can handle dynamic content that loads after page load or monitor network requests to identify GraphQL endpoints.

Using Modern HTTP Clients

For Java 11+, you can also use the built-in HTTP client:

import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.time.Duration;

public class ModernGraphQLScraper {
    private final HttpClient httpClient;
    private final ObjectMapper objectMapper;

    public ModernGraphQLScraper() {
        this.httpClient = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(10))
            .build();
        this.objectMapper = new ObjectMapper();
    }

    public JsonNode executeQuery(String query, Map<String, Object> variables) throws Exception {
        Map<String, Object> requestBody = new HashMap<>();
        requestBody.put("query", query);
        if (variables != null) {
            requestBody.put("variables", variables);
        }

        String jsonBody = objectMapper.writeValueAsString(requestBody);

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.example.com/graphql"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(jsonBody))
            .build();

        HttpResponse<String> response = httpClient.send(request, 
            HttpResponse.BodyHandlers.ofString());

        return objectMapper.readTree(response.body());
    }
}

Conclusion

Scraping GraphQL APIs in Java requires understanding the GraphQL query language and implementing proper HTTP communication. Key success factors include:

  • Using appropriate HTTP clients with proper headers
  • Constructing well-formed GraphQL queries and mutations
  • Handling authentication and rate limiting
  • Implementing robust error handling and retry logic
  • Optimizing queries for efficiency
  • Managing pagination for large datasets

With these techniques and examples, you can effectively scrape data from GraphQL APIs while maintaining good performance and reliability. Remember to always respect the API's terms of service and implement appropriate delays to avoid overwhelming the target servers.

For complex scenarios involving JavaScript-heavy applications that expose GraphQL endpoints, you may need to combine these Java techniques with browser automation tools to handle authentication in dynamic environments.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon