Table of contents

How do I implement retry logic for failed HTTP requests in Java?

Implementing retry logic for failed HTTP requests is crucial for building resilient Java applications that can handle temporary network issues, server timeouts, and transient errors. This guide covers various approaches to implement retry mechanisms, from basic manual implementations to sophisticated solutions using popular libraries.

Why Implement Retry Logic?

HTTP requests can fail for various reasons: - Network connectivity issues - Server overload (5xx errors) - Rate limiting (429 Too Many Requests) - Temporary DNS resolution failures - Connection timeouts

Implementing proper retry logic helps your application gracefully handle these transient failures and improve overall reliability.

Basic Retry Implementation

Here's a simple retry mechanism using Java's built-in HttpClient:

import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.time.Duration;

public class BasicRetryClient {
    private static final int MAX_RETRIES = 3;
    private static final Duration INITIAL_DELAY = Duration.ofSeconds(1);

    private final HttpClient client;

    public BasicRetryClient() {
        this.client = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(10))
            .build();
    }

    public HttpResponse<String> getWithRetry(String url) throws Exception {
        Exception lastException = null;

        for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
            try {
                HttpRequest request = HttpRequest.newBuilder()
                    .uri(URI.create(url))
                    .timeout(Duration.ofSeconds(30))
                    .GET()
                    .build();

                HttpResponse<String> response = client.send(request, 
                    HttpResponse.BodyHandlers.ofString());

                // Check if response is successful
                if (response.statusCode() >= 200 && response.statusCode() < 300) {
                    return response;
                }

                // For 4xx errors (except 429), don't retry
                if (response.statusCode() >= 400 && response.statusCode() < 500 
                    && response.statusCode() != 429) {
                    throw new RuntimeException("Client error: " + response.statusCode());
                }

            } catch (Exception e) {
                lastException = e;

                if (attempt < MAX_RETRIES) {
                    long delayMs = INITIAL_DELAY.toMillis() * (long) Math.pow(2, attempt);
                    Thread.sleep(delayMs);
                }
            }
        }

        throw new RuntimeException("Request failed after " + (MAX_RETRIES + 1) + 
            " attempts", lastException);
    }
}

Advanced Retry with Exponential Backoff

For more sophisticated retry logic, implement exponential backoff with jitter:

import java.util.Random;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class ExponentialBackoffRetry {
    private static final int MAX_RETRIES = 5;
    private static final long BASE_DELAY_MS = 1000;
    private static final long MAX_DELAY_MS = 30000;
    private static final double JITTER_FACTOR = 0.1;

    private final HttpClient client;
    private final ScheduledExecutorService scheduler;
    private final Random random;

    public ExponentialBackoffRetry() {
        this.client = HttpClient.newHttpClient();
        this.scheduler = Executors.newScheduledThreadPool(10);
        this.random = new Random();
    }

    public CompletableFuture<HttpResponse<String>> executeWithRetry(HttpRequest request) {
        return executeWithRetry(request, 0);
    }

    private CompletableFuture<HttpResponse<String>> executeWithRetry(
            HttpRequest request, int attempt) {

        return client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
            .thenCompose(response -> {
                if (isSuccessful(response)) {
                    return CompletableFuture.completedFuture(response);
                } else if (shouldRetry(response, attempt)) {
                    long delay = calculateDelay(attempt);
                    return scheduleRetry(request, attempt + 1, delay);
                } else {
                    return CompletableFuture.failedFuture(
                        new RuntimeException("Request failed with status: " + 
                            response.statusCode()));
                }
            })
            .exceptionallyCompose(throwable -> {
                if (attempt < MAX_RETRIES && isRetriableException(throwable)) {
                    long delay = calculateDelay(attempt);
                    return scheduleRetry(request, attempt + 1, delay);
                } else {
                    return CompletableFuture.failedFuture(throwable);
                }
            });
    }

    private CompletableFuture<HttpResponse<String>> scheduleRetry(
            HttpRequest request, int attempt, long delayMs) {

        CompletableFuture<HttpResponse<String>> future = new CompletableFuture<>();

        scheduler.schedule(() -> {
            executeWithRetry(request, attempt)
                .whenComplete((response, throwable) -> {
                    if (throwable != null) {
                        future.completeExceptionally(throwable);
                    } else {
                        future.complete(response);
                    }
                });
        }, delayMs, TimeUnit.MILLISECONDS);

        return future;
    }

    private long calculateDelay(int attempt) {
        long delay = BASE_DELAY_MS * (long) Math.pow(2, attempt);
        delay = Math.min(delay, MAX_DELAY_MS);

        // Add jitter to prevent thundering herd
        double jitter = (random.nextDouble() * 2 - 1) * JITTER_FACTOR;
        delay = (long) (delay * (1 + jitter));

        return delay;
    }

    private boolean isSuccessful(HttpResponse<String> response) {
        return response.statusCode() >= 200 && response.statusCode() < 300;
    }

    private boolean shouldRetry(HttpResponse<String> response, int attempt) {
        if (attempt >= MAX_RETRIES) return false;

        int statusCode = response.statusCode();
        return statusCode == 429 || // Rate limited
               statusCode == 502 || // Bad Gateway
               statusCode == 503 || // Service Unavailable
               statusCode == 504;   // Gateway Timeout
    }

    private boolean isRetriableException(Throwable throwable) {
        return throwable instanceof java.net.ConnectException ||
               throwable instanceof java.net.SocketTimeoutException ||
               throwable instanceof java.net.UnknownHostException;
    }
}

Using OkHttp with Retry Interceptor

OkHttp provides excellent support for retry logic through interceptors:

import okhttp3.*;
import java.io.IOException;
import java.util.concurrent.TimeUnit;

public class OkHttpRetryInterceptor implements Interceptor {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_DELAY_MS = 1000;

    @Override
    public Response intercept(Chain chain) throws IOException {
        Request request = chain.request();
        Response response = null;
        IOException lastException = null;

        for (int attempt = 0; attempt <= MAX_RETRIES; attempt++) {
            try {
                if (response != null) {
                    response.close();
                }

                response = chain.proceed(request);

                if (response.isSuccessful() || !shouldRetry(response.code())) {
                    return response;
                }

                if (attempt < MAX_RETRIES) {
                    long delay = INITIAL_DELAY_MS * (long) Math.pow(2, attempt);
                    try {
                        Thread.sleep(delay);
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                        throw new IOException("Interrupted during retry", e);
                    }
                }

            } catch (IOException e) {
                lastException = e;

                if (attempt < MAX_RETRIES && isRetriableException(e)) {
                    long delay = INITIAL_DELAY_MS * (long) Math.pow(2, attempt);
                    try {
                        Thread.sleep(delay);
                    } catch (InterruptedException ie) {
                        Thread.currentThread().interrupt();
                        throw new IOException("Interrupted during retry", ie);
                    }
                } else {
                    throw e;
                }
            }
        }

        if (lastException != null) {
            throw lastException;
        }

        return response;
    }

    private boolean shouldRetry(int statusCode) {
        return statusCode == 429 || statusCode >= 500;
    }

    private boolean isRetriableException(IOException e) {
        return e instanceof java.net.SocketTimeoutException ||
               e instanceof java.net.ConnectException ||
               e instanceof java.net.UnknownHostException;
    }
}

// Usage example
public class OkHttpRetryClient {
    private final OkHttpClient client;

    public OkHttpRetryClient() {
        this.client = new OkHttpClient.Builder()
            .addInterceptor(new OkHttpRetryInterceptor())
            .connectTimeout(10, TimeUnit.SECONDS)
            .readTimeout(30, TimeUnit.SECONDS)
            .writeTimeout(30, TimeUnit.SECONDS)
            .build();
    }

    public String get(String url) throws IOException {
        Request request = new Request.Builder()
            .url(url)
            .build();

        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) {
                throw new IOException("Unexpected response: " + response);
            }
            return response.body().string();
        }
    }
}

Using Spring Retry

For Spring applications, leverage the @Retryable annotation:

import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.retry.annotation.Recover;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.client.ResourceAccessException;
import org.springframework.web.client.HttpServerErrorException;

@Service
public class RetryableHttpService {

    private final RestTemplate restTemplate;

    public RetryableHttpService(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @Retryable(
        value = {ResourceAccessException.class, HttpServerErrorException.class},
        maxAttempts = 3,
        backoff = @Backoff(
            delay = 1000,
            multiplier = 2,
            maxDelay = 10000
        )
    )
    public String fetchData(String url) {
        return restTemplate.getForObject(url, String.class);
    }

    @Recover
    public String handleFailure(Exception e, String url) {
        // Fallback logic when all retries fail
        return "Failed to fetch data from " + url + " after retries: " + e.getMessage();
    }
}

Configuring Spring Retry:

@Configuration
@EnableRetry
public class RetryConfig {

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }

    @Bean
    public RetryTemplate retryTemplate() {
        RetryTemplate retryTemplate = new RetryTemplate();

        ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
        backOffPolicy.setInitialInterval(1000);
        backOffPolicy.setMultiplier(2);
        backOffPolicy.setMaxInterval(30000);

        SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
        retryPolicy.setMaxAttempts(3);

        retryTemplate.setBackOffPolicy(backOffPolicy);
        retryTemplate.setRetryPolicy(retryPolicy);

        return retryTemplate;
    }
}

Best Practices for Retry Logic

1. Choose Appropriate Retry Conditions

Only retry on transient errors: - Network timeouts - Connection failures - 5xx server errors - 429 (Too Many Requests)

Don't retry on: - 4xx client errors (except 429) - Authentication failures - Malformed requests

2. Implement Circuit Breaker Pattern

Combine retry logic with circuit breakers to prevent cascading failures:

import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;

public class CircuitBreaker {
    private enum State { CLOSED, OPEN, HALF_OPEN }

    private State state = State.CLOSED;
    private final AtomicInteger failureCount = new AtomicInteger(0);
    private final AtomicLong lastFailureTime = new AtomicLong(0);

    private final int failureThreshold;
    private final long timeoutMs;

    public CircuitBreaker(int failureThreshold, long timeoutMs) {
        this.failureThreshold = failureThreshold;
        this.timeoutMs = timeoutMs;
    }

    public boolean canExecute() {
        if (state == State.OPEN) {
            if (System.currentTimeMillis() - lastFailureTime.get() > timeoutMs) {
                state = State.HALF_OPEN;
                return true;
            }
            return false;
        }
        return true;
    }

    public void recordSuccess() {
        failureCount.set(0);
        state = State.CLOSED;
    }

    public void recordFailure() {
        failureCount.incrementAndGet();
        lastFailureTime.set(System.currentTimeMillis());

        if (failureCount.get() >= failureThreshold) {
            state = State.OPEN;
        }
    }
}

3. Use Jitter in Exponential Backoff

Adding randomness prevents the "thundering herd" problem when multiple clients retry simultaneously.

4. Set Maximum Retry Limits

Always implement maximum retry counts and total timeout limits to prevent infinite retry loops.

5. Log Retry Attempts

Implement comprehensive logging to monitor retry behavior:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoggingRetryClient {
    private static final Logger logger = LoggerFactory.getLogger(LoggingRetryClient.class);

    public void executeWithLogging(String url, int attempt) {
        try {
            // Execute request
            logger.info("Executing request to {} (attempt {})", url, attempt + 1);

        } catch (Exception e) {
            if (attempt < MAX_RETRIES) {
                logger.warn("Request to {} failed on attempt {}, retrying: {}", 
                    url, attempt + 1, e.getMessage());
            } else {
                logger.error("Request to {} failed after {} attempts", 
                    url, attempt + 1, e);
            }
        }
    }
}

Testing Retry Logic

Create comprehensive tests for your retry mechanisms:

@Test
public void testRetryOnTimeout() {
    // Mock server that fails first two requests, succeeds on third
    MockWebServer server = new MockWebServer();
    server.enqueue(new MockResponse().setSocketPolicy(SocketPolicy.NO_RESPONSE));
    server.enqueue(new MockResponse().setResponseCode(500));
    server.enqueue(new MockResponse().setBody("Success"));

    RetryClient client = new RetryClient();
    String result = client.getWithRetry(server.url("/test").toString());

    assertEquals("Success", result);
    assertEquals(3, server.getRequestCount());
}

Conclusion

Implementing robust retry logic is essential for building resilient Java applications. Whether you choose a simple manual implementation, leverage libraries like OkHttp, or use frameworks like Spring Retry, the key is to handle transient failures gracefully while avoiding infinite retry loops. Remember to implement proper backoff strategies, circuit breakers, and comprehensive logging to ensure your retry mechanisms work effectively in production environments.

When implementing web scraping applications that need to handle timeouts in Puppeteer or handle errors in Puppeteer, similar retry patterns can be applied to create more reliable scraping systems that gracefully handle network issues and temporary server failures.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

What is the best way to handle timeouts in Java web scraping applications?
How can I scrape data from password-protected websites using Java?
How do I extract specific elements from HTML using CSS selectors in Java?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon