Table of contents

How do I Implement Retry Logic with HttpClient (C#)?

Implementing retry logic with HttpClient in C# is essential for building resilient applications that can handle transient failures like network timeouts, temporary server issues, or rate limiting. This guide covers multiple approaches to implementing retry logic, from simple custom implementations to advanced patterns using the Polly library.

Why Retry Logic Matters in Web Scraping

When making HTTP requests for web scraping or API calls, temporary failures are common:

  • Network timeouts - Temporary connectivity issues
  • Rate limiting - Server returning 429 (Too Many Requests)
  • Server errors - 500-level errors that may resolve on retry
  • DNS resolution failures - Temporary DNS issues
  • Connection resets - Network interruptions

Without retry logic, your application would fail immediately on these transient errors, reducing reliability and data collection success rates.

Method 1: Using Polly (Recommended)

Polly is a .NET resilience and transient-fault-handling library that provides retry, circuit breaker, timeout, and other patterns. It's the industry-standard approach for implementing retry logic in C#.

Installing Polly

dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly

Basic Retry with Polly

using Polly;
using Polly.Extensions.Http;
using System.Net.Http;

public class HttpClientRetryExample
{
    private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError() // Handles 5xx and 408
            .OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
            .WaitAndRetryAsync(
                retryCount: 3,
                sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
                onRetry: (outcome, timespan, retryAttempt, context) =>
                {
                    Console.WriteLine($"Retry {retryAttempt} after {timespan.TotalSeconds}s due to {outcome.Result?.StatusCode}");
                });
    }

    public static async Task<string> FetchDataWithRetry(string url)
    {
        var retryPolicy = GetRetryPolicy();

        using var httpClient = new HttpClient();
        httpClient.Timeout = TimeSpan.FromSeconds(30);

        var response = await retryPolicy.ExecuteAsync(async () =>
        {
            return await httpClient.GetAsync(url);
        });

        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Exponential Backoff with Jitter

Adding jitter prevents retry storms when multiple clients retry simultaneously:

using Polly;
using Polly.Contrib.WaitAndRetry;

public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicyWithJitter()
{
    var delay = Backoff.DecorrelatedJitterBackoffV2(
        medianFirstRetryDelay: TimeSpan.FromSeconds(1),
        retryCount: 5);

    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
        .WaitAndRetryAsync(
            delay,
            onRetry: (outcome, timespan, retryAttempt, context) =>
            {
                Console.WriteLine($"Retry {retryAttempt} after {timespan.TotalSeconds:F2}s");
            });
}

Configuring HttpClient with Dependency Injection

For production applications, configure Polly policies with HttpClientFactory:

using Microsoft.Extensions.DependencyInjection;
using Polly;
using Polly.Extensions.Http;

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddHttpClient("WebScrapingClient")
            .SetHandlerLifetime(TimeSpan.FromMinutes(5))
            .AddPolicyHandler(GetRetryPolicy())
            .AddPolicyHandler(GetCircuitBreakerPolicy());
    }

    private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
            .WaitAndRetryAsync(
                retryCount: 3,
                sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
    }

    private static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .CircuitBreakerAsync(
                handledEventsAllowedBeforeBreaking: 5,
                durationOfBreak: TimeSpan.FromSeconds(30));
    }
}

// Usage in a service
public class WebScrapingService
{
    private readonly IHttpClientFactory _clientFactory;

    public WebScrapingService(IHttpClientFactory clientFactory)
    {
        _clientFactory = clientFactory;
    }

    public async Task<string> ScrapeWebsite(string url)
    {
        var client = _clientFactory.CreateClient("WebScrapingClient");
        var response = await client.GetAsync(url);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Method 2: Custom Retry Implementation

If you prefer not to use external libraries, you can implement retry logic manually:

using System;
using System.Net.Http;
using System.Threading.Tasks;

public class CustomRetryHandler
{
    private readonly HttpClient _httpClient;
    private readonly int _maxRetries;
    private readonly TimeSpan _initialDelay;

    public CustomRetryHandler(int maxRetries = 3, TimeSpan? initialDelay = null)
    {
        _httpClient = new HttpClient();
        _maxRetries = maxRetries;
        _initialDelay = initialDelay ?? TimeSpan.FromSeconds(1);
    }

    public async Task<HttpResponseMessage> GetWithRetry(string url)
    {
        Exception lastException = null;

        for (int attempt = 0; attempt <= _maxRetries; attempt++)
        {
            try
            {
                var response = await _httpClient.GetAsync(url);

                // Retry on specific status codes
                if (response.IsSuccessStatusCode ||
                    (!IsTransientError(response.StatusCode) && attempt == _maxRetries))
                {
                    return response;
                }

                if (attempt < _maxRetries)
                {
                    var delay = CalculateDelay(attempt);
                    Console.WriteLine($"Request failed with {response.StatusCode}. Retrying in {delay.TotalSeconds}s...");
                    await Task.Delay(delay);
                }
            }
            catch (HttpRequestException ex)
            {
                lastException = ex;

                if (attempt < _maxRetries)
                {
                    var delay = CalculateDelay(attempt);
                    Console.WriteLine($"Request failed: {ex.Message}. Retrying in {delay.TotalSeconds}s...");
                    await Task.Delay(delay);
                }
            }
            catch (TaskCanceledException ex)
            {
                lastException = ex;
                Console.WriteLine($"Request timeout on attempt {attempt + 1}");

                if (attempt == _maxRetries)
                {
                    throw new HttpRequestException($"Request failed after {_maxRetries} retries", ex);
                }

                await Task.Delay(CalculateDelay(attempt));
            }
        }

        throw new HttpRequestException($"Request failed after {_maxRetries} retries", lastException);
    }

    private bool IsTransientError(System.Net.HttpStatusCode statusCode)
    {
        return statusCode == System.Net.HttpStatusCode.RequestTimeout ||
               statusCode == System.Net.HttpStatusCode.TooManyRequests ||
               ((int)statusCode >= 500 && (int)statusCode < 600);
    }

    private TimeSpan CalculateDelay(int attempt)
    {
        // Exponential backoff: 1s, 2s, 4s, 8s...
        return TimeSpan.FromSeconds(_initialDelay.TotalSeconds * Math.Pow(2, attempt));
    }
}

// Usage
public async Task Example()
{
    var handler = new CustomRetryHandler(maxRetries: 3);
    var response = await handler.GetWithRetry("https://example.com");
    var content = await response.Content.ReadAsStringAsync();
}

Method 3: DelegatingHandler for Reusable Retry Logic

Create a custom DelegatingHandler that can be added to the HttpClient pipeline:

using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

public class RetryDelegatingHandler : DelegatingHandler
{
    private readonly int _maxRetries;

    public RetryDelegatingHandler(int maxRetries = 3) : base(new HttpClientHandler())
    {
        _maxRetries = maxRetries;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        for (int i = 0; i <= _maxRetries; i++)
        {
            try
            {
                var response = await base.SendAsync(request, cancellationToken);

                if (response.IsSuccessStatusCode || i == _maxRetries)
                {
                    return response;
                }

                if (ShouldRetry(response.StatusCode))
                {
                    var delay = TimeSpan.FromSeconds(Math.Pow(2, i));
                    await Task.Delay(delay, cancellationToken);
                    continue;
                }

                return response;
            }
            catch (HttpRequestException) when (i < _maxRetries)
            {
                var delay = TimeSpan.FromSeconds(Math.Pow(2, i));
                await Task.Delay(delay, cancellationToken);
            }
        }

        throw new HttpRequestException($"Request failed after {_maxRetries} retries");
    }

    private bool ShouldRetry(System.Net.HttpStatusCode statusCode)
    {
        return statusCode == System.Net.HttpStatusCode.RequestTimeout ||
               statusCode == System.Net.HttpStatusCode.TooManyRequests ||
               (int)statusCode >= 500;
    }
}

// Usage
var httpClient = new HttpClient(new RetryDelegatingHandler(maxRetries: 3))
{
    Timeout = TimeSpan.FromSeconds(30)
};

var response = await httpClient.GetAsync("https://example.com");

Handling Rate Limiting with Retry-After Header

Many APIs return a Retry-After header when rate limited. Here's how to respect it:

public static async Task<HttpResponseMessage> GetWithRateLimitRetry(
    HttpClient client,
    string url,
    int maxRetries = 3)
{
    for (int attempt = 0; attempt <= maxRetries; attempt++)
    {
        var response = await client.GetAsync(url);

        if (response.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
        {
            if (response.Headers.RetryAfter?.Delta.HasValue == true)
            {
                var retryAfter = response.Headers.RetryAfter.Delta.Value;
                Console.WriteLine($"Rate limited. Retrying after {retryAfter.TotalSeconds}s");
                await Task.Delay(retryAfter);
                continue;
            }

            // Default backoff if no Retry-After header
            await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, attempt)));
            continue;
        }

        return response;
    }

    throw new HttpRequestException("Rate limit exceeded after multiple retries");
}

Best Practices for Retry Logic

  1. Use exponential backoff - Prevents overwhelming the server with rapid retries
  2. Add jitter - Prevents retry storms in distributed systems
  3. Set maximum retry limits - Avoid infinite retry loops
  4. Log retry attempts - Monitor retry patterns for debugging
  5. Implement circuit breakers - Prevent cascading failures when services are down
  6. Respect Retry-After headers - Honor server-specified retry delays
  7. Configure appropriate timeouts - Balance between waiting and failing fast
  8. Use HttpClientFactory - Properly manage HttpClient lifetime
  9. Handle specific status codes - Only retry transient errors (5xx, 408, 429)
  10. Consider idempotency - Ensure retried requests are safe to repeat

Combining Retry with Timeout Policies

For web scraping applications, you often want both retry and timeout handling:

using Polly;
using Polly.Timeout;

public static IAsyncPolicy<HttpResponseMessage> GetCombinedPolicy()
{
    var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
        TimeSpan.FromSeconds(30),
        TimeoutStrategy.Pessimistic);

    var retryPolicy = HttpPolicyExtensions
        .HandleTransientHttpError()
        .Or<TimeoutRejectedException>()
        .WaitAndRetryAsync(
            retryCount: 3,
            sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

    // Wrap policies: retry wraps timeout
    return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}

Comparison with Other Languages

While this article focuses on C#, similar retry patterns exist in other languages:

Python with requests and urllib3:

from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
import requests

session = requests.Session()
retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

response = session.get('https://example.com')

JavaScript with axios:

const axios = require('axios');
const axiosRetry = require('axios-retry');

axiosRetry(axios, {
  retries: 3,
  retryDelay: axiosRetry.exponentialDelay,
  retryCondition: (error) => {
    return axiosRetry.isNetworkOrIdempotentRequestError(error)
      || error.response?.status === 429;
  }
});

const response = await axios.get('https://example.com');

Conclusion

Implementing retry logic with HttpClient in C# is crucial for building resilient web scraping and API integration applications. The Polly library provides the most robust and feature-rich approach, offering retry policies, circuit breakers, and advanced patterns like jitter and bulkhead isolation. For simpler scenarios, custom implementations using DelegatingHandler or basic retry loops can also be effective.

When building web scraping applications, combining retry logic with proper timeout handling and error handling patterns ensures your application can gracefully handle network failures, rate limiting, and temporary service disruptions while maintaining data collection reliability.

Remember to always respect rate limits, implement exponential backoff, and monitor your retry patterns to optimize the balance between resilience and resource usage.

Try WebScraping.AI for Your Web Scraping Needs

Looking for a powerful web scraping solution? WebScraping.AI provides an LLM-powered API that combines Chromium JavaScript rendering with rotating proxies for reliable data extraction.

Key Features:

  • AI-powered extraction: Ask questions about web pages or extract structured data fields
  • JavaScript rendering: Full Chromium browser support for dynamic content
  • Rotating proxies: Datacenter and residential proxies from multiple countries
  • Easy integration: Simple REST API with SDKs for Python, Ruby, PHP, and more
  • Reliable & scalable: Built for developers who need consistent results

Getting Started:

Get page content with AI analysis:

curl "https://api.webscraping.ai/ai/question?url=https://example.com&question=What is the main topic?&api_key=YOUR_API_KEY"

Extract structured data:

curl "https://api.webscraping.ai/ai/fields?url=https://example.com&fields[title]=Page title&fields[price]=Product price&api_key=YOUR_API_KEY"

Try in request builder

Related Questions

How do I set the User-Agent header in HttpClient (C#)?
How do I set the Content-Type header for HttpClient (C#) requests?
How do I read the HTTP status code from an HttpClient (C#) response?

Get Started Now

WebScraping.AI provides rotating proxies, Chromium rendering and built-in HTML parser for web scraping
Icon