WebSpark.HttpClientUtility

Drop-in HttpClient wrapper with Polly resilience, response caching, and OpenTelemetry for .NET 8-10 LTS APIs—configured in one line

Live Site: https://httpclientutility.makeboldspark.com Live Demo: https://httpclientdecorator.makeboldspark.com/

About

WebSpark.HttpClientUtility demonstrates enterprise-grade HTTP client patterns for .NET — resilience with Polly, intelligent response caching, structured logging with correlation IDs, and OpenTelemetry tracing, all configured in a single AddHttpClientUtility() call.

This repository produces three primary deliverables:

Built by Mark Hazleton — Mark Hazleton, Solutions Architect WebSpark.HttpClientUtility is part of the Make Bold Spark portfolio of technical demonstrations.

Stop writing 50+ lines of HttpClient setup. Get enterprise-grade resilience (retries, circuit breakers), intelligent caching, structured logging with correlation IDs, and OpenTelemetry tracing in a single AddHttpClientUtility() call. Perfect for microservices, background workers, and web scrapers.

🚀 Why Choose WebSpark.HttpClientUtility?

Your HTTP setup in 1 line vs. 50+

When to use WebSpark:

• ✅ Building microservices with distributed tracing requirements
• ✅ Need resilience patterns without writing Polly boilerplate
• ✅ Want intelligent caching for API rate-limit compliance
• ✅ Building web scrapers or crawlers (with Crawler package)

When NOT to use WebSpark:

• ❌ You need declarative, type-safe API clients (use Refit)
• ❌ You want maximum control and minimal magic (use raw HttpClient)
• ❌ Legacy .NET Framework 4.x projects (WebSpark requires .NET 8+)

🛡️ Production Trust

Battle-Tested & Production-Ready

• ✅ Comprehensive automated test coverage across .NET 8, 9, and 10
• ✅ Source Link enabled - step-through debugging with symbol packages (.snupkg)
• ✅ Trimming & AOT ready - annotated for Native AOT and IL trimming compatibility
• ✅ Package validation - baseline validation ensures no breaking changes
• ✅ Zero-warning builds - strict code quality (TreatWarningsAsErrors=false; warnings must be fixed before commit)
• ✅ Continuous Integration via GitHub Actions - every commit tested
• ✅ Semantic Versioning - predictable, safe upgrades
• ✅ Zero breaking changes within major versions - backward compatibility guaranteed
• ✅ Framework Support: .NET 8 LTS (until Nov 2026), .NET 9, .NET 10 LTS (until May 2028) — library packages multi-target all three; demo app targets .NET 10 only
• ✅ MIT Licensed - free for commercial use

Support & Maintenance

• 🔄 Active development - regular updates and improvements
• 📅 Long-term support - each major version supported for 18+ months
• 💬 Community support - GitHub Discussions for questions and best practices
• 📖 Comprehensive documentation - Full docs site

Breaking Change Commitment

We follow semantic versioning strictly:

• Patch versions (2.0.x): Bug fixes only, zero breaking changes
• Minor versions (2.x.0): New features, backward compatible
• Major versions (x.0.0): Breaking changes with detailed migration guides

📦 v2.5.0 - Two Focused Packages, Modern Feature Set

The library is delivered as two focused packages:

Upgrading from v1.x? Most users need no code changes! See Migration Guide.

📚 Documentation

View Full Documentation →

The complete documentation site includes:

• Getting Started
• Features
• API Reference
• Code Examples
• About

Try the Live Demo →

The demo site lets you interactively explore caching, resilience, crawling, batch execution, and streaming features running on .NET 10.

⚡ 30-Second Quick Start

Install

dotnet add package WebSpark.HttpClientUtility

Minimal Example (Absolute Minimum)

// Program.cs
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClientUtility();
var app = builder.Build();

app.MapGet("/weather", async (IHttpRequestResultService http) =>
{
    var request = new HttpRequestResult<WeatherData>
    {
        RequestPath = "https://api.weather.com/forecast?city=Seattle",
        RequestMethod = HttpMethod.Get
    };
    var result = await http.HttpSendRequestResultAsync(request);
    return result.IsSuccessStatusCode ? Results.Ok(result.ResponseResults) : Results.Problem();
});

app.Run();

record WeatherData(string City, int Temp);

That's it! You now have:

• ✅ Automatic correlation IDs for tracing
• ✅ Structured logging with request/response details
• ✅ Request timing telemetry
• ✅ Proper error handling and exception management
• ✅ Support for .NET 8 LTS, .NET 9, and .NET 10

// Program.cs
builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;      // Cache responses
    options.EnableResilience = true;   // Retry on failure
});

// WeatherService.cs
public class WeatherService
{
    private readonly IHttpRequestResultService _http;
    private readonly ILogger<WeatherService> _logger;

    public WeatherService(
        IHttpRequestResultService http,
        ILogger<WeatherService> logger)
    {
        _http = http;
        _logger = logger;
    }

    public async Task<WeatherData?> GetWeatherAsync(string city)
    {
        var request = new HttpRequestResult<WeatherData>
        {
            RequestPath = $"https://api.weather.com/forecast?city={city}",
            RequestMethod = HttpMethod.Get,
            CacheDurationMinutes = 10  // Cache for 10 minutes
        };

        var result = await _http.HttpSendRequestResultAsync(request);

        if (!result.IsSuccessStatusCode)
        {
            _logger.LogError(
                "Weather API failed: {StatusCode} - {Error}",
                result.StatusCode,
                result.ErrorDetails
            );
            return null;
        }

        return result.ResponseResults;
    }
}

// Program.cs - Advanced configuration
builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;
    options.EnableResilience = true;
    options.ResilienceOptions.MaxRetryAttempts = 3;
    options.ResilienceOptions.RetryDelay = TimeSpan.FromSeconds(2);
    options.DefaultTimeout = TimeSpan.FromSeconds(30);
});

// WeatherService.cs - Advanced usage
public async Task<WeatherData?> GetWeatherWithAuthAsync(string city, string apiKey)
{
    var request = new HttpRequestResult<WeatherData>
    {
        RequestPath = $"https://api.weather.com/forecast?city={city}",
        RequestMethod = HttpMethod.Get,
        CacheDurationMinutes = 10,
        Headers = new Dictionary<string, string>
        {
            ["X-API-Key"] = apiKey,
            ["Accept"] = "application/json"
        }
    };

    var result = await _http.HttpSendRequestResultAsync(request);

    // Correlation ID is automatically logged and propagated
    _logger.LogInformation(
        "Weather request completed in {Duration}ms with correlation {CorrelationId}",
        result.RequestDuration,
        result.CorrelationId
    );

    return result.IsSuccessStatusCode ? result.ResponseResults : null;
}

Web Crawling Features (Crawler Package)

Install Both Packages

dotnet add package WebSpark.HttpClientUtility
dotnet add package WebSpark.HttpClientUtility.Crawler

Register Services

// Program.cs
builder.Services.AddHttpClientUtility();
builder.Services.AddHttpClientCrawler();  // Adds crawler features

Use Crawler

public class SiteAnalyzer
{
    private readonly ISiteCrawler _crawler;
    
    public SiteAnalyzer(ISiteCrawler crawler) => _crawler = crawler;

    public async Task<CrawlResult> AnalyzeSiteAsync(string url)
    {
        var options = new CrawlerOptions
        {
            MaxDepth = 3,
            MaxPages = 100,
            RespectRobotsTxt = true
        };
        
        return await _crawler.CrawlAsync(url, options);
    }
}

🚀 Features

Base Package Features

• Simple API - Intuitive request/response model
• Authentication - Bearer token, Basic auth, API key providers
• Correlation IDs - Automatic tracking across distributed systems
• Structured Logging - Rich context in all log messages
• Telemetry - Request timing and performance metrics
• Error Handling - Standardized exception processing
• Type-Safe - Strongly-typed request and response models
• Caching - In-memory response caching (optional)
• Resilience - Polly retry and circuit breaker policies (optional)
• Concurrent Requests - Parallel request processing
• Fire-and-Forget - Background request execution
• Streaming - Efficient handling of large responses
• OpenTelemetry - Full observability integration (optional)
• Batch Execution - Environment x user x request orchestration with progress and statistics (optional)
• CURL Export - Generate CURL commands for debugging
• Source Link - Step-through debugging with symbol packages
• Trimming/AOT Ready - Compatible with Native AOT and IL trimming
• Package Validation - Baseline validation ensures stability

Crawler Package Features

• Site Crawling - Full website crawling with depth control
• Robots.txt - Automatic compliance with robots.txt rules
• Sitemap Generation - Create XML sitemaps from crawl results
• HTML Parsing - Extract links and metadata with HtmlAgilityPack
• SignalR Progress - Real-time crawl progress updates
• CSV Export - Export crawl results to CSV files
• Performance Tracking - Monitor crawl speed and efficiency

📚 Common Scenarios

Enable Caching

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableCaching = true;
});

// In your service
var request = new HttpRequestResult<Product>
{
    RequestPath = "https://api.example.com/products/123",
    RequestMethod = HttpMethod.Get,
    CacheDurationMinutes = 10  // Cache for 10 minutes
};

Add Resilience (Retry + Circuit Breaker)

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableResilience = true;
  options.ResilienceOptions.MaxRetryAttempts = 3;
    options.ResilienceOptions.RetryDelay = TimeSpan.FromSeconds(2);
});

Configuration Binding (TimeSpan and Seconds Aliases)

Both configuration styles are supported under HttpRequestResultPollyOptions:

{
    "HttpRequestResultPollyOptions": {
        "MaxRetryAttempts": 3,
        "RetryDelay": "00:00:02",
        "CircuitBreakerThreshold": 5,
        "CircuitBreakerDuration": "00:00:30"
    }
}

{
    "HttpRequestResultPollyOptions": {
        "MaxRetryAttempts": 3,
        "RetryDelaySeconds": 2,
        "CircuitBreakerThreshold": 5,
        "CircuitBreakerDurationSeconds": 30
    }
}

Enable Batch Execution Orchestration

builder.Services.AddHttpClientUtility(options =>
{
    options.EnableBatchExecution = true;
    options.EnableResilience = true;
});

var configuration = new BatchExecutionConfiguration
{
    Environments =
    [
        new BatchEnvironment { Name = "Local", BaseUrl = "https://localhost:5001" }
    ],
    Users =
    [
        new BatchUserContext
        {
            UserId = "john.doe",
            Properties = new Dictionary<string, string> { ["userId"] = "42" }
        }
    ],
    Requests =
    [
        new BatchRequestDefinition
        {
            Name = "GetProfile",
            Method = "GET",
            PathTemplate = "/api/users/{userId}"
        }
    ],
    Iterations = 1,
    MaxConcurrency = 4
};

var batchService = serviceProvider.GetRequiredService<IBatchExecutionService>();
var result = await batchService.ExecuteAsync(configuration);

All Features Enabled

builder.Services.AddHttpClientUtilityWithAllFeatures();

🔄 Upgrading from v1.x

If You DON'T Use Web Crawling

No code changes required! Simply upgrade:

dotnet add package WebSpark.HttpClientUtility

Your existing code continues to work exactly as before. All core HTTP features (authentication, caching, resilience, telemetry, etc.) are still in the base package with the same API.

If You DO Use Web Crawling

Three simple steps to migrate:

Step 1: Install the crawler package

dotnet add package WebSpark.HttpClientUtility.Crawler

Step 2: Add using directive

using WebSpark.HttpClientUtility.Crawler;

Step 3: Update service registration

// v1.x (old)
services.AddHttpClientUtility();

// v2.0 (new)
services.AddHttpClientUtility();
services.AddHttpClientCrawler();  // Add this line

That's it! Your crawler code (ISiteCrawler, SiteCrawler, SimpleSiteCrawler, etc.) works identically after these changes.

Need Help? See the detailed migration guide or open an issue.

📖 Documentation

• Getting Started - Complete walkthrough
• Features - All feature documentation
• API Reference - Complete API documentation
• Examples - Code scenarios and patterns
• About - Project background and contributing

🎓 Sample Projects

Explore the live demo at httpclientdecorator.makeboldspark.com — it demonstrates all major features interactively:

• Simple GET/POST requests with correlation ID tracking
• Response caching with configurable duration
• Retry and circuit breaker patterns with Polly
• Concurrent parallel request processing
• Web crawler with robots.txt compliance
• Batch execution with environment/user orchestration
• Real-time SignalR progress updates
• Streaming large responses

🤝 Contributing

Contributions are welcome! See our Contributing Guide for details.

1. Fork the repository
2. Create a feature branch
3. Add tests for your changes
4. Ensure all tests pass
5. Submit a pull request

📊 Project Stats

• Comprehensive automated test coverage across .NET 8, 9, and 10
• Supports .NET 8 LTS, .NET 9, & .NET 10
• MIT Licensed - Free for commercial use
• Active Maintenance - Regular updates

📦 Related Packages

Testing Package Features:

• FakeHttpResponseHandler - Mock HTTP responses without network calls
• Fluent API - Easy test setup with ForRequest().RespondWith()
• Sequential Responses - Test retry behavior with multiple responses
• Request Verification - Assert requests were made correctly
• Latency Simulation - Test timeout scenarios

dotnet add package WebSpark.HttpClientUtility.Testing

See the Testing documentation for examples.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• GitHub Repository
• NuGet Package
• Docs Site
• Live Demo
• Changelog
• Issue Tracker
• Discussions

Questions or Issues? Open an issue or start a discussion!