The Crash Context

It was early March 2022, and we were in the final sprint of launching a new feature in AdSpy Pro, our flagship ad tracking software. The deadline loomed like a storm cloud, and my team was working around the clock to deliver a new data collector that would streamline ad performance metrics. As we approached the finish line, we started noticing some significant slowdowns in the application's performance, especially when querying large datasets.

Initially, it seemed like a hiccup, a minor speed bump in the road. We'd been using the HttpClient class extensively, and it made sense that the high load was causing our service to lag. However, as we dove deeper into the issue, we began to notice that memory consumption was rising sharply over time with each request. What at first seemed like a simple performance degradation turned into a potential disaster.

With clients eagerly awaiting the new feature, I could feel the pressure mounting. My heart raced as we tried to reproduce the issue on our development environment, but, like a ghost, the memory leak remained elusive. We expanded our logging and monitoring, but nothing seemed to point us towards an immediate solution. Every second counted, and we were staring down the barrel of a significant release.

As I paced the office, I couldn’t shake the feeling that we were overlooking something crucial. The tension was palpable; we were on the verge of a breakthrough, yet we were still in the dark about the cause. What if this memory leak continued to grow with our user base? My mind raced as I knew we had to find the root cause before it was too late.

Raw Stack Trace

Our logging captured a surge in memory with this warning message:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Root Cause and Engine Mechanics

The Breakthrough

After hours of investigation, I took a moment to step back and rethink our implementation of HttpClient. Our usage pattern was curious; we were creating a new HttpClient instance for each request rather than reusing a single instance. This ultimately led to a memory leak as these instances were not being disposed of properly. The HttpClient is intended to be instantiated once and reused throughout the application’s lifetime to avoid socket exhaustion and leaking memory.

When I went through the code, I discovered that every invocation of the data collector would generate a new HttpClient, while we had neglected to dispose of the previous ones. Thus, with each call, memory consumption grew as these instances piled up, consuming both memory and network resources. The fault was like a time bomb ticking away, and it was only a matter of time before it exploded.

The mechanics in C# are quite clear; the garbage collector does not immediately reclaim the memory of objects that are no longer in use and are referenced, creating a buildup of memory that would eventually lead to our application crashing or, at the very least, severely degrading performance. The moment I realized this, it felt like a weight lifted off my shoulders. I immediately started refactoring the code to implement a singleton pattern for the HttpClient instance.

We had several discussions on the importance of proper resource management and the impact that improper instantiation can have on the application's health. With time running out, I quickly whipped up the refactor, eager to see if this would mitigate our memory issues.

Broken Code vs Verified Solution

It was shocking to see how easily a pattern could lead to significant performance issues if not handled correctly.

Old: Broken Code Block (Anti-pattern)

In our original implementation, each data collection request created a new instance of HttpClient:

public async Task<List> CollectAdDataAsync(string requestUri) { 
    using (var client = new HttpClient()) { 
        var response = await client.GetAsync(requestUri); 
        var content = await response.Content.ReadAsStringAsync(); 
        return JsonConvert.DeserializeObject<List>(content);
    }
}

Verified Solution Code Block (Commented)

We refactored to use a singleton instance for HttpClient, significantly improving memory management:

private static readonly HttpClient client = new HttpClient(); // Singleton instance

public async Task<List> CollectAdDataAsync(string requestUri) { 
    var response = await client.GetAsync(requestUri); // Reuses HttpClient
    var content = await response.Content.ReadAsStringAsync(); 
    return JsonConvert.DeserializeObject<List>(content);
}

Performance Results and CTA

After implementing the fix, we observed a dramatic improvement in performance metrics across the board.

The improved stability and performance not only saved our launch but also improved the overall user experience. This incident taught me the immense importance of resource management in C# and reinforced the principle that some patterns, while seemingly innocuous, can lead to catastrophic failures if not recognized and handled properly. My friends, always think twice about how you instantiate your resources. Until next time, happy coding!