Problem Statement: The 2:14 AM Out-of-Memory Incident
My phone buzzed on the nightstand at 2:14 AM. PagerDuty showed our primary payment ingestion microservice had crashed with an unhandled OutOfMemoryException. When I pulled up Grafana, the resident memory graph went vertical—climbing from 320 MB to 6.2 GB in minutes. CPU usage hovered near 100%, and thread pool starvation was dragging down every dependent HTTP endpoint.
The root cause was simple: a midnight marketing flash sale blasted our API with 25,000 transaction requests per second. Our ASP.NET Core controllers accepted every incoming payload and immediately pushed it into an unbounded ConcurrentQueue<PaymentEvent>, counting on a background task to flush records to PostgreSQL.
But PostgreSQL capped out at 4,200 writes per second. Within eight minutes, over 9.9 million unconsumed objects flooded the heap. That triggered continuous Gen-2 Garbage Collection pauses before Kubernetes finally killed the pod.
Engineers often reach for RabbitMQ, AWS SQS, or Apache Kafka the moment an incident like this happens. But adding a distributed broker introduces real costs: operational overhead, network round-trips, payload serialization, and monthly infrastructure bills.
When your workflow stays inside a single process—like buffering log events, aggregating audit trails, or batching database writes—you do not need a message broker. You need in-process backpressure. That is where System.Threading.Channels fits in.
Core Concepts: Why Channel<T> Outperforms Classic Collections
Before .NET introduced System.Threading.Channels, developers usually chose between ConcurrentQueue<T> and BlockingCollection<T>. Under production pressure, both hit major limits:
- ConcurrentQueue<T> has no built-in bounds and lacks an asynchronous wait primitive. Consumers must spin or sleep in a loop, burning CPU cycles while waiting for fresh work.
- BlockingCollection<T> supports fixed bounds, but its API blocks threads synchronously via
Take(). Holding worker threads idle during traffic surges quickly starves the .NET ThreadPool and inflates HTTP request latency.
Channel<T> fixes both problems. It offers thread-safe, fully asynchronous communication split cleanly into two contracts:
ChannelWriter<T>: Producers push items usingWriteAsync()or non-blockingTryWrite().ChannelReader<T>: Consumers pull items viaWaitToReadAsync()andReadAllAsync().
The real superpower here is bounded flow control. When a bounded channel fills up, WriteAsync() does not block the active OS thread. Instead, it yields execution via a standard task continuation until consumers clear room.
Once you have watched an out-of-control queue take down a node, you realize that backpressure is not optional. It is the boundary that keeps high-throughput services alive.
Hands-on Practice: Building a Resilient In-Memory Pipeline
Let us redesign the payment ingestion engine. We will configure a bounded channel, write an async producer controller, and consume the queue using a BackgroundService.
Step 1: Configure Channel Options
First, instantiate the bounded channel and specify how it reacts when full.
using System.Threading.Channels;
public static class ChannelFactory
{
public static Channel<PaymentEvent> CreatePaymentChannel(int capacity = 10_000)
{
var options = new BoundedChannelOptions(capacity)
{
// Apply backpressure by asynchronously pausing writers
FullMode = BoundedChannelFullMode.Wait,
// Multiple web request threads push data concurrently
SingleWriter = false,
// A single background worker consumes the queue
SingleReader = true,
// Prevent synchronous continuation deadlocks on the writing thread
AllowSynchronousContinuations = false
};
return Channel.CreateBounded<PaymentEvent>(options);
}
}
public record PaymentEvent(Guid TransactionId, decimal Amount, DateTime CreatedAt);Marking SingleReader = true tells the channel to bypass multi-consumer synchronization locks. That small switch removes atomic contention on the consumer loop and reduces CPU cache-line bouncing.
Step 2: Implement the Producer (API Controller)
Inject ChannelWriter<PaymentEvent> into your API controller. When a payment arrives, push it to the buffer asynchronously.
using System.Threading.Channels;
using Microsoft.AspNetCore.Mvc;
[ApiController]
[Route("api/[controller]")]
public class PaymentsController : ControllerBase
{
private readonly ChannelWriter<PaymentEvent> _writer;
private readonly ILogger<PaymentsController> _logger;
public PaymentsController(ChannelWriter<PaymentEvent> writer, ILogger<PaymentsController> logger)
{
_writer = writer;
_logger = logger;
}
[HttpPost]
public async Task<IActionResult> ProcessPayment([FromBody] PaymentRequest request, CancellationToken ct)
{
var paymentEvent = new PaymentEvent(Guid.NewGuid(), request.Amount, DateTime.UtcNow);
// Asynchronously awaits if the 10,000-item buffer is full
await _writer.WriteAsync(paymentEvent, ct);
return Accepted(new { paymentEvent.TransactionId, Status = "Queued" });
}
}
public record PaymentRequest(decimal Amount);Watch what happens when pending transactions reach 10,000. WriteAsync releases the request thread right back to Kestrel instead of tying it up. Inbound HTTP requests simply wait their turn without bloating memory or starving worker threads.
Step 3: Implement the Consumer Background Worker
Now build a BackgroundService that drains items from the reader and writes them downstream.
using System.Threading.Channels;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
public class PaymentProcessorService : BackgroundService
{
private readonly ChannelReader<PaymentEvent> _reader;
private readonly ILogger<PaymentProcessorService> _logger;
public PaymentProcessorService(ChannelReader<PaymentEvent> reader, ILogger<PaymentProcessorService> logger)
{
_reader = reader;
_logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_logger.LogInformation("Payment processor worker started.");
try
{
// ReadAllAsync yields items as they arrive and completes on channel closure
await foreach (PaymentEvent payment in _reader.ReadAllAsync(stoppingToken))
{
await ProcessPaymentAsync(payment, stoppingToken);
}
}
catch (OperationCanceledException)
{
_logger.LogInformation("Payment processor shutting down cleanly.");
}
}
private async Task ProcessPaymentAsync(PaymentEvent payment, CancellationToken ct)
{
// Simulate database persistence latency
await Task.Delay(2, ct);
_logger.LogDebug("Transaction {Id} processed.", payment.TransactionId);
}
}Step 4: Register Dependencies with Dependency Injection
Register the channel as a singleton in Program.cs, exposing the reader and writer separately to keep boundaries clean.
var builder = WebApplication.CreateBuilder(args);
var paymentChannel = ChannelFactory.CreatePaymentChannel(capacity: 10_000);
builder.Services.AddSingleton(paymentChannel.Reader);
builder.Services.AddSingleton(paymentChannel.Writer);
builder.Services.AddHostedService<PaymentProcessorService>();
builder.Services.AddControllers();
var app = builder.Build();
app.MapControllers();
app.Run();Production Checklist and Key Takeaways
Moving from a loose in-memory queue to Channel<T> stopped our late-night alerts for good. Under our next peak sale, memory flatlined at 185 MB, even when incoming traffic spiked past 28,000 requests per second.
Review these four rules before shipping channels to production:
- Always set a bounded capacity: Never use
Channel.CreateUnbounded<T>behind public endpoints. Unbounded buffers trade short-term throughput for eventual process crashes. - Pick the appropriate FullMode: Use
BoundedChannelFullMode.Waitwhen dropping data is unacceptable. For non-critical telemetry, switch toDropOldestorDropNewestto prioritize fresh data over old samples. - Drain during graceful shutdown: Call
writer.Complete()inside your application lifetime stopping hooks. This letsReadAllAsync()finish pending buffer items before the runtime terminates. - Leverage single-actor flags: Enable
SingleReaderorSingleWriterwhenever only one worker handles that side of the channel. The runtime cuts out internal locking, yielding measurably faster throughput.
