The Memory Wall: Why Traditional Slices Fail in Production
Production crashes are often loud, expensive, and preventable. Imagine you are building a service to process 10 million log entries from a legacy database. On your local machine, using a few hundred rows of test data, returning a slice of structs feels natural and performs perfectly.
func GetLogs() []LogEntry {
var logs []LogEntry
// Fetching millions of rows...
return logs
}The reality changes at scale. If each LogEntry struct is 128 bytes, a slice of 10 million entries consumes roughly 1.2GB of RAM before your processing logic even touches the first row. In a containerized environment with a 1GB limit, your service will hit an Out of Memory (OOM) error and restart instantly. This “all-or-nothing” approach to data handling has been a significant bottleneck for Go developers for over a decade.
The Trade-offs of the Past
Before Go 1.23, the for...range loop was restricted to built-in types like slices, maps, and channels. Handling a custom data structure—like a B-Tree or a paginated API response—required picking between two imperfect solutions.
Channels were the primary way to stream data. You could spin up a goroutine to push items into a channel while the main loop consumed them. This solved the memory spike but introduced a massive performance tax. Because channels involve internal locking and context switching, they can be 10x to 20x slower than a standard function call for simple iteration.
Callback Functions were the faster alternative. You would pass a function into an iterator, which would execute for every item. It looked like this:
func (s *Scanner) Each(fn func(item string) bool) {
for _, item := range s.data {
if !fn(item) {
break
}
}
}While efficient, the syntax felt “inverted.” You couldn’t use break, continue, or return inside the callback as naturally as you could in a standard loop. This made the code harder to read and significantly more difficult to debug.
The New Standard: range-over-func
Go 1.23 bridges the gap between performance and readability with range-over-func. This feature allows you to plug custom functions directly into the for...range loop. You get the memory efficiency of a stream combined with the clean, familiar syntax of a slice. In high-throughput telemetry systems, this shift allows services to maintain a flat memory profile even as data volume spikes by 500%.
Implementing iter.Seq for Efficient Streaming
The iter package provides the blueprints for this new behavior. Specifically, iter.Seq (for single values) and iter.Seq2 (for key-value pairs) are the types you will use most often.
1. The Yield Function
An iterator is essentially a function that accepts another function called yield. When you call yield(value), Go pauses your iterator and executes the body of the for loop. If yield returns false, it indicates the loop has terminated—perhaps due to a break statement—and your iterator should clean up and stop.
import (
"fmt"
"iter"
)
func Count(limit int) iter.Seq[int] {
return func(yield func(int) bool) {
for i := 0; i < limit; i++ {
if !yield(i) {
return
}
}
}
}2. Using the Iterator
Once defined, the iterator integrates seamlessly. The compiler recognizes the signature and handles the heavy lifting. Loop control is now intuitive again.
func main() {
for n := range Count(1000000) {
fmt.Println(n)
if n == 2 {
break // yield returns false, and the function exits
}
}
}3. Handling Pairs with iter.Seq2
If you need to return both an index and a value, or perhaps a key and an error, iter.Seq2 is the correct tool. This is ideal for streaming database rows where you want to provide both the record ID and the data object.
func StreamLogs(lines []string) iter.Seq2[int, string] {
return func(yield func(int, string) bool) {
for i, line := range lines {
if !yield(i, line) {
return
}
}
}
}
// Usage is identical to ranging over a map or slice
for index, msg := range StreamLogs(logData) {
fmt.Printf("Line %d: %s\n", index, msg)
}Engineering Impact: Why This Matters
Adopting iterators isn’t just about using the latest syntax; it’s about building more resilient systems. When you use iter.Seq, your code gains three immediate advantages:
- Constant Memory Usage: Your RAM usage stays flat whether you are processing 10 items or 10 billion.
- Lazy Evaluation: You only perform the work (like parsing JSON or reading from a socket) when the loop actually asks for the next item.
- Clean Composition: You can wrap iterators inside each other to create powerful processing pipelines without intermediate allocations.
Practical Example: Functional Filtering
One powerful pattern is creating a filter. Since iterators are just functions, you can chain them. This allows you to transform data on the fly without ever creating a temporary slice.
func Filter[V any](seq iter.Seq[V], predicate func(V) bool) iter.Seq[V] {
return func(yield func(V) bool) {
for v := range seq {
if predicate(v) {
if !yield(v) {
return
}
}
}
}
}
// Usage: Process only even numbers from a stream of millions
numbers := Count(1000000)
evenNumbers := Filter(numbers, func(n int) bool { return n % 2 == 0 })
for n := range evenNumbers {
fmt.Println(n)
}Summary
Go 1.23 iterators finally resolve the tension between clean code and high performance. You no longer need to choose between the simplicity of slices and the efficiency of streams. By moving away from large slice returns and adopting iter.Seq, you ensure your applications are ready for the unpredictable loads of production environments. If your code interacts with databases, large files, or paginated APIs, start refactoring toward range-over-func today.
