kolayxlsxstream

KolayXlsxStream - High-Performance XLSX Streaming for Go

A high-performance Go library for streaming XLSX files with constant memory usage and direct S3 support. Export millions of rows without worrying about memory constraints.

🚀 Features

• Constant Memory Usage (O(1)): Memory usage remains constant regardless of file size (<1MB for local, ~33MB for S3)
• Direct S3 Streaming: Stream directly to AWS S3 using multipart uploads (no temporary files)
• Zero Disk I/O: No temporary files created during export
• Blazing Fast: 600,000+ rows/second (local file), 50,000-110,000 rows/second (S3)
• Automatic Multi-Sheet: Automatically creates new sheets when Excel's 1,048,576 row limit is reached
• Configurable Compression: Adjust ZIP compression level (0-9) for speed vs. size tradeoffs
• Type Safety: Native Go types (string, int, float64, bool, etc.)
• Production Tested: Successfully exported 2 million rows (60MB files) with AWS S3
• Clean API: Simple, intuitive interface inspired by the PHP version

📦 Installation

go get github.com/turgutahmet/kolayxlsxstream

🎯 Quick Start

Basic Example - Local File

package main

import (
    "log"
    "github.com/turgutahmet/kolayxlsxstream"
)

func main() {
    // Create a file sink
    sink, err := kolayxlsxstream.NewFileSink("output.xlsx")
    if err != nil {
        log.Fatal(err)
    }

    // Create a writer
    writer := kolayxlsxstream.NewWriter(sink)

    // Start file with headers
    headers := []interface{}{"Name", "Email", "Age"}
    writer.StartFile(headers)

    // Write rows
    writer.WriteRow([]interface{}{"John Doe", "[email protected]", 30})
    writer.WriteRow([]interface{}{"Jane Smith", "[email protected]", 25})

    // Finish and get statistics
    stats, err := writer.FinishFile()
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Wrote %d rows in %.2f seconds", stats.TotalRows, stats.Duration)
}

S3 Streaming Example

package main

import (
    "context"
    "log"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/turgutahmet/kolayxlsxstream"
)

func main() {
    // Load AWS config
    cfg, _ := config.LoadDefaultConfig(context.TODO())
    client := s3.NewFromConfig(cfg)

    // Create S3 sink
    ctx := context.Background()
    sink, err := kolayxlsxstream.NewS3Sink(ctx, client, "my-bucket", "exports/report.xlsx")
    if err != nil {
        log.Fatal(err)
    }

    // Create writer and export
    writer := kolayxlsxstream.NewWriter(sink)
    writer.StartFile([]interface{}{"ID", "Product", "Quantity", "Price"})

    // Stream 1 million rows to S3
    for i := 1; i <= 1000000; i++ {
        writer.WriteRow([]interface{}{i, "Product", 10, 99.99})
    }

    stats, _ := writer.FinishFile()
    log.Printf("Uploaded to S3: %d rows, %.0f rows/sec", stats.TotalRows, stats.RowsPerSecond)
}

📖 Documentation

Core Components

Writer

The main XLSX writer that handles streaming data to a sink.

// Create writer with default config
writer := kolayxlsxstream.NewWriter(sink)

// Create writer with custom config
config := kolayxlsxstream.DefaultConfig()
config.CompressionLevel = 9  // Maximum compression
config.MaxRowsPerSheet = 500000  // Custom row limit
writer := kolayxlsxstream.NewWriter(sink, config)

Methods

• StartFile(headers ...[]interface{}) error: Initialize the file, optionally with headers
• WriteRow(values []interface{}) error: Write a single row
• WriteRows(rows [][]interface{}) error: Write multiple rows
• FinishFile() (*Stats, error): Finalize the file and return statistics
• SetCompressionLevel(level int) error: Set compression level (0-9)
• SetBufferSize(size int) error: Set buffer size
• SetMaxRowsPerSheet(rows int) error: Set maximum rows per sheet

Sinks

FileSink

Writes to a local file.

sink, err := kolayxlsxstream.NewFileSink("/path/to/output.xlsx")

S3Sink

Streams directly to AWS S3 using multipart uploads.

ctx := context.Background()
options := kolayxlsxstream.DefaultS3Options()
options.PartSize = 10 * 1024 * 1024  // 10MB parts
options.ACL = types.ObjectCannedACLPrivate
options.StorageClass = types.StorageClassIntelligentTiering

sink, err := kolayxlsxstream.NewS3Sink(ctx, s3Client, "bucket", "key", options)

Configuration

type Config struct {
    CompressionLevel int    // ZIP compression (0-9, default: 6)
    BufferSize       int    // Buffer size in bytes (default: 64KB)
    MaxRowsPerSheet  int    // Max rows per sheet (default: 1,048,576)
    SheetNamePrefix  string // Sheet name prefix (default: "Sheet")
}

Statistics

type Stats struct {
    TotalRows      int64   // Total data rows written
    TotalSheets    int     // Total sheets created
    FileSize       int64   // Total file size (if available)
    Duration       float64 // Duration in seconds
    RowsPerSecond  float64 // Average rows/second
    BytesPerSecond float64 // Average bytes/second
}

🎨 Complete Examples

The examples/ directory contains real-world usage scenarios:

Basic Usage

cd examples/basic && go run main.go

Large Files (250k rows, multi-sheet)

cd examples/large && go run main.go

S3 Streaming (with dry-run mode)

cd examples/s3 && go run main.go -dry-run -rows 50000
# Or with actual S3:
go run main.go -bucket my-bucket -rows 100000

CSV to XLSX Conversion

cd examples/csv-to-xlsx && go run main.go -input sample.csv -output output.xlsx

Database Export

cd examples/database && go run main.go -sample 100000 -output export.xlsx

Memory Profiling

cd examples/memory && go run main.go

🎨 Advanced Code Examples

Large Dataset with Progress Tracking

sink, _ := kolayxlsxstream.NewFileSink("large.xlsx")
writer := kolayxlsxstream.NewWriter(sink)

headers := []interface{}{"ID", "Name", "Email", "Score"}
writer.StartFile(headers)

totalRows := 5000000
batchSize := 1000

for i := 0; i < totalRows; i += batchSize {
    batch := make([][]interface{}, batchSize)
    for j := 0; j < batchSize; j++ {
        rowNum := i + j + 1
        batch[j] = []interface{}{
            rowNum,
            fmt.Sprintf("User %d", rowNum),
            fmt.Sprintf("user%[email protected]", rowNum),
            float64(rowNum % 100),
        }
    }
    writer.WriteRows(batch)

    if (i+batchSize)%100000 == 0 {
        fmt.Printf("Progress: %d rows\n", i+batchSize)
    }
}

stats, _ := writer.FinishFile()
fmt.Printf("Done! %d rows in %.2f seconds\n", stats.TotalRows, stats.Duration)

Custom Compression for Speed

// Ultra-fast mode (minimal compression)
config := kolayxlsxstream.DefaultConfig()
config.CompressionLevel = 1  // Fastest compression
config.BufferSize = 128 * 1024  // 128KB buffer

sink, _ := kolayxlsxstream.NewFileSink("fast.xlsx")
writer := kolayxlsxstream.NewWriter(sink, config)

Multi-Sheet Export

// Automatically creates new sheets every 100k rows
config := kolayxlsxstream.DefaultConfig()
config.MaxRowsPerSheet = 100000
config.SheetNamePrefix = "Data"

writer := kolayxlsxstream.NewWriter(sink, config)
writer.StartFile([]interface{}{"Column1", "Column2"})

// Write 250k rows -> creates 3 sheets: Data1, Data2, Data3
for i := 0; i < 250000; i++ {
    writer.WriteRow([]interface{}{i, fmt.Sprintf("Value %d", i)})
}

stats, _ := writer.FinishFile()
fmt.Printf("Created %d sheets\n", stats.TotalSheets)  // Output: 3

🔧 Performance Tips

1. Batch Writes: Use WriteRows() instead of WriteRow() when possible
2. Compression: Lower compression (1-3) for speed, higher (6-9) for file size
3. Buffer Size: Increase buffer size for better throughput (64KB-256KB)
4. S3 Part Size: Use larger parts (32MB-100MB) for better S3 performance

📊 Performance Benchmarks

Comprehensive Benchmark Results

Tested on Apple M4, Go 1.23, Compression Level 1 (fastest)

Note: Tests with 1M+ rows automatically create multiple sheets (Excel limit: 1,048,576 rows per sheet)

Note: ± values in S3 Memory column indicate memory fluctuation during streaming due to periodic part uploads (32MB buffer)

Understanding Memory Behavior

Local File System: True O(1) Memory

• Constant Memory: 0-0.5MB regardless of file size
• No Growth: Memory stays flat even for millions of rows
• Speed: 580,000-600,000 rows/second consistently
• Scalable: Successfully tested up to 2 million rows

S3 Streaming: Controlled Memory Growth

The ± values in S3 memory represent normal memory fluctuation during streaming:

Buffer Accumulation Phase (↑ Memory Growth)

• Data is compressed and buffered until reaching 32MB part size
• Memory grows gradually as buffer fills

Part Upload Phase (↓ Memory Drop)

• When buffer reaches 32MB, it's uploaded to S3
• After upload, memory drops back to baseline
• This creates the characteristic sawtooth pattern

Memory
  ▲
78MB │     ╱╲      ╱╲      ╱╲
     │    ╱  ╲    ╱  ╲    ╱  ╲
33MB │   ╱    ╲  ╱    ╲  ╱    ╲
     │  ╱      ╲╱      ╲╱      ╲
2MB  │─╯
     └────────────────────────────▶ Time
       ↑Upload  ↑Upload  ↑Upload

Example: 1M Rows Test

• Average memory: 33MB
• Fluctuation: ±96MB
• Pattern: Memory oscillates between ~2MB (after upload) and ~129MB (before upload)
• This is completely normal and expected behavior for streaming

Performance Highlights

• ✅ Local File System: ~600,000 rows/second with true O(1) memory
• ✅ S3 Streaming: 50,000-110,000 rows/second with controlled memory
• ✅ Memory Efficiency: Local uses <1MB, S3 averages 33MB per million rows
• ✅ Multi-sheet Support: Automatic sheet creation at Excel's 1,048,576 row limit
• ✅ Production Ready: Successfully tested with 2 million rows (60MB files)

Comparison with Other Go Libraries

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

MIT License - see LICENSE file for details.

🙏 Credits

Inspired by the PHP version: kolay-xlsx-stream

📞 Support

• GitHub Issues: Report a bug
• Documentation: pkg.go.dev

Made with ❤️ for the Go community