Calculate data pipeline throughput from records per second and record size. Estimate daily volume and bandwidth requirements.
Data pipelines move records from sources to destinations at rates ranging from hundreds to millions of records per second. Understanding the throughput—both in records/sec and bytes/sec—is critical for sizing infrastructure, provisioning network bandwidth, and planning storage capacity. A pipeline processing 10,000 records/sec at 1 KB each generates 10 MB/sec, which is 864 GB/day.
This calculator converts records-per-second throughput into meaningful capacity metrics: MB/sec, GB/hour, GB/day, and TB/month. It helps you size Kafka clusters, plan network bandwidth, estimate storage requirements, and set realistic SLAs for data freshness.
Whether you're designing a new streaming pipeline on Kafka or Kinesis, or evaluating whether your existing pipeline can handle traffic growth, this tool gives you the numbers you need for infrastructure planning.
Understanding this metric in precise terms allows technology leaders to make evidence-based decisions about scaling, architecture, and infrastructure investment priorities for their organizations. Tracking this metric consistently enables technology teams to identify system performance trends and address potential issues before they impact end users or business operations.
Pipeline capacity mismatches cause data loss, backpressure, and stale analytics. This calculator translates records/sec into storage and bandwidth requirements so you can provision infrastructure correctly before traffic peaks. Regular monitoring of this value helps DevOps teams detect anomalies early and maintain the system reliability and performance that users and business stakeholders expect.
throughput_bytes_sec = records_per_sec × avg_record_bytes; daily_GB = throughput_bytes_sec × 86400 / (1024³); monthly_TB = daily_GB × 30 / 1024
Result: 10 MB/sec avg; 864 GB/day
10,000 records/sec × 1,024 bytes = 10,240,000 bytes/sec (10 MB/sec). Daily: 10 × 86,400 = 864,000 MB = 844 GB. Monthly: ~25.3 TB. With 2× peak multiplier, provision for 20 MB/sec and 1.7 TB/day peak throughput.
Each Kafka partition handles roughly 10–50 MB/sec. Divide your total throughput by the per-partition throughput to get minimum partition count. Multiply by replication factor for total broker disk throughput. Add 30% headroom for traffic spikes.
Pipeline throughput directly consumes network bandwidth. A 100 MB/sec pipeline requires at least 1 Gbps network capacity (800 Mbps data + overhead). Cross-region replication doubles bandwidth requirements. Use compression to reduce wire size.
Daily volume = throughput × 86,400 seconds. Multiply by retention period for total storage. A 50 MB/sec pipeline with 7-day retention needs: 50 × 86,400 × 7 = 30.2 TB of raw storage, or ~10 TB compressed.
Web analytics: 1,000–50,000/sec. IoT telemetry: 10,000–500,000/sec. Financial markets: 100,000–1,000,000/sec. Log shipping: 5,000–100,000/sec. Rates vary enormously by use case and traffic volume.
Check your message broker metrics: Kafka's bytes-in-per-sec and records-in-per-sec, Kinesis's IncomingBytes and IncomingRecords. For batch pipelines, divide total bytes processed by wall-clock runtime.
Backpressure builds up—producers slow down or data is dropped. In Kafka, consumer lag increases and data freshness degrades. In Kinesis, write throttling occurs. Scale up consumers, add partitions/shards, or reduce record size.
JSON is 2–4× larger than binary formats. Protobuf and Avro are compact and schema-enforced. MessagePack sits in between. Choosing a compact format directly reduces bandwidth and storage requirements.
Yes. Kafka replication factor 3 means each record is stored 3×. A 10 MB/sec ingestion rate requires 30 MB/sec of disk throughput across brokers. Include replication in storage and network capacity planning.
Use a weighted average record size based on your message type distribution. If 80% of records are 500 bytes and 20% are 5 KB, the weighted average is 1,400 bytes. Sample real traffic to get accurate averages.