When a Kafka broker restarts unexpectedly, Storm supervisors can throw an offset out of range error that interrupts data pipelines. This error typically surfaces as an unrecoverable fetch offset failure and requires coordinated troubleshooting across Kafka and Storm layers.
The table below summarizes the core dimensions of this issue, from root causes to remediation steps, helping teams align on impact and action quickly.
| Phase | Key Indicator | Common Root Cause | Typical Remedy |
|---|---|---|---|
| Broker Restart | Controller and broker logs show clean restart sequence | Log retention purge removes segment files before consumers catch up | Increase retention.ms and segment sizes, monitor underReplicatedPartitions |
| Storm Spout Task | Spout emits WARN or ERROR about offset out of range | Consumer group offset stored in Kafka is larger than available log | Adjust auto.offset.reset, reset consumer group, or reprocess data |
| Consumer Group State | {"data": {"content": ["Current committed offset metadata incorrect after restart"]}}|||
| Operational Response | Alerts spike and Storm worker containers report repeated retries | Small replica fetch backoff and log segment deletion race condition | Coordinate controlled restart, validate offsets before resuming topology |
Impact on Storm Supervisor Task Initialization
During a Kafka broker restart, Storm supervisor tasks that rely on KafkaSpout may fail initialization when the spout tries to fetch messages from an offset that no longer exists. The broker responds with OFFSET_OUT_OF_RANGE, and the supervisor logs show repeated fetch retries. If the retention policy has already purged the required log segments, the consumer group cannot catch up, and topology processing stalls.
Kafka Log Retention and Segment Deletion Mechanics
Kafka log retention settings such as retention.ms and retention.bytes determine how long segments are kept on disk. During a forced broker restart, some segments may be deleted if the broker was offline longer than retention.ms. When the Storm Kafka spout reconnects after the restart, it reads the last committed offset from ZooKeeper or the Kafka group coordinator and attempts to fetch from that position. If the segment is gone, the broker returns an offset out of range error and the supervisor task enters a failing state.
Consumer Group Offset Management Across Restarts
Consumer offsets are stored in the internal Kafka group metadata topic. A normal shutdown updates this topic before brokers stop, but a restart can leave offsets stale if retention or compaction policies change the underlying log. Storm supervisors using Kafka clients must handle these offsets carefully, especially when new broker generations enforce stricter bounds checks. Misaligned offsets are a primary trigger for the offset out of range error observed by supervisors.
Topology Resilience and Configuration Best Practices
To reduce supervisor failures after broker restart, tune Kafka client configs and Storm topology settings together. Align auto.offset.reset, session.timeout.ms, and max.poll.records with your retention window and processing latency goals. Enable appropriate metrics in Storm to detect repeated offset errors early, and design recovery playbooks that validate committed offsets before reactivating topologies.
Operational Recovery and Prevention Checklist
- Review Kafka log.retention.hours and topic-level retention settings before any broker maintenance.
- Drain consumer lag and verify committed offsets align with available log segments.
- Coordinate a controlled restart sequence to allow consumers to commit offsets gracefully.
- Enable Storm metrics for Kafka spout offset errors and set alerts for repeated OFFSET_OUT_OF_RANGE events.
- Document offset reset and topology restart procedures for operations teams to follow during incidents.
FAQ
Reader questions
Why does Storm report offset out of range immediately after a Kafka broker restart?
The Storm Kafka spout resumes from the last committed offset, which may point to log segments that were deleted during the broker downtime due to retention policies. When the broker returns with no matching segments, it replies with OFFSET_OUT_OF_RANGE, causing the supervisor to throw an error.
How can I preserve Kafka offsets during a planned broker restart to protect Storm supervisor tasks?
Increase retention.ms and retention.bytes, delay segment deletion with clean shutdown scripts, or take a consumer group offset backup before restart so you can restore offsets after the broker comes back online and avoid gaps in available data.
What should I do when a Storm worker repeatedly logs offset out of range errors during restart recovery?
First, inspect the Kafka topic’s current log end offset and compare it to the stored consumer offset. If the consumer offset is larger, reset the group using kafka-consumer-groups.sh with --execute --reset-offsets to a valid position, then restart the Storm topology manually.
Can adjusting auto.offset.reset solve offset out of range issues in Storm supervisors after broker restart?
Setting auto.offset.reset to earliest can allow the spout to start from the earliest available message when no valid committed offset exists. This prevents the supervisor from failing, but you should verify that reprocessing historical data is acceptable for your use case and downstream sinks.