10M+
Connected IoT devices
+400
Events per second
$700M
Annual revenue
A global networking hardware company was ingesting telemetry from 10 million Wi-Fi routers and connected devices, but a fragile, multi-rule pipeline was costing between $20,000 and $25,000 per month, producing outages, and silently discarding data that product engineers needed to improve device quality. Inferenz re-architected the pipeline end to end: 20 billable IoT rules became one, Apache NiFi was replaced with serverless AWS Lambda, Kinesis Firehose compressed files to S3, and single-line JSON made every bad record diagnosable rather than lost. Monthly ingestion cost fell to $5,000. Error rates dropped by 97.6%. The cutover achieved zero downtime in one month.
More than 10 million devices emitted 20+ event types at 400+ events per second, including 300-field daily configuration payloads, variable-length thermal arrays, and kernel reboot logs, arriving on fixed and random schedules from devices across time zones with unsynchronized clocks. Each device also carried router-status events feeding a rule engine that identified the next best action per device.
The legacy pipeline could not absorb the volume, causing outages and dropped events. A device incorrectly showing offline had direct consequences for customer support and product engineering — the analytics used to diagnose quality issues were only as reliable as the data feeding them.
Multi-line JSON meant one corrupted record from a power interrupt, non-UTF character, or missing bracket could invalidate an entire block. Failed records were silently discarded with no way to identify which device, firmware, or release was responsible.
Each incoming message was charged against all 20 IoT rules whether it matched or not. A redundant uncompressed S3 bucket and an unreliable NiFi cluster compounded the expense, pushing monthly data acquisition and ingestion costs to between $20,000 and $25,000.
Inferenz re-architected the telemetry pipeline from device to data cloud while keeping the live fleet streaming throughout. Every stage — ingestion, preprocessing, compression, storage, and error handling — was redesigned to be cheaper, more resilient, and fully observable.
Because every message was billed against all 20 rules regardless of whether it matched or failed, consolidating them into a single rule eliminated the per-rule billing overhead across billions of daily messages — the single largest cost lever in the redesign.
A serverless Lambda function replaced the unreliable, costly NiFi cluster. Billed by the millisecond, Lambda converts multi-line JSON records into single-line format before passing them downstream — removing the fixed infrastructure cost entirely.
Kinesis Data Firehose buffers on whichever threshold is hit first — file size or time — writing roughly 10MB GZIP files to S3. The redundant uncompressed S3 bucket was removed entirely, cutting storage costs alongside compute costs.
Converting each record to a single line means a bad record is skipped rather than collapsing a block. Erroneous records are captured to a separate stream so product engineers can identify exactly which device, model, firmware version, and release is throwing failures.
A Smartsheet interface lets product engineers request logs by model name and firmware version. The system polls for new requests, compresses the relevant files, and delivers them by email automatically — no human intervention required at any step.
The old and new pipelines ran in parallel against a shadow Snowflake sink for one week of validation. Once results confirmed the redesign's quality and cost advantages, the new flow replaced the original with no interruption to live device telemetry.
Lower ingestion cost
Monthly data acquisition and ingestion fell from $22,000 to $5,000.
Fewer data errors
Failed records retained and diagnosable by device, firmware, and release.
Assessment to production
End-to-end delivery with zero downtime at cutover.
IoT rule
Billable rules consolidated from 20 to one, eliminating redundant per-rule charges.
Whether you’re starting with data modernization or exploring AI copilots, we’re here to help.
Contact Us
