In short 

Verizon runs scheduled speed tests against hundreds of Fixed Wireless Access devices at a time on vPIM, the network measurement platform we have built and operated with them since 2012. As the batches grew, only about 60% of tests produced a valid result and the plan called for close to ten new clusters. 

Three rounds of log analysis showed the server was not short of threads or memory but was letting stalled client connections hold its worker threads for over three minutes. A handful of small changes in vPIM – mainly bounding how long a connection may hold a thread – brought the success rate to 97% at five times the load. 

The traffic now runs on two clusters, and the same analysis method later proved that the failures still seen in one region come from the network path, not the platform.

Why It Matters: Diagnostic Data Requires 100% Trust

vPIM (Verizon Platform & Infrastructure Measurement) is the diagnostic and measurement platform we have been building and operating with Verizon since 2012. Its Data Collectors are the servers that customer devices talk to when they run a speed test. Verizon uses these tests in scheduled batch cycles, a few hundred devices per cycle, to watch the quality of the wireless network from the customer’s side.

A test that fails is a blind spot: a device whose real-world throughput Verizon cannot see. When failure rates hit double digits, the whole measurement program loses its value. Fixing these drop-offs became the top priority on our shared agenda.

The Challenge: High Failure Rates at Batch Scale

When batch sizes scaled from 50 to 300 devices per run, platform performance degraded sharply:

• High Request Density: A single cycle is more than 4,000 HTTP connections within a few minutes, and cycles follow each other around the clock. At peak wave, one Data Collector accepts close to 2,000 tests per hour (~25,000 HTTP transfers), and over the eighteen days of our analysis, the four collectors served 3.7 million transfers. 
• High Failure Rates: Only 60% of tests succeeded on busy collectors (falling to ~50% on worst days). Many devices reported Completed but returned empty measurement windows.
• The Expected Fix: The original plan assumed the platform was undersized and required expanding to nearly ten regional clusters.  
• The Goal Agreed with Verizon: Bring the success rate back to a level that makes the measurement program trustworthy, at the batch sizes they actually wanted to run, and find out what was happening before anyone bought hardware.

The Investigation: Deep Log Forensics at Scale

We built a custom streaming analyzer to process millions of raw server interactions across 18 days of logs (3.7 million total transfers). Reconstructing the exact lifecycle of every HTTP request revealed three critical findings.

Finding 1: JVM Defaults Sized for a Quieter World

The first logs, from June, showed the Data Collectors running out of Java heap during batch windows. The JVM was running with its default heap settings, sized for the modest load of individual on-demand tests, not for hundreds of devices opening seven connections each within the same minute. The endpoint tables in the local database had also grown large over years of operation, and the periodic reachability checks were loading them whole.

• Fix: Verizon raised the heap to 8 GB on the affected collectors and cleaned up the endpoint tables. On our side, vPIM 2.2.5 had already changed the reachability checks to work in pages and to purge stale endpoints first, cutting the Data Collector’s baseline memory footprint, so we asked for the upgrade to be brought forward.
• Result: The success rate jumped within days. It also, as it turned out, revealed the real problem underneath.

Finding 2: The Bottleneck Was Hold Time, Not Thread Count

With memory no longer the bottleneck, the July logs let us look at the thread pool properly. Jetty’s default pool on the Data Collector has 254 threads, of which about 226 are available for transfers. During bad hours the pool was pinned at 226, incoming connections queued for minutes, and when they were finally handled their reservations had expired, so they were rejected in bursts of thousands per second. 

While the natural conclusion was “254 threads is too few,” our duration histogram told a different story:

• Transfers did not spread across a range of durations. They took either about five seconds or about two hundred seconds, with almost nothing in between. Two hundred seconds is Jetty’s default connector idle timeout. 
• A steady 13–14% of connections on every collector stalled after the client stopped reading or writing, and each of them held a worker thread for those two hundred seconds while moving roughly five kilobytes per second. Stalled connections accumulated until the pool was full of them, and legitimate tests waited behind.

The Math Behind the Bottleneck

Little’s law says the number of busy threads equals the arrival rate times the hold time. At the busiest observed minute, around 300 accepted test-directions per minute times seven connections times a five-second hold, the pool needs about 180 threads. That fits comfortably in 254. Multiply the hold time by forty, which is what a two-hundred-second stall does, and no reasonable pool size can save you. The pool was not too small. The hold time was too long.

We confirmed the mechanism outside production with a harness that ran the unmodified Data Collector classes against a load generator holding a configurable share of connections open. Connections were accepted by the operating system within milliseconds and handled by the server 6 to 16 seconds later, once a thread became free. The delay was in the queue, not in the network.

Finding 3: Bursts, Not Uptime

Verizon’s next observation was that the success rate looked good after a restart and degraded over the following week, which sounded like a leak. The July logs let us test that directly.

We picked pairs of hours on the same collector with the same number of accepted tests, one with a high success rate and one with a low one, and tracked the busy thread count minute by minute. Both hours touched the 226-thread cap. 

The difference was how long they stayed there. In the good hour, occupancy oscillated and transfers kept starting every minute. In the bad hour, the pool sat at the cap for 45 of the 60 minutes, and there were whole minutes in which tests kept arriving and not a single transfer could start. Same load, same stall rate, opposite outcome, decided by whether the stalled connections happened to overlap.

That explained the “degrades over time” impression without any leak. Peak load in the bad week was lower than in June, the overnight thread floor showed no upward trend across five weeks, and a collector with zero memory errors degraded exactly like one with dozens. 

On the worst days the saturation tipped into a full wedge: every worker thread stuck at once, and the collector served nothing for up to three days until the dead sockets timed out at the kernel level, recovering on its own without a restart. 

A system that behaves like that is not short of capacity. It is missing a bound on how long one connection may hold a thread.

The Solution: Targeted Software Adjustments

In plain terms: a speed test is supposed to occupy the server for five seconds. Some devices stopped talking halfway through, and the server was waiting more than three minutes for each of them before giving up. 

We taught it to give up after a few seconds, made sure every connection is closed after a fixed maximum time no matter what, and trimmed the memory the collector uses for housekeeping. That is the whole change, and it is why no new servers were needed.

For the technically inclined, the fixes followed directly from the findings, and they were small.

None of these changes adds servers, threads, or memory beyond the heap adjustment. They change how long a single connection is allowed to occupy a resource, which is the variable the arithmetic pointed at.

The Results: 97% Success Rate Across Two Clusters

Verizon deployed vPIM 2.2.8 on the FWA Data Collectors in August. In September, we received a fresh set of logs, 18 days from 4 collectors, and ran the same analysis as in June and July.

• Longest HTTP transfer: 35 minutes before, 9.6 seconds after, across 3.7 million transfers.
• Stalled connections holding a thread for over 150 seconds: 13.7% before, zero after.
• Rejected connections: 17% of all connections before, about 1% after, and those are scanners and a handful of edge cases rather than real devices.
• Success rate on the Ohio collectors: about 60% before, 97% in September, at a higher batch load than in June.
• Thread pool: touches the 226 cap only in short burst minutes at the top of a batch wave, with zero rejections in those minutes.

And the number that matters most to the client: two production clusters are handling the load that was originally planned for close to ten.

What this means for Verizon in practice:

• Trustworthy measurements. Batch cycles of several hundred devices now return a valid result for 97 out of 100 tests, so the FWA quality program can rely on its own data.
• Infrastructure that was never built. The capacity plan of close to ten regional clusters, with their hardware, hosting and operations cost, was replaced by targeted changes in software and configuration on the existing two.
• No more silent outages. The multi-day wedges in which a collector served nothing are gone, along with the restarts and manual intervention they caused.
• Headroom for growth. With transfers bound at seconds, the same collectors have room for larger batches, and we know exactly where the next limit is.
• Faster answers. Each new set of logs is turned into the same tables within an hour, so questions such as “is this a platform problem or a network problem” are answered with evidence instead of debate.

Knowing When It’s Not You: Isolating External Network Faults

The September logs also showed the other side of good observability. One region still had a low success rate, and because the platform-side metrics were now clean, we could look past the server.

Matching device results with the requests the collector had received showed that most failed tests in that region were tests in which only six of the seven connections had reached the collector. The device reported a normal 30 ms TCP handshake, the six connections that arrived were served normally, and the seventh never produced a request. 

The missing connections were concentrated in specific device address pools, at rates that were identical towards both collectors in the region and absent everywhere else. That is the signature of a network path problem, not a server problem, and the analysis handed Verizon’s network team a precise list of what to look at.

The same pass also identified the next bottleneck on our side before it becomes a customer-visible one: at the very top of the largest batch waves, the per-connection transfer buffer configured for FWA tests puts pressure on the heap. It is already on our roadmap. That is what a stable platform buys you: the ability to see the next problem while it is still small.

Platform Architecture Insights: How to Scale Systems Without Adding Infrastructure 

Measure the hold time, not the thread count. Every capacity conversation about this platform started with “how many threads do we need” and ended with “how long does one connection hold a thread”. Cutting the hold time from 200 seconds to 5 did what no number of extra threads could have done.

Fix the mechanism, not the symptom. More clusters would have masked the stalls by spreading them thinner, at several times the infrastructure cost, and the wedge would still have been possible on every one of them.

Repeatable analysis is a deliverable. Because each round of logs was turned into the same tables, we could show the client not only that a fix worked but also, in the end, that a remaining problem lived outside the platform. Evidence of that kind is what turns a vendor into a partner.

Design for the client that misbehaves. One in seven device connections stalled, every day, on every collector. A server that handles requests synchronously has to assume that some of its clients will simply stop talking, and bound the cost of that.

Technologies used

Scale Your Platform Without Unnecessary Infrastructure Costs

vPIM is one of several systems we build and run for Verizon. If you are curious about the wider partnership, read how SolDevelo provides Verizon with scalable cloud development services. 

If your platform is facing scaling limits or rising infrastructure costs, get in touch with SolDevelo. We specialize in finding hidden capacity in existing architectures.