1. Summary

Ready incorporates three data sources in the current system to create the seven categories of performance indicators. The data includes FCC’s Form 477, Ookla’s global fixed broadband network performance, and Measurement Lab (M-Lab)’s Network Diagnostic Tool (NDT). This article explains the step-by-step methodology. 

You can use the data in Broadband.money’s maps to win IIJA BEAD grants.

Fig. 1 A screen capture of current performance indicators shown in BBM. The highlighted census block is classified as “Arguably Unserved” by Ready. We provide detailed data from FCC's form 477, Measurement Lab's data, and Ookla to support this classification.

The following section first explains the method to aggregate speed test data to the Census Block Level, then explains the construction of the Ready Speed Rank.

2. Method

2.1 Data Aggregation

2.1.1 Ookla

The original Ookla global fixed broadband network performance data is distributed through vector tiles (end note). Each tile is approximately 2003.937 feet by 2003.937 feet at the equator. The data contains the average download speed, upload speed, and latency of all tests performed in the tile, number of tests, number of devices, and time when the data was delivered. Ready re-interpolates the data to Census Block (CB) level considering all tiles intersected with a census block. We currently provide a list of attributes for Ookla data for each of the four quarters of 2021. Noted that for the census blocks where no speed tests are available, Ready uses the speed test data interpolated from the larger Census Block Group (CBG) level instead. The resolution of the data can be found under the Speed Source attribute. In other words, we provide an indicator to differentiate if the data is a census block level estimate or a census block group estimate.

2.1.2 M-Lab

The M-Lab’s Network Diagnostic Tool (NDT) provides download speed, upload speed, latency, and packet loss rate metrics for each speed test result collected. NDT server collects TCP statistics using TCP_INFO, via three protocols (in chronological order): web100, ndt5, and ndt7. Given both privacy concerns and the robustness of the data, M-Lab only provides the location and measured speeds of tests aggregated at certain geographic levels such as the county, census block group, or zip code. Ready first identifies census blocks with their counties and provides summary statistics of all tests taken within the county. Specifically, the number of speed tests taken within a census block is computed based on the proportion of the population of that census block to the population of its associated county. Similar to Ookla, Ready also provides quarterly updates to reflect the potential changes in Internet services.

When there are no speed tests captured within an area, Ready builds a prediction model to estimate the local speed test results using the socio-economic attributes from American Community Survey (5-year, 2015-2019) and historical speeds from both Ookla and M-Lab. The Speed Source attribute indicates if the data is collected directly from original speed test vendors or computed internally by Ready.

2.1.3 FCC’s Form 477

FCC’s Form 477 (as of June 30, 2021) provides ISP’s advertised download speed and upload speed for residential and business services at the census block level. We only consider the residential service to create the Ready Speed Rank. The detailed specification of the data can be seen here: Data as of June 30, 2021. (Noted that NTIA's current broadband indicator map uses the data as of Dec 31, 2020.) We do not include satellite providers in this consideration. In addition, to make sure that the maximum download and upload speed is provided by the same provider, we first sort the data by providers' advertised download speed to find the highest download speed and then report the associated upload speed accordingly. 

2.2 Construct the Ready Speed Rank

We present the logic behind the Ready Speed Rank in Fig 2. According to the NOFO, unserved locations lack access to Reliable Broadband Service at speeds of at least 25 Mbps downstream and 3 Mbps upstream and low latency levels to support real-time, interactive applications. Underserved locations are those without broadband service offering speeds of 100 Mbps downstream/20 Mbps upstream. We adopt a decision tree method to create the Ready Speed Rank categories.

Fig. 2 Ready Speed Rank Decision Tree Process. “FCC Ad Speed” stands for advertised consumer download and upload speed from FCC's form 477. 

2.3 Differences between Ookla and M-Lab’s measure

According to M-Lab official documentation, M-Lab’s “NDT is a single stream performance measurement of a connection’s capacity for “bulk transport”, which measures the “inter” part of the internet connection.

On the contrary, Ookla’s speedtest.net provides data from servers near the user. In short, M-Lab represents the performance of the whole network, while Ookla provides better evidence on the last-mile connection. Ready provides both measures as a comparison to show the potential places needing better Internet services.

2.4 Data Verification

Estimating Internet performance is a task currently with no ground truth. To demonstrate the validity of our speed rank categories, we produce a case study that compares address-level Internet performance survey results and our performance indicators. For detailed analysis, please refer to this post here. 

3. Known Limitations

3.1 Location precision

Although M-Lab provides geolocation of each speed test’s associated IP address, the location (pre-aggregated at zip code level) might not reflect the actual location of where the speed test was conducted.

3.2 Differentiation between business vs. residential

The speed test data from both Ookla and M-Lab do not indicate if the data was collected from a commercial building or a residential building. This may lead to overestimates of performance for residential areas with office buildings nearby.

3.3 Number of tests conducted

Many census blocks do not have enough tests conducted (we consider at least 30 samples as “enough"). We suggest accounting for this in any decision-making, especially in low-income communities.

3.4 No data or no services

Speed tests result cannot tell us whether a location is short of tests or has no Internet services. Ready suggests that local communities should coordinate with state-level broadband offices and dedicate additional resources to locations with no tests reported.

3.5 Unreported Provider in FCC’s Form 477

The current advertised speed from FCC is limited to the latest edition of the released Form 477 data. If a provider hasn’t submitted its latest Form 477, we have not yet incorporated their services. Ready will further enhance this aspect once the data is validated.

End Notes

1. Vector tile is a way to deliver geographic data in small chunks to a browser or other client application.
2. The single stream “speed” metric doesn’t measure link capacity but rather how well a single TCP stream can take advantage of that capacity.