Identifying <a href="https://broadband.money/broadband-grant-terms/unserved-location">unserved </a>and <a href="https://broadband.money/broadband-grant-terms/underserved-location">underserved </a>communities is the central challenge for policymakers when deciding where to allocate federal broadband funding flowing from the BEAD grants.&nbsp;To bring more precision to the process, Broadband Money uses multiple sets of data that measure network performance to build the Ready Speed Rank. We have combined datasets collected by <a href="https://www.measurementlab.net/">Measurement Lab</a> (MLab,) <a href="https://www.ookla.com/articles/how-ookla-ensures-accurate-reliable-data-2021">Ookla</a>, and the Federal Communications Commission’s <a href="https://www.fcc.gov/general/broadband-deployment-data-fcc-form-477">form 477 data</a>. The benefit of this approach is that it provides a multi-faceted view of geographic areas – instead of unchecked, self-reported data from providers that is submitted in the form 477 data collection process.&nbsp;Here’s how the Ready Speed Rank system is better than the FCC's when used as a sole source of truth.In this article, we compared the FCC’s form 477 data to the address level speed test <a href="https://publicservice.vermont.gov/content/broadband-availability" target="_blank" rel="noopener noreferrer">survey data</a> collected by the state of Vermont. (Remember that the FCC’s form 477 data relies instead on block-level reporting from providers.) Then we compare the Ready Speed Rank to the FCC's form 477 reports.&nbsp;When we performed this comparison, we found that 41.5% (25,359) of the unserved addresses (service tier below 25/3 Mbps) in Vermont would be classified as "served" if we only relied on the FCC's form 477 data. In contrast, the Ready Speed Rank only shows 0.6% (371) of the unserved addresses from the Vermont survey within blocks that we define as served.<h2>Compare the FCC’s Form 477 block level speed report with the address-level speed survey from the state of Vermont.</h2>We first associate the address-level speed report data from the state of Vermont to US Census Block data. A total of 310,633 addresses were included in the Vermont study, among which 61,194 (19.7%) households are reported unserved (service tier below 25/3 Mbps.) For each census block, we obtain the FCC 477 data, including the consumer level's maximum download and upload speed.Fig. 1 shows a mapping comparison of the broadband services from FCC's form 477 reports versus data from Vermont's current address level speed survey.&nbsp;Table 1 summarizes the results. We show that around 41.5% (25,359) of "unserved” addresses are currently classified as "served” per FCC’s current 477 data.<figure class="image"><img src="https://storage.googleapis.com/bbm-public-bucket/why_Ready_Speed_Rank_01_4942b0deb0.png"></figure>Fig. 1. Compare the FCC’s Form 477 report (census block level) and Vermont address-level speed survey data.<figure class="image"><img src="https://storage.googleapis.com/bbm-public-bucket/report_ready_vermont_c1e2c7b8a5.png"></figure>Table 1. Percentage of addresses classified into different categories based on the FCC form 477 reports. Around 41.51% of currently unserved addresses (reported by Vermont Speed Survey) are within the "served" territory based on FCC's 477 records. Number in each row will add up to 100%<h2>Compare the Ready Speed Rank with Vermont's address-level survey.</h2>Ready Speed Rank is a metric that integrates FCC 477 form data and Ookla, and MLab speed test data. If the FCC’s 477 report indicates no provider in a census block, we assume the census block is “unserved”. If a census block is reported with upload and download speeds above 25 Mbps/3 Mbps by FCC 477, but MLab and Ookla imply otherwise, we consider this census block “likely unserved”. The Ready Speed Rank only considers a census block as "served" when all three data sources report that a census block is “served.” In this way, we mitigate the bias brought by potential sampling errors and different speed test methods. Again, Fig. 2a shows the Ready Speed Rank distribution in Vermont, and Table 2 reports the confusion matrix*.&nbsp;<figure class="image"><img src="https://storage.googleapis.com/bbm-public-bucket/fig_02_f402767eb6.png"></figure>Fig. 2. Distribution of Ready Speed Rank<figure class="image"><img src="https://storage.googleapis.com/bbm-public-bucket/report_confusion477_vermont_f82392aa9e.png"></figure>Table 2. Compare Ready Speed Rank with Vermont speed survey. We can reduce the miss-classification of unserved addresses to 0.61%. By combining other metrics, Ready Speed Rank can capture unserved addresses under the “likely" and "arguably" categories.To conclude, by leveraging multiple data sources, Ready Speed Rank provides better estimates of the unserved and underserved communities. This is the first of our case studies to demonstrate how our team leverages data so that policymakers and grant seekers and grant makers can make better funding allocation decisions. &nbsp;<h4>Data Used in this Report</h4><ul><li>Vermont Address Level Speed Data: <a href="https://publicservice.vermont.gov/content/broadband-availability" target="_blank" rel="noopener noreferrer"><code>https://publicservice.vermont.gov/content/broadband-availability</code></a></li><li>Category of served, unserved, underserved with FCC 477 (redistributed by NTIA): <a href="https://broadbandusa.ntia.doc.gov/census-block-lists-state-and-territory" target="_blank" rel="noopener noreferrer"><code>https://broadbandusa.ntia.doc.gov/census-block-lists-state-and-territory</code></a></li><li>Ready Speed Rank &nbsp;</li></ul><h4>Footnote:</h4>*Confused? A “confusion matrix” is a data science term that summarizes the results of a classification task. Each address point could be classified as “served”, “underserved”, and “unserved” per Vermont’s speed survey and Ready Speed Rank. The matrix shows the proportion of addresses classified into the same class or different classes by the two methods.

Identifying unserved and underserved communities is the central challenge for policymakers when deciding where to allocate federal broadband funding flowing from the BEAD grants. The Ready Speed Rank pinpoints unserved and underserved communities.

Why Ready Speed Rank?

why-ready-speed-rank

<h1>1. Summary</h1>Ready incorporates three data sources in the current system to create the seven categories of performance indicators. The data includes FCC’s Form 477, <a href="https://registry.opendata.aws/speedtest-global-performance/" target="_blank" rel="noopener noreferrer">Ookla</a>’s global fixed broadband network performance, and <a href="https://www.measurementlab.net/data/" target="_blank" rel="noopener noreferrer">Measurement Lab</a> (M-Lab)’s Network Diagnostic Tool (NDT). This article explains the step-by-step methodology.&nbsp;You can use the data in Broadband.money’s maps to win IIJA BEAD grants.<figure class="image"><img src="https://storage.googleapis.com/bbm-public-bucket/readyspeedrank2_a04c6e91be.png"></figure>Fig. 1 A screen capture of current performance indicators shown in BBM. The highlighted census block is classified as “Arguably Unserved” by Ready. We provide detailed data from FCC's form 477, Measurement Lab's data, and Ookla to support this classification.The following section first explains the method to aggregate speed test data to the Census Block Level, then explains the construction of the Ready Speed Rank.<h1>2. Method</h1><h2>2.1 Data Aggregation</h2><h3>2.1.1 Ookla</h3>The original Ookla global fixed broadband network performance data is distributed through vector tiles (end note). Each tile is approximately 2003.937 feet by 2003.937 feet at the equator. The data contains the average download speed, upload speed, and latency of all tests performed in the tile, number of tests, number of devices, and time when the data was delivered. Ready re-interpolates the data to Census Block (CB) level considering all tiles intersected with a census block. We currently provide a list of attributes for Ookla data for each of the four quarters of 2021. Noted that for the census blocks where no speed tests are available, Ready uses the speed test data interpolated from the larger Census Block Group (CBG) level instead. The resolution of the data can be found under the Speed Source attribute. In other words, we provide an indicator to differentiate if the data is a census block level estimate or a census block group estimate.<figure class="table"><table style="border:0.1px solid hsl(0, 0%, 60%);"><thead><tr><th style="background-color:hsl(0, 0%, 90%);border:0.1px solid hsl(0, 0%, 30%);">Attributes</th><th style="background-color:hsl(0, 0%, 90%);border:0.1px solid hsl(0, 0%, 30%);">Description</th></tr></thead><tbody><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum Download Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum download speed of all tests performed in the tiles that intersect with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum Download Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum download speed of all tests performed in the tiles that intersect with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Median Download Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Median download speed of all tests performed in the tiles that intersect with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum Upload Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum upload speed of all tests performed in the tiles that intersect with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum Upload Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum upload speed of all tests performed in the tiles that intersect with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Median Upload Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Median upload speed of all tests performed in the tiles that intersect with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Latency</td><td style="border:0.1px solid hsl(0, 0%, 30%);">The average latency of all tests performed within the tiles intersected with a census block, represented in milliseconds</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Speed Category Ookla</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Performance categories are summarized based on the median download and upload speed. A census block with a speed test lower than 25/3 is unserved, 100/20 underserved, and served otherwise.</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Number of Tests</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Total number of tests conducted within the tiles intersected with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Number of Devices</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Total number of distinct devices participated the speed tests within the tiles intersected with a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Source</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Details on data extraction method, indicating if the data is directly extracted from the vector tiles or re-interpolated using the speed test data from the adjacent census block groups.</td></tr></tbody></table></figure><h3>2.1.2 M-Lab</h3>The M-Lab’s Network Diagnostic Tool (NDT) provides download speed, upload speed, latency, and packet loss rate metrics for each speed test result collected. NDT server collects TCP statistics using <a href="https://www.measurementlab.net/tests/tcp-info/" target="_blank" rel="noopener noreferrer">TCP_INFO</a>, via three protocols (in chronological order): <a href="https://www.measurementlab.net/tests/ndt/web100/" target="_blank" rel="noopener noreferrer">web100</a>, <a href="https://www.measurementlab.net/tests/ndt/ndt5/" target="_blank" rel="noopener noreferrer">ndt5</a>, and <a href="https://www.measurementlab.net/tests/ndt/ndt7/" target="_blank" rel="noopener noreferrer">ndt7</a>. Given both privacy concerns and the robustness of the data, M-Lab only provides the location and measured speeds of tests aggregated at certain geographic levels such as the county, census block group, or zip code. Ready first identifies census blocks with their counties and provides summary statistics of all tests taken within the county. Specifically, the number of speed tests taken within a census block is computed based on the proportion of the population of that census block to the population of its associated county. Similar to Ookla, Ready also provides quarterly updates to reflect the potential changes in Internet services.When there are no speed tests captured within an area, Ready builds a prediction model to estimate the local speed test results using the socio-economic attributes from American Community Survey (5-year, 2015-2019) and historical speeds from both Ookla and M-Lab. The Speed Source attribute indicates if the data is collected directly from original speed test vendors or computed internally by Ready.<figure class="table"><table style="border:0.5px solid hsl(0, 0%, 0%);"><thead><tr><th style="background-color:hsl(0, 0%, 90%);border:0.1px solid hsl(0, 0%, 30%);">Attributes</th><th style="background-color:hsl(0, 0%, 90%);border:0.1px solid hsl(0, 0%, 30%);">Description</th></tr></thead><tbody><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum Download Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum download speed of all tests performed within a census block group or the smallest associated geographical unit</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum Download Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum download speed of all tests performed within a census block group or the smallest associated geographical unit</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Median Download Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Median download speed of all tests performed within a census block group or the smallest associated geographical unit</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Latency</td><td style="border:0.1px solid hsl(0, 0%, 30%);">The average latency of all tests performed within a census block group or the smallest associated geographical unit, represented in milliseconds.</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Packet Loss Rate</td><td style="border:0.1px solid hsl(0, 0%, 30%);">The average loss rate from the lifetime of the connection</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum Upload Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Minimum upload speed of all tests performed within a census block group or the smallest associated geographical unit</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Median Upload Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Median download speed of all tests performed within a census block group or the smallest associated geographical unit</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum Upload Speed</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Maximum download speed of all tests performed within a census block group or the smallest associated geographical unit</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Number of Tests</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Number of tests taken within a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Number of Devices</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Number of devices participating the speed test within a census block</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Source</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Details on data extraction method, indicating if the data is directly aggregated using the speed test data within each census block group or Ready predicts the performance with adjacent data available</td></tr><tr><td style="border:0.1px solid hsl(0, 0%, 30%);">Speed Category M-Lab</td><td style="border:0.1px solid hsl(0, 0%, 30%);">Performance categories are summarized based on the median download and upload speed. A census block with a speed test lower than 25/3 is unserved, 100/20 underserved, and served otherwise.</td></tr></tbody></table></figure><h3>2.1.3 FCC’s Form 477</h3>FCC’s Form 477 (as of June 30, 2021) provides ISP’s advertised download speed and upload speed for residential and business services at the census block level. We only consider the residential service to create the Ready Speed Rank. The detailed specification of the data can be seen here: <a href="https://opendata.fcc.gov/Wireline/Fixed-Broadband-Deployment-Data-December-2020/hicn-aujz" target="_blank" rel="noopener noreferrer">Data as of June&nbsp;30, 2021</a>. (Noted that <a href="https://broadbandusa.maps.arcgis.com/apps/webappviewer/index.html?id=e2b4907376b548f892672ef6afbc0da5" target="_blank" rel="noopener noreferrer">NTIA's current broadband indicator map</a> uses the data as of Dec 31, 2020.) We do not include satellite providers in this consideration. In addition, to make sure that the maximum download and upload speed is provided by the same provider, we first sort the data by providers' advertised download speed to find the highest download speed and then report the associated upload speed accordingly.&nbsp;<h2>2.2 Construct the Ready Speed Rank</h2>We present the logic behind the Ready Speed Rank in Fig 2. According to the <a href="https://broadbandusa.ntia.doc.gov/sites/default/files/2022-05/BEAD%20NOFO.pdf" target="_blank" rel="noopener noreferrer">NOFO</a>, unserved locations lack access to Reliable Broadband Service at speeds of at least 25 Mbps downstream and 3 Mbps upstream and low latency levels to support real-time, interactive applications. Underserved locations are those without broadband service offering speeds of 100 Mbps downstream/20 Mbps upstream. We adopt a decision tree method to create the Ready Speed Rank categories.<figure class="image"><img src="https://storage.googleapis.com/bbm-public-bucket/Ready_Speed_Rank_e7e0121ac5.png"></figure>Fig. 2 Ready Speed Rank Decision Tree Process. “FCC Ad Speed” stands for advertised consumer download and upload speed from FCC's form 477.&nbsp;<h2>2.3 Differences between Ookla and M-Lab’s measure</h2>According to M-Lab official documentation, M-Lab’s “NDT is a single stream performance measurement of a connection’s capacity for “bulk transport”, which measures the “inter” part of the internet connection.On the contrary, Ookla’s <a href="http://speedtest.net/">speedtest.net</a> provides data from servers near the user. In short, M-Lab represents the performance of the whole network, while Ookla provides better evidence on the last-mile connection. Ready provides both measures as a comparison to show the potential places needing better Internet services.<h2>2.4 Data Verification</h2>Estimating Internet performance is a task currently with no ground truth. To demonstrate the validity of our speed rank categories, we produce a case study that compares address-level Internet performance survey results and our performance indicators. For detailed analysis, please refer to this post <a href="https://broadband.money/blog/why-ready-speed-rank" target="_blank" rel="noopener noreferrer">here</a>.&nbsp;<h1>3. Known Limitations</h1><h2>3.1 Location precision</h2>Although M-Lab provides geolocation of each speed test’s associated IP address, the location (pre-aggregated at zip code level) might not reflect the actual location of where the speed test was conducted.<h2>3.2 Differentiation between business vs. residential</h2>The speed test data from both Ookla and M-Lab do not indicate if the data was collected from a commercial building or a residential building. This may lead to overestimates of performance for residential areas with office buildings nearby.<h2>3.3 Number of tests conducted</h2>Many census blocks do not have enough tests conducted (we consider at least 30 samples as “enough"). We suggest accounting for this in any decision-making, especially in low-income communities.<h2>3.4 No data or no services</h2>Speed tests result cannot tell us whether a location is short of tests or has no Internet services. Ready suggests that local communities should coordinate with state-level broadband offices and dedicate additional resources to locations with no tests reported.<h2>3.5 Unreported Provider in FCC’s Form 477</h2>The current advertised speed from FCC is limited to the latest edition of the released Form 477 data. If a provider hasn’t submitted its latest Form 477, we have not yet incorporated their services. Ready will further enhance this aspect once the data is validated.&nbsp;<h1>End Notes</h1><ol><li>Vector tile is a way to deliver geographic data in small chunks to a browser or other client application.</li><li>The single stream “speed” metric doesn’t measure link capacity but rather how well a single TCP stream can take advantage of that capacity.</li></ol>

Methodology of creating the Ready Speed Rank using multiple data sources.

Broadband Grant Expert

Zhuangyuan Fan

Lessons

Articles

Enroll for free today