Measurement Methodology

There are various differences in the methodologies used by different network performance testing tools. While there many of these tests available, the two most commonly utilized methodologies are: 

1. NDT7 from Measurement Lab
2. Speedtest from Ookla

We recommend developing and deploying measurement tools that are extensible to include multiple methodologies. At a minimum, we recommend using tools that collect data using both the NDT7 and Speedtest methodology. 

Additional methodologies can be included, although further vetting should be undertaken before relying on data from other methodologies to form conclusions and inform decisions. 

Limitations of Data

While there are public datasets available from Measurement Lab and Ookla, there are several well-documented limitations to this data, including: 

Lack of metadata and contextual data - The data included in the public datasets does not include information that is necessary to fully understand the results: 

• Address-level geographic granularity 
• Type of Location - Is it a business, residence, anchor institution?
• Service subscribed to by the participant 
  • What is the tier of service?
  • Is this the highest tier available? 
  • What is the technology type of service? 
  • Who is the ISP? 
• What devices are being used for the test? 
  • Personal computing device
  • Modem/Router 
• In-home variables
  • How is the device connected? 
    • Is it disconnected from VPN? 
  • Was there crosstalk from other devices or users? 
  • Were there other bandwidth consuming processes running on the device? 

Reason for taking the test is unknown - Speed tests are initiated for a wide range of reasons, including testing out a new service connection or device, troubleshooting poor performance, assessing the performance of WiFi in different locations throughout the home, etc. 

Longitudinal data is harder to come by - Network performance measurements reflect a snapshot in time that may not accurately represent the typical performance of that connection. This makes it difficult to draw meaningful conclusions from a single speed test.

Data Collection Methods 

In order to address the limitations of the public datasets, many governmental entities and third-parties are launching their own speed test surveys to collect more local and complete datasets. When designed correctly, locally deployed speed test surveys can address many of these limitations.

Browser-Based Survey

There are several approaches to gathering this speed test data, but the most common is deploying a browser-based speed test survey that relies on the users completing the tests on their own. While this is not a perfect solution by itself, it is one that can be practically implemented and used to gather reliable data. 

The reliability of the data collected from these surveys depends on the implementation of a sound methodology. We recommend implementing the following steps: 

1. Include instructions for the user to run an accurate test: 
   1. If you are able to do so, please connect your device to the Internet using an Ethernet cable plugged into the modem or router where you receive the primary connection from your provider. (link to instructions)
   2. If you are not able to connect by Ethernet, please bring your device as close as safely possible to the Wi-Fi enabled router. If you have multiple routers, please use the one where you receive the primary connection from your provider. 
   3. Please make sure that no one in your household is streaming videos or gaming while you take this test. 
   4. If you are connected to school or work, please turn off your VPN during this test. If you don't know what a VPN is, then you are ready to take the test.
2. Confirm whether those steps were taken:
   1. Before beginning please confirm the following: 
      1. Checkboxes: Are you connected by [Ethernet], [Wi-Fi in the same room as the router], [Wi-Fi in a different room than the router], [Unable to Confirm]
      2. Checkboxes: Is anyone else in the house using the Internet for streaming, gaming, or video calls? [Yes], [No], [Unable to Confirm]
      3. Checkboxes: Are you disconnected from any VPNs? [Yes], [No]
3. Ask the contextual/metadata questions: 
   1. What is your address? 
   2. What tier of service are you subscribed to? 
      1. Download: [text box]
      2. Upload: [text box]
      3. Check box if you are unsure
   3. What type of device are you currently using?
   4. Who is your ISP?
   5. Is this the highest tier of service available from your provider?
   6. To the best of your knowledge what type of connection do you have? / Who is your provider? 
4. Encourage Multiple Tests for Longitudinal Data 
   1. (Optional) Please provide your email address [text box]
      1. *The data from this survey is being used to identify areas that lack reliable broadband access. Taking this test several more times in coming days and weeks provides more reliable data. We will only use your email address to send reminders to take the test again or follow-up to improve the quality of the test. 
5. Ask additional questions related to digital equity indicators
   1. How many people live in your home? 
   2. How many students? Seniors? Work from home? 
   3. Number of devices? 
   4. Primary uses of the Internet in your home? 
      1. Telehealth
      2. Education
      3. Work
      4. Entertainment
      5. Communication
   5. Are you satisfied with your Internet service for your needs? 
      1. Speed 
      2. Reliability
   6. Cost
      1. How much do you currently pay (per month) for your broadband internet service (choose one):
         1. $0 - $25
         2. $25 - $50
         3. $50 - $75
         4. $75 - $100
         5. $100 - $125
         6. $125+
      2. How much (per month) would you be willing to pay for high-quality broadband internet service that meets your needs/requirements?
         1. $50 - $75
         2. $75 - $100
         3. $100 - $125
         4. $125 - $150
         5. $150 +
6. Follow-up with willing participants to collect verified data. 
   1. Fixed device based measurement in potentially unserved/underserved areas. 
   2. Hands on assistance with speed tests.