Measurement Methodology

There are various differences in the methodologies used by different network performance testing tools. While there many of these tests available, the two most commonly utilized methodologies are: 

1. NDT7 from Measurement Lab
2. Speedtest from Ookla

There are key differences between NDT7 and Speedtest by Ookla, each of which provides valid and valuable insight into the network performance. To get the most robust dataset, we recommend developing and deploying measurement tools that are extensible to include multiple methodologies. 

At a minimum, we recommend using tools that collect data using a variety of methodologies, including those used by NDT7 and Speedtest methodology. Additional methodologies can be considered, although further vetting should be undertaken before relying on data from other methodologies to form conclusions and inform decisions. 

Limitations of Data

While there are public datasets available from Measurement Lab and Ookla, there are several well-documented limitations to this data, including: 

1. Lack of metadata and contextual data – The data included in the public datasets does not include information that is necessary to fully understand the results, such as: 

• Address-level geographic granularity 
• Type of Location – Is it a business, residence, anchor institution,?
  • If an anchor institution, identify whether it currently subscribes to mass-market retail broadband services.
• Service subscribed to by the participant
  • What is the tier of service?
  • Is this the highest tier available? 
  • What is the technology type of service? 
  • Who is the ISP? 
• Which devices are utilized for the test?
  • Personal computing device
  • Modem/Router 
• Additional in-home variables:
  • How is the device connected?
    • Is it disconnected from VPN? 
  • Was there crosstalk from other devices or users? 
  • Were there other bandwidth consuming processes running on the device? 

1. Unknown reason for performing test – Speed tests are initiated for a wide range of reasons, including testing out a new service connection or device, troubleshooting poor performance, assessing the performance of WiFi in different locations throughout the home, etc. 
2. Longitudinal data is harder to come by – Network performance measurements reflect a snapshot in time that may not accurately represent the typical performance of that connection. This makes it difficult to draw meaningful conclusions from a single speed test.

Data Collection Methods 

In order to address the limitations of the public datasets, many governmental entities and third-parties are launching their own speed test surveys to collect more local and complete datasets. When designed correctly, locally deployed speed test surveys can address many of these limitations.

Browser-Based Speed Test Surveys

There are several approaches to gathering this speed test data, but the most common is deploying a browser-based speed test survey that relies on the users completing the tests on their own. While this is not a perfect solution by itself, it is one that can be practically implemented and used to gather reliable data. 

The reliability of the data collected from these surveys depends on the implementation of a sound methodology. We recommend implementing the following steps: 

1. Inclusion of the following instructions for the user to run an accurate test: 
   1. If you are able to do so, please connect your device to the Internet using an Ethernet cable plugged into the modem or router where you receive the primary connection from your provider. (link to instructions)
   2. If you are not able to connect by Ethernet, please bring your device as close as safely possible to the Wi-Fi enabled router. If you have multiple routers, please use the one where you receive the primary connection from your provider. 
   3. Please make sure that no one in your household is streaming videos or gaming while you take this test. 
   4. If you are connected to school or work, please turn off your VPN during this test. If you don’t know what a VPN is, then you are ready to take the test.
2. Confirmation of whether or not those steps were taken:
3. Before beginning please confirm the following:
   1. Checkboxes: Are you connected by [Ethernet], [Wi-Fi in the same room as the router], [Wi-Fi in a different room than the router], [Unable to Confirm]
   2. Checkboxes: Is anyone else in the house using the Internet for streaming, gaming, or video calls? [Yes], [No], [Unable to Confirm]
   3. Checkboxes: Are you disconnected from any VPNs? [Yes], [No]
4. Ask the contextual/metadata questions:
   1. What is your location?
      1. Address: [Autocomplete text box]
      2. Geolocation: [GPS-enabled]
   2. What tier of service are you subscribed to?
      1. Download: [text box]
      2. Upload: [text box]
      3. Check box if you are unsure
   3. What type of device are you currently using?
   4. Who is your ISP?
   5. Is this the highest tier of service available from your provider?
   6. To the best of your knowledge what type of connection do you have? / Who is your provider? 
5. Encourage Multiple Tests for Longitudinal Data 
   1. (Optional) Please provide your email address [text box]

*The data from this survey is being used to identify areas that lack reliable broadband access. Taking this test several more times in coming days and weeks provides more reliable data. We will only use your email address to send reminders to take the test again or follow-up to improve the quality of the test. 

Digital Equity Indicators

Collecting data and mapping broadband availability and network performance is a critical component of developing a gap analysis and developing plans. However, there are other dimensions of digital equity that are important to understand and develop plans to address, including factors that impact adoption and use of broadband services. 

If possible, consider adding questions to gain a better understanding of the digital equity indicators in the community participating in the survey. 

Some examples of questions to consider asking include: 

1. More people in a home can increase the need for faster Internet. How many people currently live at this address?*
2. Children who attend school often have an increased need for home Internet access. How many people currently living at this address are under the age of 18?*
3. How many people living at this address currently take college or university courses (even if those courses are not online)?*
4. How many people living at this address are 65 or older? 
5. How many people living at this address work from home?
6. In some situations, having a computer motivates the desire to have Internet access at home. If any, how many working computers/laptops/chromebooks do you have at home?*
7. How many tablets do you have at home? (e.g., iPad, Kindle, Android tablet)*
8. Are you satisfied with your Internet service for your needs? (5-point Likert Scale recommended)
   1. Cost
   2. Speed / Performance 
   3. Reliability
9. Cost
   1. How much do you currently pay (per month) for your broadband internet service? 
   2. Would you be willing to pay more for higher quality broadband internet service?

*Example question based on the Merit Michigan Moonshot Project, a broadband initiative run by Merit Network, a non-profit research and education network, in partnership with the Quello Center at Michigan State University.

Building Stronger Datasets to Further Verify Data 

If further verification is desired or required, there are several approaches that can be employed to ensure the data that you are getting meets an even higher methodological and evidentiary standard. 

• Hands on support – If your data collection effort has the resources to support individuals spending time in the community supporting the operation of speed tests to ensure they are run reliably and any potential bottlenecks out of the ISPs control are identified and documented. 
• Fixed device based measurement in potentially unserved/underserved areas – This approach involves installing network performance measurement devices directly to home modems/routers. It is recommended that these devices run tests at least 4 times a day at different times throughout the day over the course of 7 days.

Accessibility and Inclusivity

Measuring broadband availability, network performance, and digital equity indicators requires an approach that enables the participation of all individuals. 

• Support for languages – The survey should be included in languages other than English. Human translation is the preferred method. The specific languages included in the survey will vary by community. 
• Accessible design – The survey should follow the Web Content Accessibility Guidelines. 
• Offline Responses – Individuals without Internet access need to have a way to respond. Survey should include a box to check to say I don’t have Internet access. There should also be an option to complete the survey via physical copies, automated text messages, phone calls, or other mechanisms.