5G NR Benchmark Test Volume 23: Apples to Apples The Benchmarking Game Of Crazy Comparisons! (3 Players) - Redux

Report
July 2022
Region: Global
Signals Research Group, LLC
ID: 5639043

The publisher just completed its 23rd 5G NR benchmark study. They collaborated with Rohde & Schwarz to conduct a multi-operator benchmark study, based on ETSI TR 103 559. This study repeats an earlier study that was published in early 2021. Although the market where we tested and the operators involved remained the same, the results were completely different.

Acknowledgements

This study was conducted in collaboration with Rohde & Schwarz. R&S collected the performance data on our behalf while the publisher analyzed the data based primarily on the inherent capabilities of the R&S Smart Analytics tool. The publisher takes sole responsibility for the analysis and commentary provided in this report.

Methodology

Testing took place in early February in the Dallas - Fort Worth market using the AT&T, T-Mobile and Verizon networks. The publisher used 3 smartphones per network (voice = 2; data = 1) and R&S scanners to collect the data. In total, there were 115 hours of testing, ~1,300 voice calls per network, and more than 60,000 data sessions. The publisher separately compared T-Mobile NSA versus T-Mobile SA performance. Results of the study leveraged ETSI TR 103 559 to provide an industry-accepted and repeatable set of results.

The Results are in

This study mirrors a similar study that we conducted in 2021, making it possible to provide comparative analysis on year-over-year network performance. In the publisher's 2021 study, AT&T came out on top, followed by Verizon with T-Mobile lagging well behind. In this most recent study, T-Mobile took top honors by a considerable margin. In both studies, the voice results were largely comparable across operators with meaningful differentiation only occurring with the data tests.

Chalk it up to 5G

All three networks exhibited meaningful increases in the use of 5G. For T-Mobile, there was a double-digit increase in 5G usage, plus a dramatic switch from Band n71 (2021) to Band n41 (2022). Interestingly, there was very little Band n77 present for AT&T and Verizon. The latter is most surprising given what we observed in our last Signals Ahead report, which involved testing in the Minneapolis area.

NSA versus SA

T-Mobile hadn't officially launched SA in this market but that didn't prevent us from separately testing the new architecture - we excluded these results when comparing performance with the other two networks. Suffice it to say, the SA performance wasn't as good as NSA, both in terms of throughput-related metrics and latency-related metrics. The ETSI framework focuses on the user experience, but there is still a throughput and/or latency/jitter component with most use cases under test.

Video Rules

All operators exhibited good performance gains with video, in particular increased use of higher resolution videos with YouTube streaming. In 2021, 720p was about as good as it got (360p on T-Mobile) while in 2022 there was a high percentage use of 1080p and up to 1440p/2160p in some cases.

1.0 Executive Summary2.0 Key Observations

3.0 Network Performance Score - Details by Test Type

3.1 Voice14
3.2 Data
3.2.1 HTTP Transfer18
3.2.2 HTTP Browsing
3.2.3 Social Media
3.2.4 Video
3.3 NSA Versus SA - T-Mobile Only

4.0 Data Results and Analysis

4.1 HTTP File Transfers
4.2 HTTP Capacity Tests
4.3 HTTP Browsing Tests
4.4 Video Tests
4.5 Interactivity Tests37
4.5 NSA Versus SA Tests

5.0 Network Analysis6.0 Test Methodology7.0 Final Thoughts

Index of Figures & Tables

Figure 1. Dallas - Fort Worth Drive Route
Figure 2. Overall Network Performance Score
Figure 3. Voice Performance Score
Figure 4. Data Performance Score
Figure 5. Points - Call Setup Success Ratio
Figure 6. Points - Call Drop Ratio
Figure 7. Points - Average Call Setup Time
Figure 8. Points - Call Setup Time 10th Percentile
Figure 9. Points -Call Setup Time Excess Ratio
Figure 10. Points - Voice MOS
Figure 11. Points - Voice MOS 90th Percentile18
Figure 12. Points - Voice MOS Bad Sample Ratio
Figure 13. Points - Voice Call LTE Band19
Figure 14. Points - HTTP Uplink/Downlink Success Ratio
Figure 15. Points - HTTP Downlink Average Throughput
Figure 16. Points - HTTP Downlink Throughput - 10th Percentile
Figure 17. Points - HTTP Downlink Throughput - 90th Percentile
Figure 18. Points - Points - HTTP Uplink Average Throughput
Figure 19. Points - HTTP Uplink Throughput - 10th Percentile
Figure 20. Points - HTTP Uplink Throughput - 90th Percentile
Figure 21. Points - Browsing Success Ratio
Figure 22. Points - Browsing Duration Average
Figure 23. Points - Browsing Duration Excess Ratio
Figure 24. Points - Social Media Success Ratio
Figure 25. Points - Social Media Duration Average
Figure 27. Points - Video Success Ratio
Figure 26. Points - Social Media Duration Excess Ratio
Figure 28. Points - Average Video MOS
Figure 29. Points - Video MOS - 10th Percentile Ratio
Figure 30. Points - Average Video Setup
Figure 31. Points - Video Setup Excess Ratio
Figure 32. NPS Overall Data Results - NSA Versus SA
Figure 33. NPS HTTP Downlink Results - NSA Versus SA
Figure 34. NPS HTTP Uplink Results - NSA Versus SA
Figure 35. NPS Browser Results - NSA Versus SA
Figure 36. NPS Social Media Results - NSA Versus SA
Figure 37. NPS Video Results - NSA Versus SA
Figure 38. HTTP File Transfer Distribution by Technology - Downlink
Figure 39. HTTP File Transfer Distribution by Technology - Uplink
Figure 40. HTTP File Transfer Failure Ratio - Downlink
Figure 41. HTTP File Transfer Failure Ratio - Uplink
Figure 42. HTTP File Transfer Downlink Throughput by Technology
Figure 43. HTTP File Transfer Uplink Throughput by Technology
Figure 44. Downlink Throughput
Figure 45. Uplink Throughput
Figure 46. Distribution by Technology - Downlink
Figure 47. Distribution by Technology - Uplink
Figure 48. Downlink Throughput by Operator and by Technology
Figure 49. Browser Test Count by Technology
Figure 50. Average Browser Webpage Load Times by Operator
Figure 51. Kilobytes Downloaded In First Second
Figure 52. Average Browser Webpage Load Times by Operator and Webpage
Figure 53. Video Test Count by Operator and Technology
Figure 54. Video MOS by Operator
Figure 55. Video Resolution by Operator
Figure 56. Video MOS Versus Video Resolution - Verizon
Figure 57. Video Setup Time by Operator
Figure 58. Interactivity Test Count by Operator and Technology
Figure 59. Interactivity Average Score by Operator and Technology
Figure 60. Interactivity 3GPP QoS Target Application Reached
Figure 61. Interactivity Latency - constant long profile
Figure 62. Interactivity Packet Error Rate
Figure 63. Average Downlink and Uplink HTTP Transfer Data Rates - NSA versus SA
Figure 64. Average Downlink and Uplink HTTP Transfer Times - NSA versus SA
Figure 65. HTTP Capacity Downlink Data Rates - NSA versus SA
Figure 66. HTTP Capacity Uplink Data Rates - NSA versus SA
Figure 67. Average HTTP Web Browser Load Times - NSA versus SA
Figure 68. Average HTTP Web Browser Detailed Results - NSA versus SA
Figure 69. Interactivity Test Results I - NSA versus SA
Figure 70. Interactivity Test Results II - NSA versus SA
Figure 71. 5G NR and LTE Carrier Aggregation Utilization
Figure 72. Aggregate LTE Carrier Bandwidth
Figure 73. 5G NR Band Utilization by Operator
Figure 74. 5G NR Throughput by Band and by Operator
Figure 75. AT&T Band n5 Coverage
Figure 76. AT&T Band n5 Signal Quality
Figure 77. AT&T Band n77 Coverage
Figure 78. AT&T Band n77 Signal Quality
Figure 79. T-Mobile Band n71 Coverage
Figure 80. T-Mobile n71 Signal Quality
Figure 81. T-Mobile Band n41 Coverage
Figure 82. T-Mobile n41 Signal Quality
Figure 83. Verizon n2 Coverage52
Figure 84. Verizon n2 Signal Quality
Figure 85. Verizon n77 Coverage
Figure 86. Verizon n77 Signal Quality
Figure 87. Verizon n260 Coverage
Figure 88. Verizon n260 Signal Quality
Figure 89. Verizon n260 Coverage - Dallas Downtown
Figure 90. Verizon n260 Coverage - Dallas Residential
Figure 91. T-Mobile Band n41 Coverage - 2022
Figure 92. T-Mobile Band n41 Coverage - 2021
Figure 93. Drive Test Route
Figure 94. NPS Test Plan
Table 1. NPS Scoring Insights
Table 2. NPS Scoring and Weighting Insights - Voice
Table 3. NPS Scoring and Weighting Insights - Data
Table 4. Interactivity KPIs

Companies Mentioned

T-Mobile
AT&T
Verizon

Acknowledgements

Methodology

The Results are in

Chalk it up to 5G

NSA versus SA

Video Rules

Table of Contents

Companies Mentioned

Related Topics

Related Reports

5G NR Benchmark Study Vol 32: 5G mmWave 4CC Uplink Performance

5G NR Benchmark Study Vol 34: 5G Uplink Carrier Aggregation

5G New Radio Market - Global Industry Size, Share, Trends Opportunity, and Forecast 2018-2028

5G for the Connected World. Edition No. 1

5G New Radio Market Size, Share & Trends Analysis Report By Offering, By Operating Frequency, By Architecture, By Application, By Industry, By Region, And Segment Forecasts, 2023-2030