Benchmark mobile phone voice quality – Part 1
Part one: Introduction to the benchmark.
This article is the first part of a two-part series covering an independent benchmarking project to evaluate mobile phone voice quality. The project aims to verify if the voice quality performance of a range of phones meets the standards demanded by the four major operators in the U.S. The tests will provide a comparison between phones equipped with LifeVibes VoiceExperience and those using alternative solutions.
This first article gives a brief background to the project and describes the main tests that will be performed to determine if the acoustic performance meets the requirements of the operators. In the second article, we will present the results of the benchmarking tests.
Mobile voice quality standards set by the main operators in the U.S., including AT&T, T-Mobile and Verizon, are among the strictest in the world. They often go beyond the basic requirements defined by 3GPP and the ITU (International Telecommunications Union) and aim to ensure the highest audio quality, free of any echo or noise problems.
So it’s important for phone makers to ensure their devices meet these criteria, otherwise they will not be added to the operator’s portfolios. Furthermore, qualifying for these standards generally means their devices will be accepted globally. And for end-users the standards guarantee that talking on the phone is a pleasurable experience, not disrupted by effects such as echo, intrusive background noise or voice attenuation.
So, just how good is the sound quality of today’s mobile phones? To find out, we asked Spirent (formerly Metrico Wireless) to conduct independent benchmark testing on a selection of up-to-date smartphones on the market. A highly regarded testing house in the mobile phone industry, top mobile operators rely on Spirent’s testing services for device validation.
In the benchmarking tests, Spirent will carry out benchmark tests on two batches of five new phone models from different makers. A comprehensive set of tests will be performed, covering performance in both handset and speakerphone mode, and in both narrowband and wideband. The tests involve detailed measurement of:
- Acoustic Echo Cancellation (AEC)
- Noise Suppression (NS)
- Round-trip delay
- Send and receive Loudness (SLR and RLR)
- Send and receive Frequency Response
- Send and receive Idle Noise
- Send and receive Distortion
Benchmark set up
For NXP Software the tests that focus on acoustic echo cancellation (AEC) and noise suppression (NS) are most important. These are the key areas governing a phone’s ultimate sound quality, and where our VoiceExperience software can really make a difference.
Testing is made recording a live phone call on a Rhode & Schwarz CMU200 mobile network simulator. The network simulator provides a stable and controlled GSM/UMTS network and allows us to record the call. The phone is mounted to a head and torso simulator, or HATS, fitted with artificial ears and an artificial mouth. An ACQUA system from Head Acoustics is used to analyze the audio recordings. The measurements in this benchmark are carried out with the device in handset mode and hand-held hands-free mode.
Acoustic echo loss testing
In handset and hand-held hands-free modes, the measured terminal’s echo loss performance has to meet the conformance requirements of 3GPP (3rd Generation Partnership Project) TS 26.131.
For each configuration, the 3GPP requirements provide a minimum echo loss figure that ensures a guaranteed level of speech quality, as indicated in the table below. Any lower than this and the echo can be loud enough to have an adverse effect on voice clarity.
|Echo loss at max volume||46 dB (NB) / 55 dB (WB)|
|Echo loss at nominal volume||55 dB|
|HANDHELD SPEAKER MODE|
|Echo loss at max volume||40 dB|
|Echo loss at nominal volume||46 dB|
|Echo loss at any volume||40 dB|
In addition to the effectiveness of the echo cancellation, the quality of the remaining speech is also important. Two additional tests are performed to measure the clarity of the remaining voice signal. The first is a MOS measurement rated on a score of 1-5, with 1 being poor and 5 being excellent. The second is a measurement of the remaining speech signal level compared to the signal overload level, expressed in dBov.
Sending and receiving loudness ratings
The device’s sending and receiving loudness ratings are determined in handset, hand-held hands-free and headset modes. The measured results must conform to the requirements of 3GPP TS 26.131, ensuring optimal volume levels for communication. For example, in narrowband handset mode the nominal values are:
- SLR = 8 +/- 3 dB
- RLR = 2 +/- 3 dB
Sending and receiving frequency responses
In handset, hand-held hands-free and headset modes, the device’s frequency response in both sending and receiving conditions must conform to the requirements of 3GPP TS 26.131. This means each frequency response has to be within a certain sensitivity band (upper and lower limits) across the whole frequency range. Meeting this requirement ensures correct reproduction of voice patterns without any emphasis on a particular frequency.
When a handset is used in speaker mode in noisy environments, the use of 2 or more microphones is essential to achieve a good noise suppression (NS) performance. Spirent will therefore benchmark the noise suppression capabilities of the phones when applied to both transmitted speech, known as uplink noise suppression, and received speech, or downlink noise suppression. Of the two, uplink noise suppression is the most important for most operators.
MOS and SNRI
These tests require three objective measurements, speech mean opinion score (S-MOS), noise mean opinion score (N-MOS), and SNR improvement (SNRI). These measurements are obtained by simulated listening tests using a 3QUEST measurement.
SNRI is among the most important objective measures of the NS software’s effectiveness. It takes into account the background noise level compared to the speech volume. So an improvement in the signal-to-noise ratio shows that the NS solution works correctly – it is reducing noise without reducing the speech volume. A variety of test signals are used for the measurements, including clean speech from male and female speakers, and noise such as car noise and street noise. The improvement is calculated in decibels.
The device has to achieve a minimum standard for each measurement – S-MOS, N-MOS and SNRI – according to the operators’ guidelines. Furthermore multiple tests are made, in both narrowband and wideband, covering a wide range of noisy environments such as traffic, offices and cafes.
Example minimum MOS and SNRI requirements in narrowband
|Traffic crossroad||3.8||3.2||18 dB|
|Business office||3.8||3.2||18 dB|
Similar measurements will also be made for NS in speakerphone mode. Again, the SNRI is the most important. A minimum SNRI of 6 dB has to be delivered for all environments.
Awaiting the results
Performing the benchmarks takes some time, but results from the first batch of phones are being prepared now, with batch two results to follow shortly. Check out the second article of this series to see how the phones performed. This will be published in February 2014.