Benchmark mobile phone voice quality – Part 2
Part two: Voice effects software benchmark.
This article is the second part of a two-part series covering an independent benchmarking project to evaluate mobile phone voice quality. The project aimed to verify if the voice quality performance of a range of phones were able to meet the standards demanded by the four major operators in the U.S. This article will illustrate the test results and provide comparisons between phones equipped with LifeVibes VoiceExperience and those using alternative solutions.
Scope of the tests
In total eight smartphones were tested, seven by Spirent (formerly known as Metrico Wireless) and one by NXP Software (see table). The devices tested were:
|Samsung ATIV S Neo for Sprint*
LifeVibes VoiceExperience 3.1
|Samsung SHV-E300S Galaxy S4 for SK Telecom March 2013
|Samsung Galaxy S4 for AT&T
|Samsung ATIV S Neo for AT&T
LifeVibes VoiceExperience 3.1
|Samsung Galaxy Note 3 N9005
September 2013 (EU)
LifeVibes VoiceExperience 4.0
|Samsung Galaxy Note 3 SM-N900
September 2013 (India)
|Apple iPhone 5s
The tests were carried out on the devices in handheld and speakerphone modes. Where possible, the tests also were performed in both narrowband (NB), used for conventional phone calls, and wideband (WB), used for bandwidth intensive calls such as HD. Certain phones like the Samsung Galaxy S4 for AT&T were only tested in NB mode as, at the time of writing, AT&T did not support wideband calls over circuit-switched networks. In addition, the Samsung Galaxy Note 3 N9005 (EU) was only tested in WB (VoIP) mode as we wanted to perform a comparison between CS and VoIP calls.
The following graphs show the Noise Suppression (NS) and Acoustic Echo Cancellation (AEC) results for the smartphones. The closer a particular handset is to top right-hand corner of any given graph, the better the result; delivering clearer calls, with non-intrusive background noise or residual voice feedback.
One of the major challenges when tuning a phone is to achieve a good balance between performance and speech quality (Graph 1). Tuning too aggressively for performance can detract from speech quality, while tuning for speech quality can result in a poor NS or AEC performance.
Graph 1. Balancing performance with speech quality
Acoustic echo cancellation (AEC), handset mode, narrowband
Acoustic echo cancellation prevents other people on the call from hearing their own voice back as an echo by cancelling out sound picked up by the phone’s microphones from its speakers. The graph below, (Graph 2) shows the AEC performance for NB calls in handset mode. Of the eight devices tested, the two highest performers were handsets running LifeVibes VoiceExperience software. The Samsung ATIV S for Sprint (GSM) and Samsung ATIV S Neo for AT&T both recorded 66 dB for echo loss at maximum volume, significantly above the competitor average at around 60 dB, and the 55 dB minimum set by the 3GPP (3rd Generation Partnership Project ) standard. While most devices performed well above the 55 dB minimum value, this is because the minimum value may not be enough to adequately suppress all the echoes generated in a conversation.
Graph 2. AEC performance for handset mode NB devices
The graph also shows that these devices are in the upper tier of speech quality (MOS). On a score of 1-5, with 1 being poor and 5 being excellent, both handsets rank at 4.3 MOS compared to the 3.9 MOS competitor average. This is also apparent with the active speech level (ASL), with both devices displaying lower levels of disruption to speech clarity at -29.5 to -28.5 dB compared to -31.3 competitor average.
Acoustic echo cancellation (AEC), handset mode, wideband
The next AEC graph is for devices in wideband handset mode. It illustrates that the Samsung Galaxy Note 3 N9005 EU is clearly the best overall performer, recording above 66 dB and 3.5 MOS for speech quality versus echo loss. This is almost 4 dB and 2 MOS above the weakest competitor model. Note that the Galaxy Note 3 N9005 (EU) was the only device tested in a VoIP call environment and not in a circuit-switched network. However, this would not have had any impact on the results as all phones were tested under the same conditions.
Graph 3. AEC performance for handset mode WB devices
In terms of ASL, the competitors range from -34.8 to -30.2 dB with the echo loss at maximum volume hovering at the 63 dB mark. The Galaxy Note 3 is again above 66 dB at max volume and close to -31.3 for ASL. In comparison, the Samsung Galaxy Note 3 SM-N900 records a consistent 62 dB on both graphs concerning echo loss at maximum volume. However, while it had the lowest MOS score of any of the handsets tested, it recorded the best ASL. This shows that while operators may use similar handsets, they can be tuned differently to suit regional requirements.
Acoustic echo cancellation (AEC), speakerphone mode, narrowband and wideband
The following bar graphs (Graph 4 and 5) focus on AEC performance for devices on NB and WB handheld speaker mode. The higher the echo loss reading, the greater the echo suppression. The 3GPP TS 26.131 minimum requirement is 40 dB.
The Samsung Galaxy Note 3 N9005 EU and ATIV S Neo for AT&T were not included in these two tests because it would be difficult to make a fair comparison. These devices feature a 1-mic speakerphone solution only while all the other devices have a 2-mic solution.
Graph 4. AEC performance for handheld mode WB devices
On the NB graph, two handsets miss the minimum requirement, the Xiaomi MI2a and Apple’s iPhone 5s. The other phones are at least 20 dB above the standard. For the WB test, all of the smartphones (apart from the iPhone 5s) exceeded the 3GPP TS 26.131 requirement. The Samsung Galaxy Note 3 N9005 EU with LifeVibes has the highest echo suppression, 26.5 dB above the standard and 3.8 dB past its closest rival.
Graph 5. AEC performance for handheld mode NB devices
Noise suppression (NS), handset mode, narrowband and wideband
Noise suppression eliminates or greatly reduces unwanted background sounds that may disrupt a mobile phone conversation. The graph for smartphones on NB, handset mode (Graph 6), shows that the Samsung Galaxy S4 and Samsung’s ATIV S Neo are the best in class for 2-microphone noise suppression performance.
Graph 6. NS performance for handset mode NB devices
These two phones have the highest SNRi scores, above 28 dB, illustrating that they are significantly above the 24 dB competitor average. And both phones have high Speech-MOS and Noise-MOS readings at around 4.2 SMOS and 4.2/4.3 N-MOS respectively. Both phones are available from AT&T which could be One of the reasons why they measure so highly may be because both phones are from AT&T. AT&T has stricter requirements than 3GPP, so its phones are more finely tuned.
An interesting comparison can also be made between the two phones with LifeVibes software. While the Samsung ATIV S Neo for Sprint is well above the competitor average and above at least four of the phones tested, it does not quite reach the heights of the AT&T version. The reason for this is tuning. AT&T’s version was tuned by the acoustic experts at NXP Software rather than by the OEM, resulting in significantly better performance even though they share identical hardware and software.
Looking at the WB handset mode graph (Graph 7), the Samsung Galaxy Note 3 N9005 EU and Samsung ATIV S Neo for Sprint (GSM) with LifeVibes, are the two phones with the best overall performance. While both just fall short of the S-MOS rating of 4 (good) compared to the competitor average at 4.2, they exceed SNRi score with 27 and 25 dB, more than 3 dB above the average. The only phone with readings close to matching their results is the Samsung SHV-E300S, while the other phones are noticeably inferior.
Graph 7. NS performance handset mode for WB devices
The same results can be seen in N-MOS readings. The three Samsung phones (mentioned earlier) are the highest tiered phones positioned around the N-MOS competitor average, at 4.2 but around 3-5 dB better than the 22 dB SNRi average. The Indian model of the Samsung Note 3 SM-N900 and the iPhone 5s are the phones in the middle tier. The Xiaomi MI2a is consistently below average on the SNRi reading at just above 18 dB, but performs slightly better in the S-MOS and N-MOS readings, tallying above 4 in both cases.
Noise Suppression (NS), speaker mode, narrowband and wideband
The data for NB and WB handheld speaker mode are still pending for LifeVibes VoiceExperience. Nevertheless, the two graphs (Graph 8 and 9) show interesting results. In the NB graph, the Samsung SHV-E300S has the best rating for SNRi in both S-MOS and N-MOS. It’s clearly ahead of the competitor average and its nearest rivals, the other two Samsung handsets. With a SNRi reading of 27 dB on both graphs and around 3.5 on S-MOS and 3.7 on N-MOS, this device stands on its own. Three of the smartphones, the two Samsung Galaxy handsets and the iPhone 5s, are clustered together on both graphs, influencing the competitor average which is closely placed to those devices. The Xiaomi’s MI2a handset is well below the competitor average and the other phones tested.
Graph 8. NS performance for handheld speaker mode NB devices
The WB graph is similar to the NB results, although the Samsung Galaxy Note 3 SM-N900 device is slightly closer to the Samsung SHV-E300’s NS performance. The two phones have excellent SNRi readings on both graphs, at least 4 dB points above the competitor average and between 3.1 – 3.5 on the S-MOS and N-MOS axes. Again the iPhone 5s is positioned around the average mark and Xiaomi MI2a falls well short of this reading.
Graph 9. NS performance for handheld speaker mode WB devices
From the AEC tests, it can be seen that LifeVibes outperforms competitor solutions, excelling in terms of echo loss, MOS and ASL for both narrowband and wideband modes. However, it can also be seen that operators may sometimes choose to ignore the minimum specifications. The iPhone 5s is a prime example, as indicated in the two bar graphs. But would an operator really refuse to include the latest iPhone in its portfolio because it didn’t meet the minimum requirements?
For the NS tests in handset mode, the two dominant phones are the Samsung ATIV S Neo and the Galaxy S4, both operated by AT&T. While AT&T’s Neo – utilizing LifeVibes VoiceExperience 3.1 – was on-par with the Galaxy, it out-performed all other solutions tested. The Samsung ATIV S Neo for Sprint (GSM) also surpassed most of the other rival software solutions. And in WB mode, the Samsung Galaxy Note 3, running VoiceExperience 4.0, provided the best NS performance.
LifeVibes VoiceExperience 4.0 data is still pending for speakerphone mode. However from initial results, the three Samsung handsets are clearly the best performers. In particular, the Samsung SHV-E300S and Galaxy Note 3 SM-N900 models provide the highest SNRi and N-MOS results.
In conclusion, smartphones that incorporate LifeVibes VoiceExperience software show very positive results. They are typically best-in-class, and perform at least on-par with devices using other solutions.