Voice Effects in Android ICS
As from Android release 4.0.x (Ice Cream Sandwich or ICS), Android comes with an extension of the audio effect framework. This extension allows device manufacturers to add voice signal enhancements in the microphone recording path. The new mechanism is an extension of an existing mechanism designed jointly by NXP Software and Google for Android release 2.3.x (Gingerbread). The original audio effect framework enabled a plugin mechanism for audio effects on the audio playback path.
More information can be found here: http://developer.android.com/reference/android/media/audiofx/package-summary.html
For ICS, NXP Software again collaborated with Google to extend the existing audio effect framework to enable a similar plugin mechanism in the audio capture path. The new mechanism allows voice signal pre-processing effects to be implemented in the Hardware Abstraction Layer (HAL).
The AudioEffects framework for the recording path allows Android Application developers to improve the user experience in several ways. For example, applying the Noise Suppression Audio Effect to the microphone signal gives more accurate results for speech recognition services. Or VoiceMemo or Camcorder applications can remove the “new email” alert sound, that happened to be played while recording, from the recorded audio signal using AEC AudioEffect. Or finally, the camcorder app can automatically adjust the audio level: no clipped or too soft audio anymore.
The fact that Google standardized the API of the native AudioEffects in Audio_Effect.h also means that device manufacturers can optimize the effects provided on their devices. They can include their own Audio Effects or Audio Effects provided by third party Software Vendors. In this way they can differentiate from the standard Android audio quality (or the quality of competitor’s phones) to bring more and better audio effects, making the end-user experience even more enjoyable. This might be more required than you would assume according to reviews of stock Android devices (e.g. check the section on poor microphone quality on the Nexus 10 review by ZDNet).
Architecture of Audio Effects framework in Android ICS
Fig. 1 below shows the major components relevant for the Android AudioEffect framework for ICS. Android applications can playback and record audio using the AudioTrack and AudioRecord components, respectively. Audio signals (in PCM format) are transferred through those components to the hardware microphone and speaker. The transfer takes place across quite a few layers, with the most relevant for the AudioEffect framework depicted in the diagram.
All audio streams on Android are routed through AudioFlinger which resamples (if required) and down-mixes each stream. AudioFlinger then routes the PCM signal to the LibAudio (or Hardware Abstraction Layer for Audio provided by the platform or device). Both AudioFlinger and LibAudio are implemented natively in the Android User Space. In addition to resampling and mixing, AudioFlinger also allows AudioEffects to be attached to each playback stream. AudioEffects are requested either by the Android Application using the AudioEffect interface or automatically via the AudioPolicyService.
Android provides some default AudioEffects but device manufacturers can add their own or even third party audio effects. Android applications can not add AudioEffects to the system since they need to be added to the native system libraries.
Before ICS, audio effects could only be applied to playback streams. Starting with Android ICS, AudioEffects can now also be applied to record streams. In contrast to playback streams, the application of the AudioEffect in the recording stream is done in LibAudio (See Fig. 1). So LibAudio is responsible for creating and applying the audio effects depending on the various use-cases. LibAudio also implements the echo reference used to provide a synchronized reference from the playback audio path to the capture audio path. This echo reference (or loopback stream) is mainly used for AudioEffects that implement Acoustic Echo Cancellation.
Another difference between playback Audio Effects and record Audio Effects in that playback Audio Effects are controlled by the application. This means that applications need to explicitly request audio effects to be applied to their streams. For record Audio Effects, the activation or deactivation of the effects is determined by the AudioPolicyManager depending on the Android use-case.
The audio effects defined by default in Android release 4.0.x for voice signal pre-processing are:
Limitations of Audio Effects framework in Android ICS
For Android release 4.0.x, the AudioEffects framework requires that all effects are implemented as a single library. This means that each library needs to support all three default pro-processing effects. Note that for playback, several libraries can be supported.
Because the AudioEffects in the Record stream are performed inside the LibAudio or HAL, the Android Applications are dependent on device specific integrations. This means that all device manufacturers and platform providers must comply with the reference HAL released by Google.
Neither the AudioEffects API, nor the reference implementation provided by Google prevent the system from applying double effects. It is possible that the lower layers of the hardware platform already implement AudioEffects like AEC or AGC. On top of this, applications can also apply AudioEffects to the audio streams (e.g. the audio effects performed by Skype to remove echo). Unlike playback effects in Android ICS, applications have no control on the record AudioEffects, so cannot query if they are present or request them to be enabled or disabled.
In one of the next posts, we will provide more details on the limitions of the ICS framework and investigate how they can be resolved.