Context: How NXP Software makes sense of smartphone and smartwatch sensors
You’re in business trip, you had a long intercontinental flight followed by a full day of meetings with eventually a lengthy dinner!
At 23:30pm local time you were exhausted and went to sleep.
At 3:24am the hotel alarm emergency rings. You don’t even remember where you are on earth and you have to exit immediately.
But your smartphone was always on listening, recognized the alarm bell and has automatically switch to emergency mode where it is showing you the indoor navigation system. Just follow the instructions and you’ll be safe in 1 minute and 3 seconds.
Contextual sensing is what turns dumb devices into truly smart personal assistants. “We call the current devices smartphones and smartwatches,” says Alexandre Henon, Product Management Director at NXP Software, “but until those devices can infer our needs based on the context we’re in, they can become truly smart”
Henon and a team of software engineers at NXP Software are working on the algorithms that will enable smart devices to automatically sense and adapt to their surroundings. “If you’re in a theatre you want the phone to behave differently than if it’s in a car,” Henon says. “Nobody wants to be that person whose phone rings during a play … and when you’re in the car, you want bluetooth or loud speaker mode to kick in automatically so you don’t miss a call.”
But doesn’t GPS let your phone know you’re in a car? “Not really,” Henon explains. “The GPS knows where geographically you are and that you’re moving, but you could be in a bus or on a tram. By overlaying data from the audio channel, we make your phone’s data more accurate and relevant to your context. Our job is to deliver the accurate data that will enable the next generation of exciting context-aware apps.”
You’ll be surprised how many phone sensors there are
The NXP Software team have identified over 15 different types of sensor that could be used in smart device applications. Accelerometers, barometers, gyroscopes and optical heartbeat monitors are well known. And just about all smartphones today have proximity and light sensors. But what about detecting carbon monoxide or propane gas, or monitoring your oxygen levels when taking exercise? They’re all possible. NXP Software customer Samsung already has temperature and humidity sensors in their Galaxy S5 handset … and the list of sensors goes on.
In an article by TechRadar (link below) titled ‘Making sense of sensors: what you don’t know your phone knows about you,’ Emilio Miluzzo, a technical specialist at AT&T is quoted as saying: “The trend of adding more sensors to mobile devices will continue. It would be great to have air and water quality sensors, some forms of medical sensing, 3D/stereo cameras, even radar and sonar… the wishlist could certainly grow if we could have an understanding of how quickly sensor miniaturisation will proceed.”
The same TechRadar article refers to work by the US Environmental Agency (link below) and The University of California to create air quality sensors that are small enough and low power enough for mobile devices.
There’s even a bluetooth key ring called the Sensordrone that contains 11 different sensors (link below).
“What all these sensors mean,” Henon explains, “is that your device has a huge amount of raw data about where the device is. Now our ability to add contextual information about the ambient soundscape can make all of that data even more granular. For example, mix high carbon monoxide levels with the user gasping for breath – or not breathing at all – and an emergency call could be initiated.”
Algorithms and integration for contextual sensing
Henon explains how NXP Software started to pioneer contextual sensing. “As a company we’ve made our reputation in improving the audio quality for feature phones and smartphones,” Henon says. “Part of that ability comes from sensing the ambient noise level to adjust a phone’s microphone and speaker accordingly, so people can still have clear calls on noisy streets. But our audio specialists soon realised that the soundscape information they were collecting was a lot more granular. Like being able to distinguish between the different audio signatures for being in, say, a lift, meeting room, subway or theatre. Technology could even identify if the user is stressed or relaxed from their speech pattern.”
Today NXP Software has its voice clarity algorithms installed in over 3 billion smartphones and smartwatches, along with ITU-compliant audio labs and technical teams to support device makers locally in China, India, South Korea, Europe and the US.
“Those 3 billion smart device installations have taught us some other lessons,” Henon adds. “Like how to reduce power consumption and how to integrate our algorithms onto the software stack of a wide range of devices and chipsets.” Henon says that with the correct power management platform, the impact to battery life should not be noticeable.
So the next time you cough, your smartphone could discreetly start monitoring your temperature to see if you’re getting flu … or at least direct you to the nearest water fountain.
The image provided with this post is not an actual Android smartphone screenshot. It is used to show a possible use-case with NXP Software’s speech and sensing software.
Reference. Sources referred to in this article:
TechRadar. Making sense of sensors: what you don’t know your phone knows about you.
US Environmental Protection Agency. Next Generation Air Measuring Research.
Sensordrone key ring sensor.