Apple Watch Series 12 adds Audio Intelligence features, prompting debate over nearby consent

Macworld The Apple Watch Series 12, announced at the Surprise and Shine event on Wednesday, doesn’t offer a great deal by way of momentous hardware changes. But Apple did manage to spring a few surprises with an array of software upgrades for the device, including a cool feature called Live Rewind. Live Rewind, which is accessed via a double-press of the Digital Crown, displays the last 15 seconds of your conversation in text form on the watch’s screen. This is a neat and potentially very useful feature for those moments when you miss something important and don’t want to, or cannot, ask the speaker to repeat themselves. But it does raise some privacy questions. After all, the fact that the feature works at all indicates that the Apple Watch is constantly eavesdropping on your conversations, likely without the knowledge or consent of the people you’re with. But Apple has rushed to offer reassurance. The Apple Watch Series 12, like so many current tech devices, may be constantly listening (otherwise “Hey Siri” wouldn’t be able to work), but users cannot directly access the audio of its eavesdropping secretly. The feature makes a distinct chiming sound when activated, Apple says, and this sound cannot be disabled. There’s also a screen animation. Even the physical method of activating the feature, the company claims, will help those around you to see what you’re up to. A new support document for the Audio Intelligence watch features contains two privacy-related sections: one headed “How Audio Intelligence features protect your privacy” and another headed “How Audio Intelligence features protect the privacy of those nearby.” The latter section contains the following comment about Live Rewind: “When [the feature is] activated, an audible chime plays from the speaker on your Apple Watch, even if your Apple Watch is on silent or you have headphones connected. A full-display animation and microphone indicator appear on the display, and a distinct double-press gesture is required to activate Live Rewind. This is to alert those nearby that the feature has been activated.” And even with these measures in place, Apple is keen to advise customers to use the feature responsibly. “Be mindful of those around you,” the support document goes on to say. “While audio is not recorded and stored, and no speakers are attributed, consider those around you where conversations might be private or sensitive.” On the whole this seems like a wise move. Apple is fond of portraying itself as the pro-privacy tech company, and suffered reputational damage when the AirTag became associated with stalking. There’s a deep public distrust of smart glasses because of their capacity to surveil and record without consent; audio surveillance doesn’t feel quite as intrusive, but it’s sensible to take control of the narrative and head off such accusations before they become too widespread.

Macworld With the Apple Watch Series 12 and Ultra 4, Apple is introducing some neat new audio-based AI features that seem genuinely useful and exciting. The new S11 processor includes hardware that makes it possible to continually listen and (lightly) process audio without ruining battery life, and Apple is using that to provide four new AI-powered features. It calls these features “Audio Intelligence” because Apple can’t help but brand everything. Full-time Shazam: The watch can detect music playing and automatically identify it using Shazam, displaying it on widget in the Smart Stack. Apple Watches already do this on demand, but the new thing here is that it does it continuously in the background like Pixel phones. Sound recognition: The same feature you can get on HomePods and iPhone now comes to Apple Watch. You can get notified visually when your watch hears sounds such as doorbells, fire alarms, sirens, etc. You used to need your iPhone nearby to use this feature, but the Series 12 and Ultra 4 can do it independently. Live Rewind: Double-tap the digital crown to see a text readout of the last 15 seconds of what was said around you. You can then ask Siri about it or save it to the Siri app. Siri Recap: Get a high-level summary of an entire conversation in the Siri app, so you don’t forget specifics or need to take notes. The first two of those features don’t really bother anyone. Those are existing features, and the only new thing is that the Apple Watch can do them in the background without iPhone intervention. The last two will be available later this year as a beta, and those are the ones that have people freaking out. Are people’s Apple Watches going to be recording you all the time? Apple has issued a Audio Intelligence Security Overview white paper to help users understand how all this works and how it protects everyone’s privacy. It’s worth a read, but here are the highlights. How Live Rewind works On the Apple Watch Series 12 and Ultra 4, audio from the microphone is continuously flowing into a little buffer in the Secure Enclave. This actually happens on lots of Apple products; it’s how they know if you’ve said “Hey Siri” or not. This is a rolling buffer, where new data continuously replaces old data. The Secure Enclave is a completely separate hardware block that is not accessible to the rest of the operating system, or any apps, or even Apple. It’s where your Face ID and Touch ID data is stored, for example. Live Rewind will show you what you missed—privately.Apple When you double-press the Digital Crown, the last 15 seconds of that buffer is sent via an encrypted secure transfer to your iPhone, where an AI model transcribes it. This happens in the iPhone’s Secure Enclave, so the audio cannot be accessed by the rest of iOS, or any app, or the user, or even Apple. This is just a straight voice-to-text transcription; it doesn’t identify or label people who are speaking. The audio is then immediately discarded. The text of the translation is then sent back to Apple Watch where it appears on the display. It is not saved unless you ask Siri about it or save it to the Siri app to revisit later. The Siri app’s data is end-to-end encrypted and synced between devices, and users can control whether conversations are automatically deleted after 30 days, 1 year, or saved forever. Apple cannot decrypt this data, even with a court order—only the user has the key. And one more thing, an audible chime plays from the speaker on your Apple Watch when you implement Live Rewind, even if your Apple Watch is on silent or you have headphones connected. So that’s how Live Rewind protects your privacy and those around you. Raw audio data can’t ever be accessed by the user, the system, Apple, or any apps. It’s just straight translated to text, which is then optionally saved in the end-to-end encrypted Siri app. How Siri Recap works Siri Recap works in a similar way, but there’s more going on. You’ll be able to leave it on all the time, only when you trigger it manually, or specify certain places or times when it can’t operate. First, a tiny little lightweight model simply detects if a conversation is happening at all. If it is, that encrypted and secure audio buffer is sent to the iPhone (via an encrypted data transfer) for processing. It is transcribed to condensed text, and the raw audio is then immediately discarded. It is never accessible to the OS, apps, the user, or Apple. This transcribed text passes through a safety filter than will remove potentially harmful terms. Then it is encrypted and sent to Apple’s Private Cloud Compute servers to be summarized, along with relevent contextual data that can make the summary better (such as what music you might be listening to, a relevant calendar entry, or high-level place information like what city you’re in). Apple Private Cloud Compute generates a title and summary with key points. It removes any specific identifiable or sensitive info such as place names or financial data, while leaving generic places names like “park” or “grocery store.” That summary is then sent back to your devices, encrypted, where it is stored in the Siri app in a new Recaps tab. All data on Apple’s Private Cloud Compute servers is always discarded immediately after processing; Apple never saves any of it. The summaries in the Recaps tab are deleted automatically after seven days unless you choose to save it. Like Live Rewind, Siri Recap does not create a recording, does not identify speakers, and does not even create a verbatim transcript. No audio is ever accessible at any step of the process by the user, any app, or Apple. Finally, Live Rewind and Siri Recap are not available for users under the age of 13. For users under 18 on a Child Account, access to Live Rewind and Siri Recap must be turned on by a parent or guardian.

Apple's new Audio Intelligence features for the Apple Watch Series 12 are drawing privacy concerns because they can process nearby conversations without explicit consent from everyone involved. "Live Rewind lets you instantly see the last 15 seconds of a conversation as text," Apple explains in its technical summary (PDF). "Siri Recap summarizes conversations throughout your day and produces high-level Apple Intelligence-generated notes so you can stay present in the moment and catch up later." The Register reports: The latest Apple Watch comes with Audio Intelligence, a set of AI audio processing capabilities tuned for the company's S11 chip. Its features include: Sound Recognition, Music Recognition with Shazam, Live Rewind, and Siri Recap. [...] With the double-press of the Digital Crown -- as Apple grandly refers to the button on its Watch -- Live Rewind takes in an audio stream from the Watch microphone, processes it in a Secure Exclave on the S11 chip, and routes the data to the user's nearby iPhone, which runs a speech-to-text algorithm on the 15-second audio segment. The resulting text is saved and the audio is discarded. The wearer's Watch emits an audible tone, even in silent mode, to alert those in the vicinity and provides a visual cue for those able to see the face of the device. Nonetheless, bystanders alerted to the recording -- to the extent they recognize the meaning of the tone -- have not consented to being recorded, which is a legal requirement in 11 US states that have all-party consent laws. Apple characterizes its implementation of brief eavesdropping as respectful of personal privacy. It makes that claim in a section titled, "How Live Rewind respects those around you," citing the audible chime and visual on-screen animation. Siri Recap, meanwhile, "summarizes conversations throughout your day and produces high-level Apple Intelligence-generated notes so you can stay present in the moment and catch up later." This too, Apple describes as an act of respect. "By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers," Apple's technical documentation explains. "The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone." Read more of this story at Slashdot.