Microsoft Patents AI System That Can Alter Words and Faces in Live Video in Real Time

Microsoft's AI System Promises Real-Time Video Manipulation

Microsoft has filed a patent application for an AI-powered technology that could fundamentally change how audiences experience live content. First flagged by patent-tracker Patentlyze, the Microsoft patent describes a "near real-time artificial intelligence enhancement of video or augmented reality presentations of live events."1

1

2

2

The AI system would use specialized AI agents to identify, alter, or remove specific words while simultaneously editing a speaker's face, lip movements, and facial expressions to match the modified speech.

The technology would operate based on preset parameters or individual user profiles, allowing viewers to customize what they hear during live broadcasts. According to the patent filing, the system would also edit video and audio segments immediately before and after any removed or replaced word to ensure smooth transitions.1

1

This capability to modify spoken words in live broadcasts represents a significant leap beyond traditional video editing, bringing AI-driven content manipulation into the realm of live events.

How the Real-Time Translation and Error Correction Would Work

One of the primary applications outlined in the patent involves real-time translation. The AI system would enable viewers to watch live events and hear translated versions in their preferred language, with the speaker's mouth movements adjusted accordingly to create a seamless viewing experience.2

2

This could make international broadcasts and augmented reality presentations more accessible to global audiences.

Beyond translation, the technology could remove unwanted words like filler sounds including "um" and "ah," and fix speech errors such as stutters or speech impediments.1

1

2

2

The patent suggests the AI system could eliminate "mumbled words, mispronounced words, misused words, misspoken words, offensive words, insensitive words, or inappropriate words." Replacement words may even come from internet searches, according to the filing.1

1

The technology could also target "personal words or phrases" like coined terms, as well as colloquial terms such as "y'all" or phrases like "bite the bullet."1

1

Potential applications span various types of live content, including lectures, presentations, television productions like awards ceremonies, game shows, cooking shows, and live sports broadcasts.

Ethical Concerns About Authenticity and Manipulation

The near real-time artificial intelligence enhancement technology raises significant questions about authenticity in an era already grappling with AI-fueled deepfakes and disinformation. The system described in the Microsoft patent may usher in a new world of filtering tools that subtly alter reality as it unfolds, creating highly personalized versions of live events.1

1

If viewers receive different versions of the same speech, they may not always know what the speaker actually said.2

2

This capability to alter live video and audio could fundamentally change how audiences engage with live footage. Rather than simply turning off content they dislike, viewers could instead adjust someone else's speech to their personal preferences. Language translation aside, using a tool that eliminates errors or erases words and phrases associated with a speaker's local dialect raises questions about whether this pushes society further into personalized bubbles.1

1

The patent does acknowledge potential misuse by mentioning that an event's original live video and audio may be recorded and stored for the sake of "ground truth."1

1

2

2

This suggests Microsoft recognizes the confusion and ethical concerns that could arise from such technology. However, questions remain about whether preserving original recordings is sufficient to address the authenticity challenges this AI system introduces.

What to Watch For

As this technology moves from patent to potential product, watch for how Microsoft addresses concerns about manipulation and whether the company implements safeguards beyond simply storing ground truth recordings. The ability to remove regional expressions and accents based on viewer preferences could have cultural implications worth monitoring.2

2

Microsoft did not immediately respond to requests for comment about the patent filing.1

1