The problem

In sign languages, the face is grammar. An AI codec that redraws the face can change what a sentence means.

A new kind of video call is arriving. Instead of sending the picture, your phone sends a short description of how your face and head are moving, and the other end redraws you from a single photo. It saves an enormous amount of data, and it is already going into real products and into a video standard.

All of it is built and tested around hearing people talking. For Deaf signers the stakes are different. The hands carry the words, and hands are exactly where these systems are known to fail. The face carries the grammar: raised eyebrows make it a yes/no question, lowered brows make it a "who, what, where" question, a head shake makes it a negative. If the system smooths away an eyebrow raise, it has not made the video look worse. It has made the sentence mean something else.

The failure mode

Sender

brows raised

“Are you coming?”

Receiver

brows flattened by the codec

“You're coming.”

Illustration of the failure mode under study, not a measured result. In ASL, raised eyebrows are the grammatical marker of a yes/no question. Same person, same face: the codec keeps the head and loses the brows, and with them the question.

Why now

Generative and avatar-based video has moved from papers into products such as NVIDIA's AI face codec and Apple Vision Pro Personas, and generative face video coding is being standardised. Meanwhile a 2025 field study of Vision Pro with Deaf and hard-of-hearing users found that Personas lose signs made near the face [5].

A survey of more than forty generative face-coding methods names expression and emotion fidelity as an open problem [4]. I found no published work that tests these codecs on sign language, or measures whether grammatical facial markers survive them.

What I want to know

  1. How much harder is signing to understand after these systems redraw it, and which signs suffer most?
  2. How often does the meaning actually flip, because the face lost or invented a question, a negative, or a mouth pattern?
  3. Can a little extra information, sent alongside, stop those flips without using more data?

How I plan to do it

1. Benchmark. Run open generative codecs over public sign-language datasets (How2Sign and WLASL for ASL, with PHOENIX-2014T, BOBSL and iSign for German, British and Indian Sign Language). Compare sign recognition and translation on original versus regenerated video, and add a dedicated detector for grammatical facial markers so meaning flips can be counted, not just quality loss.

2. A sign-aware codec. Carry the few signals that carry grammar (brows, head, mouth, hand keypoints) in a tiny side channel, or train with a loss that penalises losing them, and let the receiver check each frame against them.

3. With Deaf signers. Test comprehension over realistic low-bandwidth connections, designed together with Deaf collaborators.

How I want to work

This research is about how Deaf people experience technology, so Deaf signers' judgement is the measurement. Deaf consultants are paid for their time, and anyone who shapes the work is credited as a co-author.

If you're a Deaf signer, an interpreter or a sign-language researcher and this interests you, I'd like to hear from you: amoghbajpai8@gmail.com.

Reading

  1. VC4ASL. Enabling ASL Communication Under Low Data Rates. 2025. arXiv:2510.23056: pose-based sign transmission for calls.
  2. Moryossef et al. Pose-Based Sign Language Appearance Transfer. 2024. arXiv:2410.13675: recognition loss on re-rendered signers.
  3. Vahdati et al. Unmasking Puppeteers. NeurIPS 2025. arXiv:2510.03548: identity verification in talking-head codecs.
  4. Chen et al. Generative Models at the Frontier of Compression: A Survey on Generative Face Video Coding. 2025. arXiv:2506.07369
  5. In-the-Wild Accessibility Evaluation of Apple Vision Pro for DHH Users. VRST 2025. doi:10.1145/3756884.3765997
  6. Towards AI-driven Sign Language Generation with Non-manual Markers. CHI 2025. arXiv:2502.05661