NZSL Live
There's rarely an NZSL interpreter on hand for spontaneous moments — the pharmacy, a lecture, an unplanned video call — and most accessibility tooling is built around English alone. NZSL Live is a web app and browser extension bringing live captions, real hand-tracking, and an NZSL avatar into everyday conversations, with te reo Māori words recognised alongside English.
Key Features
- Live speech-to-text captions on a phone camera view
- Real 21-point hand-tracking
- NZSL avatar for common signed words
- Te reo Māori word recognition (whānau, aroha)
- Type-to-speak reply for two-way conversation
- Browser extension captions Zoom/Meet/Teams calls
Technologies
- Next.jsReactComputer VisionBrowser Extension
Challenges
Build something honest about a genuinely unsolved problem — automated sign-language recognition — without overclaiming or building something toothless.
Solution
A working proof of concept combining live captioning, real hand-tracking, and an NZSL avatar for common words, with anything not genuinely solved labelled 'coming soon' rather than faked for the demo.
Results
A working proof of concept demoed at the AI Hackathon Festival alongside SafeImage NZ, with a clear next step toward licensed NZSL content and Deaf-led review rather than pretending the hardest problem — sign recognition — is solved.
Gallery