How are Google and the University of Waterloo transforming education? AI prototypes that teach sign language and more

MarGib August 05, 2026

🌐 🇵🇱 Polski · 🇬🇧 EN

AI prototypes emerging from the collaboration between Google and the University of Waterloo have an ambitious goal: to make education more accessible and interactive. Will tools like "Sign Language Tutor" or "AI Classroom Assistant" truly change the way we learn – and what are the technologies behind them?

Interakcja uczniów i nauczycieli z holograficznym interfejsem AI uczącym języka migowego w klasie — Illustration: How can AI support sign language learning in education?

AI prototypes from Waterloo: What has been created so far?

In May 2026, during the Google I/O conference, prototypes developed as part of the collaboration between Google and the University of Waterloo were unveiled. These projects, developed under the "Futures Lab" banner, focus on practical applications of artificial intelligence in education and accessibility. The most advanced among them are:

"Sign Language Tutor" – a tool for learning American Sign Language (ASL) that analyzes user gestures via camera and provides real-time feedback.
"AI Classroom Assistant" – a system that automatically generates quizzes and lesson summaries based on video recordings.
"Accessibility Translator" – translating text into simplified language or symbols, aimed at people with cognitive difficulties.

Although most of these solutions remain in the pilot phase, their potential is immense. "Sign Language Tutor" has already been tested by a group of 50 students from Waterloo, 82% of whom rated it as "helpful" or "very helpful" in learning ASL. Meanwhile, "AI Classroom Assistant" has the potential to automate part of a teacher's workload by generating teaching materials based on recorded lessons.

Who are these tools for?

The target groups for these prototypes are diverse, but they all share one thing: the need for more inclusive and flexible education.

Students – especially those learning sign language or needing additional learning materials.
Teachers – tools like "AI Classroom Assistant" can relieve them of routine tasks, such as creating quizzes.
People with disabilities – "Accessibility Translator" is intended to facilitate access to content for individuals with cognitive difficulties.
Non-profit organizations – collaborating with Google on testing solutions, e.g., Canadian Hearing Services.

It is worth remembering, however, that most of these tools are not yet publicly available. "Sign Language Tutor" can be tested in beta on the Toca Carousel website, but access is limited to selected users. There are also no public code repositories, which makes independent verification of the technology difficult.

Technologies behind the prototypes

The Waterloo projects are based on the latest Google solutions, including the Gemini Omni 3.5 model, which was presented during Google I/O 2026. It is a multimodal AI model capable of analyzing text, images, video, and audio – making it an ideal tool for gesture recognition in "Sign Language Tutor".

Key technologies:

Gemini Omni 3.5 – real-time video analysis, generation of visual and text responses.
Google Beam – a framework for data stream processing, used in "AI Classroom Assistant" to generate quizzes.
tensorflow – a machine learning library used in, among other things, computer vision algorithms.
mediapipe (likely) – Google's library for motion tracking, although there is no official confirmation of its use.

The prototypes run on both mobile devices (smartphones, tablets) and in the Google Cloud. Hardware requirements are not excessive – all that is needed is a device with a camera, at least 8 GB of RAM, and a processor with AI acceleration, such as Tensor G3.

It is worth noting that Google has released some technical documentation, including a publication on arxiv describing the Gemini Omni 3.5 architecture. However, details regarding the implementation of the Waterloo prototypes remain unclear, making it difficult to assess their true capabilities.

Test results: What do users and creators say?

The "Sign Language Tutor" pilot conducted in March and April 2026 yielded mixed results. On one hand, 82% of test participants rated the tool positively, highlighting its usefulness in learning ASL. On the other hand, there were issues with latency in gesture recognition (averaging 1.2 seconds) and difficulties in distinguishing between similar gestures.

"The prototype shows the potential of AI in personalizing education, but it still requires work on the fluidity of interaction."

— Prof. Sarah Johnson, University of Waterloo, interview from May 15, 2026

During the Google I/O 2026 Keynote, Sundar Pichai emphasized that projects like "Sign Language Tutor" aim to "build bridges to a more inclusive world." However, beyond enthusiastic statements, there is a lack of detailed data on tests of other prototypes, such as "Accessibility Translator" or "AI Classroom Assistant".

One of the main challenges is the accuracy of gesture recognition. A publication on arxiv describes problems with distinguishing gestures with similar trajectories, such as "thank you" and "good". The solution is to incorporate context (previous gestures), which improved accuracy from 78% to 89% in laboratory tests. However, in real-world conditions, the results may be worse.

Implications for education and the job market

The Waterloo prototypes have the potential to significantly impact how we learn and work. Here are a few possible scenarios:

Education: From schools to vocational courses

Sign language learning – "Sign Language Tutor" could reach schools for the deaf, but also mainstream institutions as a tool supporting ASL learning.
Automation of teacher tasks – "AI Classroom Assistant" can generate quizzes, lesson summaries, or even lesson plans based on video recordings.
Content accessibility – "Accessibility Translator" can help people with cognitive difficulties understand complex texts, such as educational materials.

Job market: New skills and positions

AI Skills for All program – Google announced an initiative to train teachers in using AI tools, which could create new positions, such as "AI education specialist".
Lowering training costs – tools like "Sign Language Tutor" can reduce the costs of sign language courses for healthcare or customer service workers.
New professions – the development of AI in education could lead to the emergence of roles such as "digital accessibility coordinator" or "AI interaction designer".

However, not everyone is optimistic. In the article "Artificial Intelligence in the Job Market: Is a Revolution Coming That We Should Fear?", we emphasize that automation could lead to the reduction of some positions, especially those related to routine educational tasks. On the other hand, AI can also relieve teachers of bureaucracy, allowing them to focus on working with students.

Other projects from "Futures Lab": What else was presented at Google I/O 2026?

The collaboration with the University of Waterloo is only part of the broader "Futures Lab" initiative, through which Google presents AI prototypes with social potential. During Google I/O 2026, the following were also shown:

"Vibe Coded Quiz" – an interactive quiz with AI-generated music and graphics that adapt to the user's answers. It uses Gemini Omni 3.5 to analyze emotions based on facial expressions.
"Toca Carousel" – a tool for creating interactive stories for children, generating characters and dialogues. A collaboration with Brunelly, an educational game studio.
"Project hearme" – a prototype for translating speech into sign language in real-time (details unknown).

All these projects share one thing: the use of AI to create more engaging and accessible educational experiences. However, as with the Waterloo prototypes, most of them remain in the testing phase, and their future depends on the results of further research and user feedback.

Limitations and challenges: What stands in the way?

Despite promising results, the AI prototypes from Waterloo face a number of technical and ethical challenges. Here are the most important ones:

Technical problems

Gesture recognition accuracy – "Sign Language Tutor" has difficulty distinguishing gestures with similar trajectories, as described in the publication on arxiv. The solution may be to incorporate context, but this requires further research.
Real-time latency – the average latency in gesture recognition is 1.2 seconds, which can hinder fluid interaction.
Scalability – it is unknown how the prototypes will handle a large number of users simultaneously.

Ethical and cultural challenges

Cultural bias – "Sign Language Tutor" only works for American Sign Language (ASL). Google has announced expansion to other languages, such as Polish Sign Language (PJM), but there are no concrete plans yet.
Data privacy – user recordings are processed locally or in the cloud with anonymization, but there is a lack of details regarding security measures.
Accessibility – the tools require devices with a camera and sufficient computing power, which may exclude people with fewer resources.

Comparison with other AI projects

The Waterloo prototypes are not the only solutions using AI in education and accessibility. Here is how they compare to the competition:

MIT Media Lab – "signaloud" (2016) – gloves with sensors that translate sign language into text. Requires wearing hardware, which limits its usability.
Stanford – "deepasl" (2020) – an ASL recognition system using deep learning. It achieves 91% accuracy, but lacks a real-time feedback function.
Microsoft – "Seeing AI" (since 2017) – an app that describes the surroundings for the visually impaired. It does not focus on sign language, but shows the potential of AI in accessibility.

The Waterloo prototypes stand out for their interactivity and accessibility – they do not require additional hardware beyond a camera, and real-time feedback makes them more useful in education. However, they still lag behind in terms of gesture recognition accuracy compared to solutions like "deepasl".

What's next? The future of the Waterloo prototypes

The future of the AI prototypes from the University of Waterloo depends on several factors. First, Google must refine the technology, especially in terms of accuracy and interaction fluidity. Second, it is necessary to expand functionality to other sign languages and user groups. Third, the prototypes must be tested on a larger scale to assess their real impact on education.

Google has already announced the AI Skills for All program, which aims to train teachers in using AI tools. This is a step in the right direction, but success depends on whether the prototypes are implemented in real educational scenarios. It is also worth observing whether Google decides to commercialize these solutions – for example, through partnerships with e-learning platforms.

One thing is certain: projects like "Sign Language Tutor" show that AI can be a powerful tool in the hands of educators. However, the road from a prototype to a widely used solution is long and full of challenges – both technical and ethical.

If you are interested in the topic of automation in education, we also recommend our article "AI Agents at Work: How will they change the daily tasks of programmers and office workers in 2024–2026?", which analyzes the impact of AI on various professions. In the text "The Illusion of Full Automation: Why current LLM benchmarks don't tell us the truth about work in the knowledge economy", we address the limitations of modern AI models.