Yandex.Browser neural networks have learned to translate videos from Chinese and automatically overlay Russian voice acting, a Yandex representative said. The technology is already working on YouTube, support for the popular Chinese video platform Bilibili will be available later.
It is reported that the Yandex developers assembled their own dataset from scratch to train the neural network, since there were no ready-made solutions. Using a special algorithm, they processed 1.5 million videos and selected 100,000 with high-quality subtitles in Northern Chinese, which is spoken by more than 1 billion people.
The Browser’s neural networks have learned thousands of hieroglyphics, each of which is read as a syllable. In order for the neural network to learn to understand them, the developers coded everything necessary to work with the Chinese dictionary in the form of 10 thousand tokens.
For comparison: the five European languages that Browser learned to translate earlier fit in a library of 5000 tokens. A token is equal to one syllable or a regularly used combination of two syllables.
At the same time, the neural network was trained to divide a continuous stream of hieroglyphs into semantic segments in order to more accurately synchronize the translation with the original speech and improve its quality.
In a month, the Navigator also learned the four tones of the Chinese language that determine the meaning of what is said.
The same Yandex.Browser offers to translate the video from Chinese to YouTube immediately after the user opens it. You need to click the “Translate Video” button.
- Previously, Yandex added live broadcast voiceover machine translation technology to its browser for some YouTube channels, including TED, Google, Apple, SpaceX.
- In March 2022, interactive subtitles for videos in English, French, German and Spanish appeared on Yandex Browser. The company also added a word highlighting option: by clicking on an unknown word, the user will receive an instant translation.
Author:
anastasia mariana
Source: RB
I am a professional journalist and content creator with extensive experience writing for news websites. I currently work as an author at Gadget Onus, where I specialize in covering hot news topics. My written pieces have been published on some of the biggest media outlets around the world, including The Guardian and BBC News.
