Continuing to push the new language to its limits and testing the new graphics and camera capabilities of Unity I created a dancing Bolt that will react LIVE to sound input. This sound can be a piece of music played on the computer or a live line in from the school's musical instruments. Even a microphone can be used.
The app uses a number of tweaked algorithms to allow bolt to react slightly differently to any beat or sound that hits him. Best part is, he doesn't need any preparation and will instantly adapt. I've given the computer control of the lights, camera and pyrotechnics and in the example video it's the computer that chooses to use them. My person favourite is the Flash Gordon music by queen. Those perfectly timed pyrotechnics and explosions are all the computer's doing.
Later this year Unity will gain some Machine Learning capabilities. That is, Artificial Intelligence, Neural Nets and all that jazz. As an experiment and to learn more about the subject I started to play around with machine learning. Using some off the shelf software I created a 'brain' and fed it multiple photos of Suzanne. In fact I fed it 20,000 frames of video and 850 high resolution photographs going back over the last 18 years. So no need to worry about 'faking' other people as I had to use 2 decades of memories!
At first all I was getting was blurry images but after a week, yes a whole week, the brain had studied Suzie's face so much it was producing much better results. After a few silly attempts, like changing faces in music videos I decided something more difficult. The result is a 14 year old Suzie talking about fashion in Hebrew. To make it seem more authentic I added some VHS distortions and pitched the voice closer to Suzie's. In Unity we should be able to use Machine Learning for many things like hand-writing recognition, voice commands, object recognition and other fun stuff. The more I looked into the subject, which I only dabbled into in the past, the more fascinating it is. Here is the video and afterwards a frame grab of the 'original' video so you can see the face that was altered to become Suzie. Surprisingly, the brain also altered the girl's slightly awkward expressions to be more like Suzie too, the result is an overly confident teenager talking about the fashion industry.
Like a lot of Nintendo products, Labo has a weird name and a function that almost seems silly: It is, simply put, mostly cardboard. Cardboard that's precut for you to pop out and construct remarkably elaborate contraptions with moving parts and buttons, like a fishing rod with an actual reel, or a motorcycle handlebar with a working throttle and handbrake. Then you put your creations—called Toy-Con—together with a Nintendo Switch, and things get wild, in ways that you can probably imagine (combining the fishing rod with the Switch screen for a fishing mini-game) and ways that will blow your freaking mind, like making a fully-functional toy piano.
