Voy empowers all participants to explore the beauty of data-driven, computer-generated images by using predetermined hand gestures in midair.
What if we could explore something beautiful? Something magical that is hidden in multidimensional space? To answer these questions, we would like to take you on a little journey and give you some insights into our process.
Earlier this year, we received a call from an old college friend, who invited us to be a part of his art and culture festival. He wanted us to create something special. The main idea was to inspire all visitors of the festival to participate and to be part of an overall experience.
This call also revealed that there were no restrictions in developing ideas or potential solutions. We were fired up from the first moment and amazed about the opportunity.
Equipped with pen and paper, we immediately started our creation and brainstorming session for the next couple of days. During this time, we talk about suitable development strategies and upcoming tech trends. We did a lot of research and started to create an accurate picture of how all parts could fit together. One of the first outcomes during this time was that this work should have the capability of generating a physical output, to which every participant builds an emotional bond.
We thought about the mechanic of a fortune cookie. Every now and then you grab one of these cookies, and you really care about what’s written on the little paper inside. Maybe these words buzz around in your head all day… So, if we can achieve that all participants think about or even get reminded of the great experience they had, although it’s already over, the job is done. But since we’re visually driven, text-only does not work in our case. Some kind of eye-pleasing component should not be missing.
After all the research we did, we came up with the idea to take advantage of machine learning and image processing to generate unique content. On its own, this topic already means a lot of work if you want to achieve quite good results. Lucky for us, we found an amazing tool called RunwayML, which is a video editing engine powered by machine learning.
With the help of Runway, we easily had access to a variety of pre-trained StyleGAN2 models. Using these models, we created numerous computer-generated images which perfectly fitted our needs. The visuality generates an overall awareness of its process of creation and inspires to explore its countless details, unknown forms, patterns, and splashing colors.
Based on these fascinating and unpredictable forms, we continued to craft our kind of fortune cookie paper, our souvenir. As shown above, our souvenir consists of an image complemented with an inspiring tagline. Finally, at the bottom, there is the guiding principle of our art installation.
We discover. We explore. We learn.
We are user-orientated and aim to combine several technologies to create an overall experience. Therefore, we wanted all participants to be able to control and navigate our installation just with their hands. No other physical devices are needed.
Especially during dark lighting conditions, we had to make sure that the ease-of-use doesn’t drop off. Considering this requirement, we decided to make use of infrared-based hand tracking.
Using this kind of tracking, we limited the predetermined hand gestures for control. There are 1. Open hand and 2. Clenched fist. During use, the open hand influences the playing direction of the visuals on the screen. Positioning far-right means playing in the forwarding direction, far left stands for the opposite direction. It’s even possible to control the playback speed. A fast playback speed means a more harmonious appearance of the visuals on the screen. The further you get into the middle of the tracking area, the slower the speed becomes. This mechanic also works the other way around.
If you come across something visually appealing, it’s possible to get a more detailed view by clenching your fist. Finally, if you are charmed by what you see on the screen and decide to make it your own, clinch your first again and wait for the print to start.
From a development perspective, we kept our tech stack clear and up to date. Opting for web technologies gave us the advantage of being able to deploy a once-developed application to different types of channels. For example, we can implement this application in a physical installation, limited in place and amount of participants.
Or, we can make this same application available to many people on the internet, detached from time and place. To accomplish an overall look and feel on how things work together, we planned and simulated the arrangement for all components of our installation on-site.
The setup consists of a 60-inch 4K monitor, an infrared-based hand tracking, a printer, and a quite powerful computer as well as the application itself. The appearance of the freestanding stele is mainly inspired by the work Modulor.
This work provided us with some good reference points to determine the actual height and the final placement of the hand tracking sensor. Its visuality derives from the previously crafted design direction.
Finally, the testing of all functionalities was a crucial part of the workflow. We had to ensure that everything worked as expected. Especially the functionality of the printer. Therefore, we evaluated our work many times and went through a lot of iteration steps during design and development.
During the festival day, we had a tight schedule and some other tasks to manage before we could finally start with our work. Sebastian and I headed from Hamburg, northern Germany, to the Midwest of the country within a couple of hours. Fortunately, we arrived on time to make some final adjustments and to make sure that everything works correctly.
Final adjustments have been made, the test routine has been completed, and we made sure, everything works as expected.
As you can see, the container has just the right size for our setup. In addition, it is just large enough to accommodate only one person at a time inside and in front of the installation.
Due to the current situation, this was a mandatory procedure and the number of participants during the festival was quite manageable.
In the end, we all had a lot of fun. But most importantly, we did this work for our old college friend and his art and culture festival, who was very happy with the result!
We hope the situation will soon be normal again, and we all can start to enjoy art and culture in a more harmonic and relaxed way.
Stay updated ✌
Twitter:
https://twitter.com/tmjns92
https://twitter.com/Sebasti20083087
More visuals:
https://vimeo.com/user125935172
Links:
https://runwayml.com/
