In 2021 I had the pleasure to work for electronic musician Nicolas Jaar and his art collective “Shockforest group”.
In this creative coding project, we built a multi-sensory user interface for an experimental chatbot, combining text with abstract visuals and physics animations (gravity, directional forces), as well as custom music and sound effects.
This photo shows the original application built with Unity. (Credits: Jelger Kroese)
What I did
My task was to migrate an existing application that was written in Unity (by Jelger Kroese) into a web-based application (using HTML, CSS and Javascript) and optimised for both mobile and desktop screens.
Design and UX
To enable a good user experience on smaller screens, I adapted the original interaction patterns as well as tweaked the layout and UI design. For example, the original circular layout was not optimal for vertical mobile screens, so constrained the UI to the middle area of the circle.
We included additional affordances into the UI to make it clear to the user that they can interact with the shapes by dragging them.
Technicalities
I built the application as a React application
For physics animation I used the javascript library matter.js.
For the canvas visualisations I used react-conva.
Extra: This image was created by recording users interactions with the application. I think it is a beautiful data visualisation.
Each graphical pattern reveals how the user interacted with the circular UI by dragging the mouse (or finger).
