Interactive Music Visualization
(2019)
The following is a development blog of sorts for my early experiments creating interactive AV tools.

Connecting Instruments to a Game Engine (October, 2020)

I dug this project off an old hard drive. I had thrown it together in an afternoon a few years ago and it seemed relevant, so I tried to record some of it. The original idea was to explore how one could connect a game engine with real time MIDI information coming from external instruments and applications. The proof of concept was simple: create a virtual keyboard :)
  • Entirely blueprint based (coding this would have been easier, but debugging these blueprints is a music visualization itself)
  • A MIDI event manager, which receives incoming MIDI events (note onset/offset/velocity etc.)
  • Procedurally generated keyboard mesh, where each key is linked to a corresponding MIDI value
  • Scale and emissive materials react to the incoming signals
This is already a blast to sit in front of while playing, and I'm excited to extend this pattern to control more exciting behavior in a virtual world.

PS: For any of those interested, just improvising over some 2-5-1 in C#.

Using AI to Distort Videos in 3D (October, 2020)

While not exactly interactive, I wanted to follow up on the last experiment which explored how semantic understanding of video content could drive effects on top of that video. In that first experiment (using the ipod commercial), the effect was driven by a simple color threshold... which is far from a semantic understanding of the content. So to follow up, I decided to distort the 3D geometry of a mesh based on the content of a video, as interpreted by an artificial neural network. The rough process is as follows:
​
  • Create masks for each frame of video using an Instance Segmentation Network trained to segment people from images. Funny enough...  this was easiest and quickest part of the experiment by far.
  • Create 2 synchronized videos (1 original, 1 mask) and media textures for each
  • Create a procedural grid mesh with vertices linked to the the 2 media textures through UVs
  • Color vertices of the mesh by sampling the media texture for the original video
  • Offset vertex positions by sampling the media texture for the mask video

While the end result isn't as exciting or visually pleasing as past explorations, I enjoyed incorporating aspects of AI and computer vision to drive the visual effects. I think this could be used to drive more subtle effects (particles, distortions etc) in something like a music video or hyper stylized film.

Segmenting Objects in Videos (July, 2019)

Messing with video textures was a success, and I'm eager to see if the 3D effects can be used to target more semantically meaningful content in the video. This experiment segments dancers in the video and applies depth effects to the geometry selectively. This is accomplished entirely inside the shader, and the entire grid is a single mesh.

Adding Dimensions to Medium, Playing Video (July, 2019)

I was playing around with video textures for another project (interactive cloth simulation), and thought it would be fun to try adding a dimension to what is traditionally a very 2D medium (video).

Interaction in Virtual Reality (July, 2019)

Further along the path to interactive. This experiment extends the previous, by using the proximity of tracked hand controllers to control the behavior/intensity of the reactive visualizer.

​Video coming soon. Desktop capture in VR is wonky.

Adding Interaction (July, 2019)

The goal is interactivity, and up until now all these visuals have been reactive but not interactive. I started with a simple experiment where the intensity/behavior of a reactive visualizer is influenced by the proximity of a target object.

Visualizing Frequency Spectrum, Spherical (October, 2020)

Similar principles applied in spherical coordinates
  • Procedurally generated form (fibonacci sphere)
  • Instanced static meshes for performance
  • Can be modified in real-time
  • Implemented various interpolation methods to yield smoother, or more pixelated visuals
  • Implemented various visualization styles (fibonacci sphere index, equator & inv, latitude & inv, longitude & inv, lat+long & inv)

Visualizing Frequency Spectrum, 2D (July, 2019)

For the next reactive visualizer experiment, I extended the same principles to a 2D grid of cubes.
  • Procedurally generated grid
  • Instanced static meshes for performance
  • Can be modified in real-time
  • Implemented various interpolation methods to yield smoother, or more pixelated visuals
  • Implemented various visualization styles (row, col, radial, inverse radial etc.)

Visualizing Frequency Spectrum, 1D (July, 2019)

I decided to switch from using my custom DSP library, to using an existing audio analysis plugin, mostly for the ability to process external audio streams (ie: PC audio) rather an imported file. This allowed me to focus effort on the visualizations.

I started by creating a ReactiveVisualizer interface and a manager which pipes the audio analysis to any implementations that exist in the scene. The first implementation was a simple exercise in moving from the familiar 2D frequency spectrum visualizations to actual geometry. A procedurally generated row of cubes corresponds to various frequency ranges. Each cube grows in size based on the intensity/amplitude of the corresponding frequency.

Analyzing Audio (July, 2019)

I started this project by writing a custom digital signal processing library for UE4 which relied on the FMOD library. I borrowed inspiration from www.parallelcube.com/2018/03/01/using-thirdparty-libraries-in-our-ue4-mobile-project/ 

This worked for a basic frequency analyzer!

However, after some experimentation, it seemed like FMOD would be an "all in" kind of solution to audio analysis, and I was after something a bit more flexible.