|
|
Interactive Music Visualization
|
|
The following is a development blog of sorts for my early experiments creating interactive AV tools.
Segmenting Objects in Videos (July, 2019)
Messing with video textures was a success, and I'm eager to see if the 3D effects can be used to target more semantically meaningful content in the video. This experiment segments dancers in the video and applies depth effects to the geometry selectively. This is accomplished entirely inside the shader, and the entire grid is a single mesh.
|
|
|
Adding Dimensions to Medium, Playing Video (July, 2019)
I was playing around with video textures for another project (interactive cloth simulation), and thought it would be fun to try adding a dimension to what is traditionally a very 2D medium (video).
Interaction in Virtual Reality (July, 2019)
Further along the path to interactive. This experiment extends the previous, by using the proximity of tracked hand controllers to control the behavior/intensity of the reactive visualizer.
Video coming soon. Desktop capture in VR is wonky.
Video coming soon. Desktop capture in VR is wonky.
Adding Interaction (July, 2019)
The goal is interactivity, and up until now all these visuals have been reactive but not interactive.I started with a simple experiment where the intensity/behavior of a reactive visualizer is influenced by the proximity of a target object.
Visualizing Frequency Spectrum, 2D (July, 2019)
For the next reactive visualizer experiment, I extended the same principles to a 2D grid of cubes.
- Procedurally generated grid
- Instanced static meshes for performance
- Can be modified in real-time
- Implemented various interpolation methods to yield smoother, or more pixelated visuals
- Implemented various visualization styles (row, col, radial, inverse radial etc.)
Visualizing Frequency Spectrum, 1D (July, 2019)
I decided to switch from using my custom DSP library, to using an existing audio analysis plugin, mostly for the ability to process external audio streams (ie: PC audio) rather an imported file. This allowed me to focus effort on the visualizations.
I started by creating a ReactiveVisualizer interface and a manager which pipes the audio analysis to any implementations that exist in the scene. The first implementation was a simple exercise in moving from the familiar 2D frequency spectrum visualizations to actual geometry. A procedurally generated row of cubes corresponds to various frequency ranges. Each cube grows in size based on the intensity/amplitube of the corresponding frequency.
I started by creating a ReactiveVisualizer interface and a manager which pipes the audio analysis to any implementations that exist in the scene. The first implementation was a simple exercise in moving from the familiar 2D frequency spectrum visualizations to actual geometry. A procedurally generated row of cubes corresponds to various frequency ranges. Each cube grows in size based on the intensity/amplitube of the corresponding frequency.
Analyzing Audio (July, 2019)
I started this project by writing a custom digital signal processing library for UE4 which relied on the FMOD library. I borrowed inspiration from www.parallelcube.com/2018/03/01/using-thirdparty-libraries-in-our-ue4-mobile-project/
This worked for a basic frequency analyzer!
However, after some experimentation, it seemed like FMOD would be an "all in" kind of solution to audio analysis, and I was after something a bit more flexible.
This worked for a basic frequency analyzer!
However, after some experimentation, it seemed like FMOD would be an "all in" kind of solution to audio analysis, and I was after something a bit more flexible.