David Young
  • Home
  • About
  • Contact
  • Personal Projects
    • Computer Science >
      • Computer Vision >
        • 2016 - Homography w/ RANSAC
        • 2016 - Fundamental Matrix & Triangulation
        • 2016 - Laplacian Blob Detector
        • 2016 - Photometric Stereo: Shape From Shading
        • 2015 - Optical Character Recognition w/ OpenCV and Deep Learning
        • 2015 - Feature Detection
        • 2015 - Feature Description
        • 2015 - Feature Matching
        • 2015 - Panoramas (Alignment, Stitching, Blending)
        • 2015 - Facial Detection & Recognition
        • 2015 - Single View Modeling
      • Artificial Intelligence >
        • 2019 - Talk: How Neural Networks See the World
        • 2018 - Generating Text and Poetry
        • 2015 - Optical Character Recognition w/ OpenCV and Deep Learning
        • 2015 - Constraint Satisfaction Problems
        • 2015 - Adversarial Search
        • 2015 - Path Planning (Mazes + Pacman)
        • 2015 - Digit Classification (Bayes)
        • 2015 - Text Document Classification (Bayes)
        • 2015 - Multi-Class Perceptrons
        • 2015 - Markov Decision Processes & Reinforcement Q-Learning
        • 2015 - Simulating Neuronal Learning during Brain-Machine Interface
      • Machine Learning >
        • 2016 - Naive Bayes Classifiers in R
        • 2016 - Stochastic Gradient Descent (SVM in R)
        • 2016 - Comparing Classifiers in R
        • 2016 - Visualize High Dim Data: Blob Analysis + PCA
        • 2016 - Image Segmentation w/ EM
        • 2016 - Regression Kernel Smoothing
        • 2016 - Multinomial Regression on Wide Datasets
      • Robotics >
        • 2017 - 3dof Parallel Motion Simulator
        • 2015 - Designing a Hybrid Controller
        • 2015 - Controlling Pendubot with a Kinect
      • Computer Architecture >
        • 2016 - Architecture Support for Accelerator Rich CMPs
        • 2014 - Weighted Vector Addition with Cuda Framework
        • 2014 - Parallel Reduction with Cuda Framework
        • 2014 - Designing a Pipelined CPU
        • 2014 - Intel SSE Intrinsics Applications in Rudimetary Matrix Algorithms
        • 2014 - LIFC to MIPS Compiler and Assembler
      • Web Development >
        • 2014 - Javascript Calendar
        • 2014 - Multi-Room Chat Server
      • Graphics >
        • 2015 - Basic Animation w/ WebGL
        • 2015 - Diamond Square Terrain Generator
        • 2015 - Flight Simulator w/ WebGL
        • 2015 - Multi-Program Texture Mapping WebGL
      • Software >
        • 2015 - Consumer Grade Gaze Pattern Recognition Software
        • 2015 -Test History Jenkins Plugin
      • Other >
        • 2014 - Hashtable for Genomic DNA Sequences
        • 2014 - Closest Pair of Points
    • Virtual Reality, Game Design, & Animation >
      • 2019 - Interactive Music Visualization
      • 2016 - Visualizing Runtime Flowpath in VR
      • 2016 - Fiducial Marker Tracking for Augmented Reality
      • 2015 - Experimenting with PhysX & APEX Destruction
      • 2015 - Rigging Tank Treads using MEL in Maya
      • 2015 - Automated Simulation Teddy Bear Bin
      • 2014 - Networked Multiplayer Game of Set
      • 2014 - Asymmetrical Multiplayer Destruction
      • 2016 - Tracking & Depth Perception
      • 2014 - 8 Week Game Design (Cave Survival)
      • 2015 - Experimenting with Nvidia FLEX
    • Computers >
      • Custom and Watercooled PCs
      • Component Reviews
      • Installation Guides
    • Quantitative Physiology >
      • Computational >
        • 2015 - Modelling Neurons & Action Potentials
        • 2015 - Simulating Neuronal Learning during Brain-Machine Interface
        • 2014 - Imaging: Rabbit Optical Mapping
        • 2014 - Simulating Electrical Stimulation w/ Comsol
        • 2014 - Ion Channels
        • 2013 - Designing Filters to Simulate Olfactory Sensation
        • 2014 - CardioVascular Mechanics
        • 2014 - Renal
        • 2013 - Principal Component Analysis & Singlar Value Decomposition
        • 2013 - 3D Printed Frog Muscle Holder
      • Physical >
        • 2013 - Biomedical Signal Acquisition
        • 2013 - Electrooculogram
        • 2013 - Compound Action Potential in Frog Sciatic Nerve
        • 2013 - Contractile Properties of Frog Skeletal Muscle
        • 2013 - Locust Olfaction
        • 2014 - Voltage Clamp
        • 2013 - Dive Response
        • 2014 - Frog Heart Muscle
        • 2013 - Ultrasound
        • 2014 - Biological Signal Conditioning
        • 2014 - EKG, Vector Cardiograms & Pulse Wave Velocity
    • Electrical Projects >
      • Self Balancing Robot Pendulum
      • Custom Beer Pong Tables
      • 4-axis Robotic Arm
      • Modified Electric MiniBike
      • Secret Knock Detecting Automatic Door Opener
      • Car Audio
      • Tree-House Wiring
      • Laser Harp
    • Auto & Mechanical Projects >
      • Single Turbo Lexus SC300
      • Track Day Mx-5
      • Karting
      • Racing Simulator Rig
      • 50cc Barbie Jeep
    • Random Other Projects >
      • Talk: Embodied Cognition
      • Bathymetry Coffee Table
      • Not your average Tree House
      • Pneumatic Tennis Ball Cannon
  • Experience
    • Resume
    • Work Experience
    • Programming Experience
    • Research Experience
    • Service Work
  • Education
  • Hobbies
    • Motorsports
    • Art
    • Music
    • Gaming
    • Dancing

Interactive Music Visualization
(2019)


The following is a development blog of sorts for my early experiments creating interactive AV tools.

Connecting Instruments to a Game Engine (October, 2020)

I dug this project off an old hard drive. I had thrown it together in an afternoon a few years ago and it seemed relevant, so I tried to record some of it. The original idea was to explore how one could connect a game engine with real time MIDI information coming from external instruments and applications. The proof of concept was simple: create a virtual keyboard :)
  • Entirely blueprint based (coding this would have been easier, but debugging these blueprints is a music visualization itself)
  • A MIDI event manager, which receives incoming MIDI events (note onset/offset/velocity etc.)
  • Procedurally generated keyboard mesh, where each key is linked to a corresponding MIDI value
  • Scale and emissive materials react to the incoming signals
This is already a blast to sit in front of while playing, and I'm excited to extend this pattern to control more exciting behavior in a virtual world.

PS: For any of those interested, just improvising over some 2-5-1 in C#.

Using AI to Distort Videos in 3D (October, 2020)

While not exactly interactive, I wanted to follow up on the last experiment which explored how semantic understanding of video content could drive effects on top of that video. In that first experiment (using the ipod commercial), the effect was driven by a simple color threshold... which is far from a semantic understanding of the content. So to follow up, I decided to distort the 3D geometry of a mesh based on the content of a video, as interpreted by an artificial neural network. The rough process is as follows:
​
  • Create masks for each frame of video using an Instance Segmentation Network trained to segment people from images. Funny enough...  this was easiest and quickest part of the experiment by far.
  • Create 2 synchronized videos (1 original, 1 mask) and media textures for each
  • Create a procedural grid mesh with vertices linked to the the 2 media textures through UVs
  • Color vertices of the mesh by sampling the media texture for the original video
  • Offset vertex positions by sampling the media texture for the mask video

While the end result isn't as exciting or visually pleasing as past explorations, I enjoyed incorporating aspects of AI and computer vision to drive the visual effects. I think this could be used to drive more subtle effects (particles, distortions etc) in something like a music video or hyper stylized film.

Segmenting Objects in Videos (July, 2019)

Messing with video textures was a success, and I'm eager to see if the 3D effects can be used to target more semantically meaningful content in the video. This experiment segments dancers in the video and applies depth effects to the geometry selectively. This is accomplished entirely inside the shader, and the entire grid is a single mesh.

Adding Dimensions to Medium, Playing Video (July, 2019)

I was playing around with video textures for another project (interactive cloth simulation), and thought it would be fun to try adding a dimension to what is traditionally a very 2D medium (video).

Interaction in Virtual Reality (July, 2019)

Further along the path to interactive. This experiment extends the previous, by using the proximity of tracked hand controllers to control the behavior/intensity of the reactive visualizer.

​Video coming soon. Desktop capture in VR is wonky.

Adding Interaction (July, 2019)

The goal is interactivity, and up until now all these visuals have been reactive but not interactive. I started with a simple experiment where the intensity/behavior of a reactive visualizer is influenced by the proximity of a target object.

Visualizing Frequency Spectrum, Spherical (October, 2020)

Similar principles applied in spherical coordinates
  • Procedurally generated form (fibonacci sphere)
  • Instanced static meshes for performance
  • Can be modified in real-time
  • Implemented various interpolation methods to yield smoother, or more pixelated visuals
  • Implemented various visualization styles (fibonacci sphere index, equator & inv, latitude & inv, longitude & inv, lat+long & inv)

Visualizing Frequency Spectrum, 2D (July, 2019)

For the next reactive visualizer experiment, I extended the same principles to a 2D grid of cubes.
  • Procedurally generated grid
  • Instanced static meshes for performance
  • Can be modified in real-time
  • Implemented various interpolation methods to yield smoother, or more pixelated visuals
  • Implemented various visualization styles (row, col, radial, inverse radial etc.)

Visualizing Frequency Spectrum, 1D (July, 2019)

I decided to switch from using my custom DSP library, to using an existing audio analysis plugin, mostly for the ability to process external audio streams (ie: PC audio) rather an imported file. This allowed me to focus effort on the visualizations.

I started by creating a ReactiveVisualizer interface and a manager which pipes the audio analysis to any implementations that exist in the scene. The first implementation was a simple exercise in moving from the familiar 2D frequency spectrum visualizations to actual geometry. A procedurally generated row of cubes corresponds to various frequency ranges. Each cube grows in size based on the intensity/amplitude of the corresponding frequency.

Analyzing Audio (July, 2019)

I started this project by writing a custom digital signal processing library for UE4 which relied on the FMOD library. I borrowed inspiration from www.parallelcube.com/2018/03/01/using-thirdparty-libraries-in-our-ue4-mobile-project/ 

This worked for a basic frequency analyzer!

However, after some experimentation, it seemed like FMOD would be an "all in" kind of solution to audio analysis, and I was after something a bit more flexible.

Proudly powered by Weebly