Add audio visualization

[shakeelmohamed]

First of all, I'm not sure if this is even possible since we don't have direct access to audio files (for YouTube or Soundcloud), but it would be a cool option to add.

Of course this needs to be off by default otherwise it defeats the purpose of this project 😃

One possible viz library we can use is Wayou/HTML5_Audio_Visualizer as long as we leave a reference to his work. We may want to fork it to customize it to our liking.

If we can't do this directly, there may be some APIs in Chrome (and other browsers) that we can access with an extension.

edit: just use some of @MrAnyone's code from https://github.com/MrAnyone/YouTube-Audio-Visualizer as it's MIT licensed 🎉

[shakeelmohamed]

A blog post on creating a custom viz: https://www.patrick-wied.at/blog/how-to-create-audio-visualizations-with-javascript-html

[MatMercer]

Using a WebGL library is the best way to do that, because audio processing is in real time (can be at 60FPS) and the WebGL have the power to render at 60FPS, making the visualizer veeery smooth.

Currently I use Pixi.js

[timchoh585]

To add on, i think the hardest thing would be just to get the mp3 file from YouTube or Soundcloud. Not too sure how to do that myself, but that would be the biggest challenge.

I would go as far to say that even creating something with D3 could be a good solution as well. Not too familiar with how this would be compared to Pixi.js I would like to try and implement a version with D3 just because i am more familiar with it. But will definitely look into Pixi.js because it looks pretty sleek.

[shakeelmohamed]

I would like to try and implement a version with D3 just because i am more familiar with it.

@timchoh585 go for it! It looks like @MrAnyone is using audioContext to get audio data, more on there here

[timchoh585]

@shakeelmohamed thanks! I will definitely be working on this throughout the week! I'll try and keep this as updated as possible

[shakeelmohamed]

@timchoh585 excellent, please ping me if you need anything

[timchoh585]

@shakeelmohamed sorry, got pretty busy with school and work. should have something to show within the week.

[andreasvirkus]

@timchoh585 @shakeelmohamed Would it be ok for me to take a crack at this?
Any specific design(colour palette)/positioning you had in mind?

[shakeelmohamed]

@andreasvirkus go for it, maybe go with the blue we use for links? Let's get something working then we can tweak the colors later

[andreasvirkus]

After two nights of tinkering, I'm calling out for some help (if a solution exists at all).
All of the AV libs rely on AudioContext, which can be accessed through an HTML5 <video> or <audio> element. Since youtube doesn't allow CORS, we can't access the video element in the iFrame. So we'd need the url for YT's source files to create our own tag (which we can scrape). But YouTube's blocked access to those URLs so you get a 403..

I'm not aware of any way to access the audio stream through the iFrame. Any thoughts/suggestions on how to proceed/what to try?