Posts Tagged ‘emscripten’

Video Funhouse – Convert Videos In Your Web Browser

Monday, November 11th, 2013

We (Nicole, Aaron, and I) have had a lot of fun doing Node Knockout the past few years. Last year (in 2012), our Missouri-based team got first place in the ‘Completeness’ category for our app asciigram, which converted a webcam stream into ASCII art.

This year we decided to up the game a little bit, and to try and convert any video file into another video format, while allowing filters to be applied to the video – all inside of the browser, without uploading anything. The result is Video Funhouse (here is our devcomo team page). I’m particularly proud of our project and team this year, since it was a very busy weekend all around.

Obviously, we knew we wouldn’t be able to do this in a weekend on our own without relying on great libraries and tools. There is a project called Emscripten, which is an LLVM to JavaScript compiler, so we figured we could try building FFmpeg or avconv into JavaScript to do the heavy lifting for the project. Here’s a demo video of the result (keep reading further to see more technical information):



Again, here is the link to the application if you’d like to play with it, and here is a link to the team page.

videoconverter.js

I have open sourced the build utilities and the ffmpeg.js file that we are using in the project as: videoconverter.js.

Building these projects with Emscripten was a little tricky, but actually quite simple when you consider what it was doing for us. What is also amazing about these tools is that I have little experience with build systems like this, and the tools generally just worked after a bunch of trial and error.

Why Would You Compile FFmpeg Into JavaScript?

Mostly just to see if it would work. Also, it seemed like it would be a fun project.

So, How Do You Compile FFmpeg Into JavaScript?

Here are some of notes I made while doing this, for future reference:

Compiling avconv seemed more promising at first, but when running it we bumped into some weird runtime errors with swscaler. Couldn’t figure it out quickly, so focused on FFmpeg. We pulled down the latest Emscripten, and the latest FFmpeg:

git clone git@github.com:kripken/emscripten.git
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

You may need to also get the SDK to make sure Emscripten will work. The have documentation on their site about getting this to work. Here are the basic configuration options we ended up using. Most of the configuration options are actually disabling features that either don’t make sense, or are there to save space. After a lot of trial and error, here is the minimal amount of flags to pass into emconfigure: --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic.

The first time we got a successful build it was a 100MB JS file. By removing a bunch of the other stuff, we get it down to 25MB. It could actually get smaller as well – we were unable to minify the script because the closure compiler process ran out of memory. The following commands are updated after we figured out some better options. The demo as it is right now doesn’t run any optimizations on the JS file, and it is about double the size. We can’t update the demo, as the weekend is over, but this commit is promising: ffmpeg.js: 190 additions, 1463201 deletions not shown.

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib
 
emmake make;
 
emcc -O2 -s VERBOSE=1 -s ASM_JS=0 -s ALLOW_MEMORY_GROWTH=1 -v libavutil/*.o libavcodec/*.o libavformat/*.o libavdevice/*.o libswresample/*.o libavfilter/*.o libswscale/*.o *.o -o ../ffmpeg.js --pre-js ../ffmpeg_pre.js --post-js ../ffmpeg_post.js

Peformance

The FFmpeg process runs inside of a web worker to prevent locking up the UI. This was the first optimization, as it was necessary for doing any testing on the project at all. Data is piped back from stdout and stderr to the frontend of the demo app, which is listening for these and logging them to a pre tag. This logging could be limited or removed to limit the impact on the DOM, as a lot of data can be generated by the system.

We experimented with firing up 2 different workers, one for the metadata probe and actual processing, and one for taking screenshots of a video right after receiving it. But we noticed some general instability when both were running at once (many times a tab or browser would crash), so we removed the screenshotting functionality. I’m sure we could have traced it down and improved the performance if we had more than a weekend for the project.

The processing itself is a little slow, but I haven’t done benchmarks to compare with native. I generally avoided processing videos of much size. Also, our demo app prints a lot of logs which may not be helping matters. We are looking into how to set up performance tests, but if anyone wants to help with this please let me know or submit a PR on the videoconverter.js project.

I believe this could be quite a bit faster – it couldn’t actually use asm.js because it is explicitly disabled when ALLOW_MEMORY_GROWTH is defined. It would be worth experimenting to see if we could set a large max memory size and enable asm to see what kind of speedup we saw.

I’m interested to see if we can get some sort of performance benchmarking set up to compare how fast it runs in different JavaScript environments, and to see how it compares with native performance.

Potential Uses

  • This could be extended to bundle other encoding libraries. I wasn’t able to figure out how to get the linking to work for libvpx, x264, libtheora, zlib, etc over the weekend, so certain formats cannot be encoded. It would be neat to have this functionality, as it would allow conversion into web friendly video, which could then allow further previewing and processing. As performance improves and FileSystem API support improve it may be possible to build a web-based video or audio editing tool using this.
  • Compile the other utilities – there are other programs that ship with FFmpeg that we are excluding right now. ffprobe in particular seems like it could be better for gathering the metadata after the initial load of the file.
  • As mentioned previously, this could be useful as a benchmarking tool.
  • Probably other things I haven’t thought of as well.