Archive for the ‘Web Development’ Category

Video Funhouse – Convert Videos In Your Web Browser

Monday, November 11th, 2013

We (Nicole, Aaron, and I) have had a lot of fun doing Node Knockout the past few years. Last year (in 2012), our Missouri-based team got first place in the ‘Completeness’ category for our app asciigram, which converted a webcam stream into ASCII art.

This year we decided to up the game a little bit, and to try and convert any video file into another video format, while allowing filters to be applied to the video – all inside of the browser, without uploading anything. The result is Video Funhouse (here is our devcomo team page). I’m particularly proud of our project and team this year, since it was a very busy weekend all around.

Obviously, we knew we wouldn’t be able to do this in a weekend on our own without relying on great libraries and tools. There is a project called Emscripten, which is an LLVM to JavaScript compiler, so we figured we could try building FFmpeg or avconv into JavaScript to do the heavy lifting for the project. Here’s a demo video of the result (keep reading further to see more technical information):



Again, here is the link to the application if you’d like to play with it, and here is a link to the team page.

videoconverter.js

I have open sourced the build utilities and the ffmpeg.js file that we are using in the project as: videoconverter.js.

Building these projects with Emscripten was a little tricky, but actually quite simple when you consider what it was doing for us. What is also amazing about these tools is that I have little experience with build systems like this, and the tools generally just worked after a bunch of trial and error.

Why Would You Compile FFmpeg Into JavaScript?

Mostly just to see if it would work. Also, it seemed like it would be a fun project.

So, How Do You Compile FFmpeg Into JavaScript?

Here are some of notes I made while doing this, for future reference:

Compiling avconv seemed more promising at first, but when running it we bumped into some weird runtime errors with swscaler. Couldn’t figure it out quickly, so focused on FFmpeg. We pulled down the latest Emscripten, and the latest FFmpeg:

git clone git@github.com:kripken/emscripten.git
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

You may need to also get the SDK to make sure Emscripten will work. The have documentation on their site about getting this to work. Here are the basic configuration options we ended up using. Most of the configuration options are actually disabling features that either don’t make sense, or are there to save space. After a lot of trial and error, here is the minimal amount of flags to pass into emconfigure: --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic.

The first time we got a successful build it was a 100MB JS file. By removing a bunch of the other stuff, we get it down to 25MB. It could actually get smaller as well – we were unable to minify the script because the closure compiler process ran out of memory. The following commands are updated after we figured out some better options. The demo as it is right now doesn’t run any optimizations on the JS file, and it is about double the size. We can’t update the demo, as the weekend is over, but this commit is promising: ffmpeg.js: 190 additions, 1463201 deletions not shown.

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib
 
emmake make;
 
emcc -O2 -s VERBOSE=1 -s ASM_JS=0 -s ALLOW_MEMORY_GROWTH=1 -v libavutil/*.o libavcodec/*.o libavformat/*.o libavdevice/*.o libswresample/*.o libavfilter/*.o libswscale/*.o *.o -o ../ffmpeg.js --pre-js ../ffmpeg_pre.js --post-js ../ffmpeg_post.js

Peformance

The FFmpeg process runs inside of a web worker to prevent locking up the UI. This was the first optimization, as it was necessary for doing any testing on the project at all. Data is piped back from stdout and stderr to the frontend of the demo app, which is listening for these and logging them to a pre tag. This logging could be limited or removed to limit the impact on the DOM, as a lot of data can be generated by the system.

We experimented with firing up 2 different workers, one for the metadata probe and actual processing, and one for taking screenshots of a video right after receiving it. But we noticed some general instability when both were running at once (many times a tab or browser would crash), so we removed the screenshotting functionality. I’m sure we could have traced it down and improved the performance if we had more than a weekend for the project.

The processing itself is a little slow, but I haven’t done benchmarks to compare with native. I generally avoided processing videos of much size. Also, our demo app prints a lot of logs which may not be helping matters. We are looking into how to set up performance tests, but if anyone wants to help with this please let me know or submit a PR on the videoconverter.js project.

I believe this could be quite a bit faster – it couldn’t actually use asm.js because it is explicitly disabled when ALLOW_MEMORY_GROWTH is defined. It would be worth experimenting to see if we could set a large max memory size and enable asm to see what kind of speedup we saw.

I’m interested to see if we can get some sort of performance benchmarking set up to compare how fast it runs in different JavaScript environments, and to see how it compares with native performance.

Potential Uses

  • This could be extended to bundle other encoding libraries. I wasn’t able to figure out how to get the linking to work for libvpx, x264, libtheora, zlib, etc over the weekend, so certain formats cannot be encoded. It would be neat to have this functionality, as it would allow conversion into web friendly video, which could then allow further previewing and processing. As performance improves and FileSystem API support improve it may be possible to build a web-based video or audio editing tool using this.
  • Compile the other utilities – there are other programs that ship with FFmpeg that we are excluding right now. ffprobe in particular seems like it could be better for gathering the metadata after the initial load of the file.
  • As mentioned previously, this could be useful as a benchmarking tool.
  • Probably other things I haven’t thought of as well.

console.log helper function

Thursday, April 4th, 2013

I have often found it tricky to get a ‘just-right’ console.log wrapper in JavaScript. I would prefer to just type log, but writing a wrapper function is a little trickier than at first glance. A variation of this used to exist in the HTML5 Boilerplate plugin.js but is now missing.

I have two versions – one is bigger, and is intended for an application – it stubs out console methods when they don’t exist, and exposes a log function to the window. The second is portable – it is meant for use inside of plugins.

The neat thing about this technique is that this preserves the original line of code inside of the console.log statement, rather than the line in which the log function is declared.


The code is in a gist, which is also embedded below:

Devtools Colorpicker Palette

Saturday, February 2nd, 2013

I’m still trying to figure out how to get a list of all the colors in use on a page from within devtools, but I’ve been hacking together a little demo with a custom Web Inspector frontend that loads a color palette on the side.

When you click one of the swatches, it sets the current color of the element. The idea is that it would include all the colors in your page, maybe ordered by most used. Possibly could even get more advanced and suggest a set of complementary or analagous colors based on your current selection.

A prototype implementation of a palette using color swatches within devtools

A prototype implementation of a palette using color swatches within devtools

jQuery UI 1.9 Boilerplate

Wednesday, October 17th, 2012

jsbin has made having to remember some of these links less important. However, with jQuery UI 1.9 coming out, there are some CDN links that aren’t working. Here is the updated boilerplate I use to quickly test out a jQuery UI project. If all goes well, you should have a dialog that you can drag around and resize.

<!doctype html>
<html>
<head>
  <title></title>
 
  <link rel="stylesheet" href="http://code.jquery.com/ui/1.9.0/themes/base/jquery-ui.css">
 
  <script type="text/javascript" src="http://code.jquery.com/jquery-1.8.2.js"></script>
  <script src="http://ajax.aspnetcdn.com/ajax/jquery.ui/1.9.0/jquery-ui.js" type="text/javascript"></script>
 
  <style type="text/css">
    #modal {
	 background: orange;
    }
  </style>
 
  <script type='text/javascript'>
    $(function() {
        $("#modal").dialog({ title: "It works" })
    });
  </script>
</head>
 
<body>
<div id='modal'>jQuery UI!</div>
</body>
 
</html>

As a bonus, here is the markup in jsbin.

DevTools Feature Request – console.scope()

Wednesday, September 12th, 2012

Sometimes I just want to see all (or many) of the variables inside of the function, and it can be a bit of work to copy all of the variables into the console.log() call when you know that you are just going to delete the line afterwards anyway.

It would be great to have a console function that handles this. It would be called console.scope. When you call console.scope() it would act similarly to console.trace(), except instead of seeing the call stack in the console, you would see the same thing that you see in the ‘Scope Variables’ panel when inside a breakpoint in the Sources Panel (see images).

I have also opened a thread about this on the devtools mailing list.

TinyColor On npm

Wednesday, September 12th, 2012

I have published my TinyColor package to the npm repository – https://npmjs.org/package/tinycolor2. Unfortunately it is called tinycolor2 since tinycolor was already taken.

After running

npm install tinycolor2

You can use tinycolor like so:

var tinycolor = require("./tinycolor");
console.log(tinycolor("red").toHex());

See the project homepage or the README for more info.

DevTools Feature Request: Show Stack Trace In Event Listeners

Thursday, June 21st, 2012

I was asked a question in my DevTools talk at ComoRichWeb this month.

Q: How do the event listeners work when the events are bound with jQuery?

A: Not well. I rarely use that section when using a framework to bind events, because the event listeners never point back to the actual function being bound – instead they link to the jQuery source.

The info you see on event listeners bound with jQuery is not very useful. It shows you the actual function where the events were bound (which is inside of the jQuery source).

Open up this simple event listener test case in devtools (inspect the button and view event listeners) to see what I mean.

What Information Do You Want From Event Listeners?

What I am actually interested in is the actual callback that jQuery is adding. Of course devtools shouldn’t have to handle jQuery (or other frameworks) specifically, but what if it showed the callstack that caused the event to be bound?

So, by setting a breakpoint to the actual event event binding, here is what I got. This is what I am actually interested in:

Solution?

And putting my photo editing skills to the test, here is what it could look like (of course, this should be styled consistently). Clicking on one of the entries in the callstack would take you into the relevant place in the scripts panel.

This would be ideal. Is this even possible? Or there is an existing or better way to get around this that I don’t know about?

DevTools Talk – Slides and Links

Thursday, June 21st, 2012

I presented on using Chrome Developer Tools at ComoRichWeb, a web developer user group in Columbia, MO on June 20th. Here are the slides and some other links:

Slides and Demo

Other Links Mentioned in the Talk

We had a good turnout last night, was glad to get a chance to share some front-end development tips with local developers.

FileReaderSync

Friday, June 1st, 2012

I have been working on updating the FileReader.js JavaScript file reading library lately to help out with a few different projects. This library has provided a quick way to get started with reading files on a few different projects, like mothereffinganimatedgif.com and Instant Sprite.

One thing I noticed was that there is a FileReaderSync API, meant for loading files synchronously.

You might wonder why on earth would you want to load a file synchronously in your browser – that seems like it could block the entire UI until the file is loaded! It turns out you can’t, at least not in the normal window context. FileReaderSync only exists inside of the context of a WebWorker:

Implementation

View a A working JS Fiddle using FileReaderSync.

I also wrote about how to load web workers without a JavaScript file, but this technique works just fine using a normal external reference.

markup

<input type='file' id='files' multiple onchange='handleFileSelect()' />

page javascript

function processFiles(files, cb) {
    var syncWorker = new Worker('worker.js');
    syncWorker.onmessage = function(e) {
        cb(e.data.result);
    };
 
    Array.prototype.forEach.call(files, function(file) {
        syncWorker.postMessage(file);
    });
}
 
function handleFileSelect() {
    var files = document.getElementById('files').files;
    processFiles(files, function(src) {
        var img = new Image();
        img.src = src;
        document.body.appendChild(img);
    });
}

worker.js

self.addEventListener('message', function(e) { 
    var data=e.data; 
    try { 
        var reader = new FileReaderSync(); 
        postMessage({ 
            result: reader.readAsDataURL(data)
        });
   } catch(e){ 
        postMessage({ 
            result:'error'
        }); 
   } 
}, false);

The jsFiddle demo is a little more complicated than this, since it handles checking for support and an inline worker.

Gotchas

Something that was a little weird is that since you can’t detect support from the main window, I need to spawn off a worker to post the message of whether it supports FileReaderSync. See a jsFiddle to detect FileReaderSync support. There may be a better way to do this, but I don’t know of it.

This can be pretty complicated, but I have been tying it all into the filereader.js plugin, to make reading with FileReaderSync just an option along with the standard FileReader

Performance

It’s hard for me to accurately measure the performance. On one hand, the FileReaderSync seems to load the images in a slower time per image (measured in milliseconds). I assume that this is due to the overhead and message passing with the worker, or possibly because it is a newer implementation.

However, on large images and videos, it definitely feels like the UI does not lock up as much when processing the files.

Purpose

I feel like maybe part of the point of this API is when you want to some heavy lifting with the file after it is loaded but still inside the worker, which isn’t currently supported in FileReader.js. I could imagine ways this use case could be supported though (maybe by passing in a process() function as a string that the worker could call?

Check out the FileReader.js demo and see if you can tell a difference! I’d love to get any kinks worked out and get some feedback – I have been thinking of setting up a js-file-boilerplate project on Github to tie together a bunch of this functionality in a sample project.

Load Web Workers Without A JavaScript File

Friday, May 25th, 2012

Ever want to load a JavaScript Web Worker without specifying an external JavaScript file? There are a couple of different ways to do this by creating a blob and object URL – I wrapped this functionality up into a little plugin to share.

Here is the code. Or checkout out a working jsfiddle demo

// makeWorker is a little wrapper for generating web workers from strings
function makeWorker(script) {
    var URL = window.URL || window.webkitURL;
    var Blob = window.Blob;
    var Worker = window.Worker;
 
    if (!URL || !Blob || !Worker || !script) {
        return null;
    }
 
    var blob = new Blob([script]);
    var worker = new Worker(URL.createObjectURL(blob));
    return worker;
}
 
<div id='log'></div>
 
<script type='text/worker' id='worker-script'>
    self.addEventListener('message', function(e) { 
        postMessage(e.data / 2); 
    },false);
</script>‚Äč
 
<script type='text/javascript'>
 
// Load a worker from a string, and manually initialize the worker
var inlineWorkerText = 
    "self.addEventListener('message', function(e) { postMessage(e.data * 2); } ,false);"
;
var inlineWorker = makeWorker(inlineWorkerText);
inlineWorker.onmessage = function(e) {
    document.getElementById('log').innerHTML += '<br />Inline: ' + e.data;
};
 
 
// Load a worker from a script of type=text/worker, and use the getWorker helper
var scriptTagWorker = makeWorker(
    document.getElementById('worker-script').textContent
);
 
scriptTagWorker.onmessage = function(e) {
        document.getElementById('log').innerHTML += '<br />Script Tag: ' + e.data;
};
 
inlineWorker.postMessage(1);
inlineWorker.postMessage(2);
inlineWorker.postMessage(100);
scriptTagWorker.postMessage(1);
scriptTagWorker.postMessage(2);
scriptTagWorker.postMessage(100);
 
</script>