Posts Tagged ‘javascript’

Video Funhouse – Convert Videos In Your Web Browser

Monday, November 11th, 2013

We (Nicole, Aaron, and I) have had a lot of fun doing Node Knockout the past few years. Last year (in 2012), our Missouri-based team got first place in the ‘Completeness’ category for our app asciigram, which converted a webcam stream into ASCII art.

This year we decided to up the game a little bit, and to try and convert any video file into another video format, while allowing filters to be applied to the video – all inside of the browser, without uploading anything. The result is Video Funhouse (here is our devcomo team page). I’m particularly proud of our project and team this year, since it was a very busy weekend all around.

Obviously, we knew we wouldn’t be able to do this in a weekend on our own without relying on great libraries and tools. There is a project called Emscripten, which is an LLVM to JavaScript compiler, so we figured we could try building FFmpeg or avconv into JavaScript to do the heavy lifting for the project. Here’s a demo video of the result (keep reading further to see more technical information):



Again, here is the link to the application if you’d like to play with it, and here is a link to the team page.

videoconverter.js

I have open sourced the build utilities and the ffmpeg.js file that we are using in the project as: videoconverter.js.

Building these projects with Emscripten was a little tricky, but actually quite simple when you consider what it was doing for us. What is also amazing about these tools is that I have little experience with build systems like this, and the tools generally just worked after a bunch of trial and error.

Why Would You Compile FFmpeg Into JavaScript?

Mostly just to see if it would work. Also, it seemed like it would be a fun project.

So, How Do You Compile FFmpeg Into JavaScript?

Here are some of notes I made while doing this, for future reference:

Compiling avconv seemed more promising at first, but when running it we bumped into some weird runtime errors with swscaler. Couldn’t figure it out quickly, so focused on FFmpeg. We pulled down the latest Emscripten, and the latest FFmpeg:

git clone git@github.com:kripken/emscripten.git
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

You may need to also get the SDK to make sure Emscripten will work. The have documentation on their site about getting this to work. Here are the basic configuration options we ended up using. Most of the configuration options are actually disabling features that either don’t make sense, or are there to save space. After a lot of trial and error, here is the minimal amount of flags to pass into emconfigure: --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic.

The first time we got a successful build it was a 100MB JS file. By removing a bunch of the other stuff, we get it down to 25MB. It could actually get smaller as well – we were unable to minify the script because the closure compiler process ran out of memory. The following commands are updated after we figured out some better options. The demo as it is right now doesn’t run any optimizations on the JS file, and it is about double the size. We can’t update the demo, as the weekend is over, but this commit is promising: ffmpeg.js: 190 additions, 1463201 deletions not shown.

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib
 
emmake make;
 
emcc -O2 -s VERBOSE=1 -s ASM_JS=0 -s ALLOW_MEMORY_GROWTH=1 -v libavutil/*.o libavcodec/*.o libavformat/*.o libavdevice/*.o libswresample/*.o libavfilter/*.o libswscale/*.o *.o -o ../ffmpeg.js --pre-js ../ffmpeg_pre.js --post-js ../ffmpeg_post.js

Peformance

The FFmpeg process runs inside of a web worker to prevent locking up the UI. This was the first optimization, as it was necessary for doing any testing on the project at all. Data is piped back from stdout and stderr to the frontend of the demo app, which is listening for these and logging them to a pre tag. This logging could be limited or removed to limit the impact on the DOM, as a lot of data can be generated by the system.

We experimented with firing up 2 different workers, one for the metadata probe and actual processing, and one for taking screenshots of a video right after receiving it. But we noticed some general instability when both were running at once (many times a tab or browser would crash), so we removed the screenshotting functionality. I’m sure we could have traced it down and improved the performance if we had more than a weekend for the project.

The processing itself is a little slow, but I haven’t done benchmarks to compare with native. I generally avoided processing videos of much size. Also, our demo app prints a lot of logs which may not be helping matters. We are looking into how to set up performance tests, but if anyone wants to help with this please let me know or submit a PR on the videoconverter.js project.

I believe this could be quite a bit faster – it couldn’t actually use asm.js because it is explicitly disabled when ALLOW_MEMORY_GROWTH is defined. It would be worth experimenting to see if we could set a large max memory size and enable asm to see what kind of speedup we saw.

I’m interested to see if we can get some sort of performance benchmarking set up to compare how fast it runs in different JavaScript environments, and to see how it compares with native performance.

Potential Uses

  • This could be extended to bundle other encoding libraries. I wasn’t able to figure out how to get the linking to work for libvpx, x264, libtheora, zlib, etc over the weekend, so certain formats cannot be encoded. It would be neat to have this functionality, as it would allow conversion into web friendly video, which could then allow further previewing and processing. As performance improves and FileSystem API support improve it may be possible to build a web-based video or audio editing tool using this.
  • Compile the other utilities – there are other programs that ship with FFmpeg that we are excluding right now. ffprobe in particular seems like it could be better for gathering the metadata after the initial load of the file.
  • As mentioned previously, this could be useful as a benchmarking tool.
  • Probably other things I haven’t thought of as well.

Firefox DevTools Experiment – Pixel Inspector

Monday, September 2nd, 2013

Heather and I spent some of our @FirefoxDevTools team meetup working together on an experiment we named the Pixel Inspector. The idea is to focus on an individual region of the page with an eyedropper to pick colors and a zoom feature to debug tricky alignment issues. You can get separate programs that do some of this, such as Digital Color Meter, or addons like ColorZilla, but it would be great to have something built into the developer tools to do this. Here is how it works:

  • Open the panel with a keyboard shortcut or by clicking a button in DevTools.
  • Move your mouse around the page to the location you want to inspect. Click on the pixel to lock it in place.
  • Use the zoom slider to increase the size of each individual pixel.
  • Press the ‘Copy’ button to copy the color to your clipboard, or use the format dropdown to change the color format.

See the screencast and screenshot below to check it out. Would this be a useful feature to add to DevTools?





pixel inspector

How does it work?

Pixel Inspector makes use of the drawWindow canvas method (which is only available with “chrome only”, meaning a script running with elevated privileges that has access to browser internals). There is an additional flag, named context.mozImageSmoothingEnabled = false; that allows the image to get the pixelated effect we want. Most of the code can be seen in this file: https://github.com/harthur/devtools/blob/magnifier/browser/devtools/magnifier/magnifier.js.

DevTools Links

Monday, July 8th, 2013

Links I’ve collected while working with and contributing to various browser developer tools, all in one place. If you are wanting to contribute to any of these tools, get in contact with the teams that build them, or just want to browse the code (they are all open source), here are some links to help you get started.

Thanks to Heather Arthur, Paul Irish, Timothy Hatcher and Mike Ratcliffe for contributing to this guide. Please let me know if you have any updated or additional information to add to this list by leaving a comment or pinging @bgrins on Twitter.


Chrome DevTools

Getting ahold of Chrome DevTools team


Firefox DevTools

Getting ahold of Firefox DevTools team


Safari Web Inspector

Getting ahold of Safari Web Inspector team


Devtools Snippets

Sunday, June 16th, 2013

I have been working on a small collection of snippets that are handy to reuse inside of browser devtools. These can be used in any devtools. You can jump straight to the devtools snippets project page and readme if you want to see some screenshots and details about each individual snippet.

The actual snippets can be seen here: https://github.com/bgrins/devtools-snippets/tree/master/snippets

Why would you use snippets

A common usage for snippets is to replace developer bookmarklet type of functionality. The problem with bookmarklets is that they are a pain to create and share, and even using them can be difficult. You may need to show a bookmarks bar in the UI, run it, and open up devtools to see if it worked. Then you would need to make changes to a giant single line of JavaScript inside of a bookmark manager, or use a generator to convert a .js file into a bookmarklet. Snippets are a much better way to handle this workflow. Examples of this type of snippet would be adding jQuery to a page without it or printing the HTTP headers from the current page.

Another use would be saving console commands that you use during development. For example, if you wrote something to help export information from a page, or print out some debugging data for your application, you can use the multiline snippet editing interface rather than typing in the console. Plus you can save it when you are done in case you need it later.

Browser Support

Firefox has multiline support using the scratchpad editor, and Chrome has a new ‘snippets’ feature.

Currently, in Chrome you have to enable snippets support in chrome://flags and inside of devtools settings (see full instructions and screenshots). They are enabled by default in Canary, so this feature set will soon be widely available. Read more about snippets on the Chrome developer tools documentation.

Firefox Scratchpad is really cool for experimenting with code. In addition to being able to run an entire snippet, you can select just a portion of the code and execute it. Read more about scratchpad on the Firefox developer tools documentation. One thing I would like to see is better management of snippets (‘scratches’?). For example, see the UI for snippet editing in Chrome:

Future Functionality

Things that would be awesome to add:

  • Firefox: Better management of snippets. As seen in the screenshot above, it can be really handy to have a UI for storing, editing, and saving changes to the snippets. To be fair, right now you can save scratchpad contents to a file and then ‘open’ or ‘open recent’ later. Maybe if this was pulled more prominently into the UI without needing to open up a file explorer that be enough. It is cool that it actually saves the files to a known place on your disk.
  • Chrome: Keyboard shortcut to run a snippet (or part of a snippet). Scratchpad uses Cmd+R/Ctrl+R, which makes sense but may be reserved for reloading the page. Also, storing the snippets in a known place on the disk is really handy, as they could be backed up, copied directly into a project, etc.
  • All devtools: Import / export snippets. You can then sync your snippets across computers or share them with others. Maybe some sort of zip or JSON file containing the contents of all the snippets. Or a maybe a URL importer that loads the response into a new snippet, named the path of the request by default. For example, pasting https://raw.github.com/bgrins/devtools-snippets/master/snippets/performance.js would create a new snippet called performance.js with the contents of that request. Then you could just load up the importer with a list of URLs:
     
    https://raw.github.com/bgrins/devtools-snippets/master/snippets/performance.js
    https://raw.github.com/bgrins/devtools-snippets/master/snippets/log.js
    https://raw.github.com/bgrins/devtools-snippets/master/snippets/base64.js
    ...

Snippets inside of devtools are big timesavers and are going to be really helpful for a lot of developers. You should check them out if you haven’t yet. Going forward, I’m hoping there will be an easy way to use, manage, and discover new snippets that work across browsers.

Devtools Colorpicker Palette

Saturday, February 2nd, 2013

I’m still trying to figure out how to get a list of all the colors in use on a page from within devtools, but I’ve been hacking together a little demo with a custom Web Inspector frontend that loads a color palette on the side.

When you click one of the swatches, it sets the current color of the element. The idea is that it would include all the colors in your page, maybe ordered by most used. Possibly could even get more advanced and suggest a set of complementary or analagous colors based on your current selection.

A prototype implementation of a palette using color swatches within devtools

A prototype implementation of a palette using color swatches within devtools

jQuery UI 1.9 Boilerplate

Wednesday, October 17th, 2012

jsbin has made having to remember some of these links less important. However, with jQuery UI 1.9 coming out, there are some CDN links that aren’t working. Here is the updated boilerplate I use to quickly test out a jQuery UI project. If all goes well, you should have a dialog that you can drag around and resize.

<!doctype html>
<html>
<head>
  <title></title>
 
  <link rel="stylesheet" href="http://code.jquery.com/ui/1.9.0/themes/base/jquery-ui.css">
 
  <script type="text/javascript" src="http://code.jquery.com/jquery-1.8.2.js"></script>
  <script src="http://ajax.aspnetcdn.com/ajax/jquery.ui/1.9.0/jquery-ui.js" type="text/javascript"></script>
 
  <style type="text/css">
    #modal {
	 background: orange;
    }
  </style>
 
  <script type='text/javascript'>
    $(function() {
        $("#modal").dialog({ title: "It works" })
    });
  </script>
</head>
 
<body>
<div id='modal'>jQuery UI!</div>
</body>
 
</html>

As a bonus, here is the markup in jsbin.

Devtools Colorpicker

Monday, July 2nd, 2012

It is great that so many important platforms that we use on the web are open source, and open to contributions from anyone.

It took a lot of work, but it was worth it. I’ve been doing some web design work recently, and love how much easier it is to play with different colors in a design.

Now I’m thinking of different ways that design work could be further improved with the colorpicker. A couple of ideas right now: (a) an eyedropper to select a color from the screen and (b) a color palette consisting of currently used and recommended colors.

DevTools Talk – Slides and Links

Thursday, June 21st, 2012

I presented on using Chrome Developer Tools at ComoRichWeb, a web developer user group in Columbia, MO on June 20th. Here are the slides and some other links:

Slides and Demo

Other Links Mentioned in the Talk

We had a good turnout last night, was glad to get a chance to share some front-end development tips with local developers.

FileReaderSync

Friday, June 1st, 2012

I have been working on updating the FileReader.js JavaScript file reading library lately to help out with a few different projects. This library has provided a quick way to get started with reading files on a few different projects, like mothereffinganimatedgif.com and Instant Sprite.

One thing I noticed was that there is a FileReaderSync API, meant for loading files synchronously.

You might wonder why on earth would you want to load a file synchronously in your browser – that seems like it could block the entire UI until the file is loaded! It turns out you can’t, at least not in the normal window context. FileReaderSync only exists inside of the context of a WebWorker:

Implementation

View a A working JS Fiddle using FileReaderSync.

I also wrote about how to load web workers without a JavaScript file, but this technique works just fine using a normal external reference.

markup

<input type='file' id='files' multiple onchange='handleFileSelect()' />

page javascript

function processFiles(files, cb) {
    var syncWorker = new Worker('worker.js');
    syncWorker.onmessage = function(e) {
        cb(e.data.result);
    };
 
    Array.prototype.forEach.call(files, function(file) {
        syncWorker.postMessage(file);
    });
}
 
function handleFileSelect() {
    var files = document.getElementById('files').files;
    processFiles(files, function(src) {
        var img = new Image();
        img.src = src;
        document.body.appendChild(img);
    });
}

worker.js

self.addEventListener('message', function(e) { 
    var data=e.data; 
    try { 
        var reader = new FileReaderSync(); 
        postMessage({ 
            result: reader.readAsDataURL(data)
        });
   } catch(e){ 
        postMessage({ 
            result:'error'
        }); 
   } 
}, false);

The jsFiddle demo is a little more complicated than this, since it handles checking for support and an inline worker.

Gotchas

Something that was a little weird is that since you can’t detect support from the main window, I need to spawn off a worker to post the message of whether it supports FileReaderSync. See a jsFiddle to detect FileReaderSync support. There may be a better way to do this, but I don’t know of it.

This can be pretty complicated, but I have been tying it all into the filereader.js plugin, to make reading with FileReaderSync just an option along with the standard FileReader

Performance

It’s hard for me to accurately measure the performance. On one hand, the FileReaderSync seems to load the images in a slower time per image (measured in milliseconds). I assume that this is due to the overhead and message passing with the worker, or possibly because it is a newer implementation.

However, on large images and videos, it definitely feels like the UI does not lock up as much when processing the files.

Purpose

I feel like maybe part of the point of this API is when you want to some heavy lifting with the file after it is loaded but still inside the worker, which isn’t currently supported in FileReader.js. I could imagine ways this use case could be supported though (maybe by passing in a process() function as a string that the worker could call?

Check out the FileReader.js demo and see if you can tell a difference! I’d love to get any kinks worked out and get some feedback – I have been thinking of setting up a js-file-boilerplate project on Github to tie together a bunch of this functionality in a sample project.

Load Web Workers Without A JavaScript File

Friday, May 25th, 2012

Ever want to load a JavaScript Web Worker without specifying an external JavaScript file? There are a couple of different ways to do this by creating a blob and object URL – I wrapped this functionality up into a little plugin to share.

Here is the code. Or checkout out a working jsfiddle demo

// makeWorker is a little wrapper for generating web workers from strings
function makeWorker(script) {
    var URL = window.URL || window.webkitURL;
    var Blob = window.Blob;
    var Worker = window.Worker;
 
    if (!URL || !Blob || !Worker || !script) {
        return null;
    }
 
    var blob = new Blob([script]);
    var worker = new Worker(URL.createObjectURL(blob));
    return worker;
}
 
<div id='log'></div>
 
<script type='text/worker' id='worker-script'>
    self.addEventListener('message', function(e) { 
        postMessage(e.data / 2); 
    },false);
</script>‚Äč
 
<script type='text/javascript'>
 
// Load a worker from a string, and manually initialize the worker
var inlineWorkerText = 
    "self.addEventListener('message', function(e) { postMessage(e.data * 2); } ,false);"
;
var inlineWorker = makeWorker(inlineWorkerText);
inlineWorker.onmessage = function(e) {
    document.getElementById('log').innerHTML += '<br />Inline: ' + e.data;
};
 
 
// Load a worker from a script of type=text/worker, and use the getWorker helper
var scriptTagWorker = makeWorker(
    document.getElementById('worker-script').textContent
);
 
scriptTagWorker.onmessage = function(e) {
        document.getElementById('log').innerHTML += '<br />Script Tag: ' + e.data;
};
 
inlineWorker.postMessage(1);
inlineWorker.postMessage(2);
inlineWorker.postMessage(100);
scriptTagWorker.postMessage(1);
scriptTagWorker.postMessage(2);
scriptTagWorker.postMessage(100);
 
</script>