Posts Tagged ‘programming’

FileReaderSync

Friday, June 1st, 2012

I have been working on updating the FileReader.js JavaScript file reading library lately to help out with a few different projects. This library has provided a quick way to get started with reading files on a few different projects, like mothereffinganimatedgif.com and Instant Sprite.

One thing I noticed was that there is a FileReaderSync API, meant for loading files synchronously.

You might wonder why on earth would you want to load a file synchronously in your browser – that seems like it could block the entire UI until the file is loaded! It turns out you can’t, at least not in the normal window context. FileReaderSync only exists inside of the context of a WebWorker:

Implementation

View a A working JS Fiddle using FileReaderSync.

I also wrote about how to load web workers without a JavaScript file, but this technique works just fine using a normal external reference.

markup

<input type='file' id='files' multiple onchange='handleFileSelect()' />

page javascript

function processFiles(files, cb) {
    var syncWorker = new Worker('worker.js');
    syncWorker.onmessage = function(e) {
        cb(e.data.result);
    };
 
    Array.prototype.forEach.call(files, function(file) {
        syncWorker.postMessage(file);
    });
}
 
function handleFileSelect() {
    var files = document.getElementById('files').files;
    processFiles(files, function(src) {
        var img = new Image();
        img.src = src;
        document.body.appendChild(img);
    });
}

worker.js

self.addEventListener('message', function(e) { 
    var data=e.data; 
    try { 
        var reader = new FileReaderSync(); 
        postMessage({ 
            result: reader.readAsDataURL(data)
        });
   } catch(e){ 
        postMessage({ 
            result:'error'
        }); 
   } 
}, false);

The jsFiddle demo is a little more complicated than this, since it handles checking for support and an inline worker.

Gotchas

Something that was a little weird is that since you can’t detect support from the main window, I need to spawn off a worker to post the message of whether it supports FileReaderSync. See a jsFiddle to detect FileReaderSync support. There may be a better way to do this, but I don’t know of it.

This can be pretty complicated, but I have been tying it all into the filereader.js plugin, to make reading with FileReaderSync just an option along with the standard FileReader

Performance

It’s hard for me to accurately measure the performance. On one hand, the FileReaderSync seems to load the images in a slower time per image (measured in milliseconds). I assume that this is due to the overhead and message passing with the worker, or possibly because it is a newer implementation.

However, on large images and videos, it definitely feels like the UI does not lock up as much when processing the files.

Purpose

I feel like maybe part of the point of this API is when you want to some heavy lifting with the file after it is loaded but still inside the worker, which isn’t currently supported in FileReader.js. I could imagine ways this use case could be supported though (maybe by passing in a process() function as a string that the worker could call?

Check out the FileReader.js demo and see if you can tell a difference! I’d love to get any kinks worked out and get some feedback – I have been thinking of setting up a js-file-boilerplate project on Github to tie together a bunch of this functionality in a sample project.

Colorwheel 1K

Sunday, February 12th, 2012

I decided to make an HSV colorpicker for the js1k contest. Here is a demo of the colorpicker and the source code.

Getting Started

I found a great start for actually drawing the wheel at the canvas color wheel post from Ariya Hidayat. I modified that code and got the wheel on the screen in no time.

Also, I had written TinyColor, a JavaScript color parsing library, so the HSV to RGB conversion function was pretty easy to grab (though I did have to make some changes to shrink the function size, more on that later). This let me remove some of the code from the color wheel function that I also needed on mouse move events.

I tied together some buggybasic bounds checking on the circle for mousemove events and printed out the current RGB using the conversion function above, and I had a cool little demo – weighing in at around 2K.

Smaller… Much Smaller

So I needed to drop about half the weight of the project. I learned that you could use ~~x instead of Math.floor(x) provided that the number is positive, so that helped. And there are always tricks about passing in assignments to functions to save a semicolon, storing references to global objects, commonly used math functions, and so on. But it was not quite the time for dropping individual bytes, I needed some big changes.

First, DOM and CSS rules take up a lot of bytes. b.appendChild(d.createElement("input")) is just huge and cannot really be minified beyond what is there. My takeaway was that I needed less DOM elements. I was originally using a container div with a width equal to that of the canvas (400px) and had a little ‘spot’ div that was absolutely positioned based on the current X and current Y value. All of these lines are problems when are you trying to squeeze an ‘application’ into 1k:

container = b.appendChild(doc.createElement("div"));
spot = b.appendChild(doc.createElement("b"));
container.style.cssText='position:absolute;width:400px;margin:auto'
spot.style.cssText='position:relative;height:5px;width:5px'

By moving this functionality into the canvas drawing method (with fillRect, and later fillText), I dropped a couple hundred more bytes.

Next, the hsvToRgb conversion function got an overhaul (it went from 291 bytes to 205, and 33 of those are for the CSS color rule needed for the RGB background color). Takeaway here is that case statements have a lot of control characters, and this particular one was nicely shrunk down using 3 arrays.

 
// 291 bytes <a href='http://closure-compiler.appspot.com/code/jsc1c76aaeacd4bc7fcf6686709cc174fc3/default.js'>minified</a>.
function hsvToRgbBIG(h, s, v) {
    var r, g, b;
 
    var i = math.floor(h * 6);
    var f = h * 6 - i;
    var p = v * (1 - s);
    var q = v * (1 - f * s);
    var t = v * (1 - (1 - f) * s);
 
    switch(i % 6) {
        case 0: r = v, g = t, b = p; break;
        case 1: r = q, g = v, b = p; break;
        case 2: r = p, g = v, b = t; break;
        case 3: r = p, g = q, b = v; break;
        case 4: r = t, g = p, b = v; break;
        case 5: r = v, g = p, b = q; break;
    }
 
    return { r: r * 255, g: g * 255, b: b * 255 };
}
 
// 205 bytes <a href='http://closure-compiler.appspot.com/code/jsc1c76aaeacd4bc7fcf6686709cc174fc3/default.js'>minified</a>.
// Also includes the css color rule as the final return value.
function hsvToRgbSMALL(h, s, v) {
    h*=6;
    var i = ~~h,
        f = h - i,
        p = v * (1 - s),
        q = v * (1 - f * s),
        t = v * (1 - (1 - f) * s),
        mod = i % 6,
        r = [v, q, p, p, t, v][mod] * two55,
        g = [t, v, v, q, p, p][mod] * two55,
        b = [p, p, t, v, v, q][mod] * two55;
 
    return [r, g, b, "rgb("+ ~~r + "," + ~~g + "," + ~~b + ")"];
}

From there, it was mostly reorganizing and adding hacks to make the code smaller. A tip for anyone trying to do this: I wrap all the code into a executing function, so that minifiers will use one letter names for variables (rather than assuming they are global). Then I run the code through Closure Compiler online frequently when making changes to see how effective changes are. Don’t forget to remove the wrapper from the minified file, though!

(function() {
    var oneHundred = 100;
    // All code goes inside here
})();

1k

I finally got to 1024 bytes! It was a fun project, I will probably to it again when another js1k comes along.

A* Search Algorithm in JavaScript (Updated)

Wednesday, September 15th, 2010

See the updated pathfinding demo of A* Search in JavaScript.
Get all the updated source code from github.

When I first wrote the A* Search in JavaScript article I knew there were some things that could make the pathfinding faster. I didn’t know realize that it would be this much faster. Here are my (totally unscientific) results:

Observed Speed of A* Search on Chrome 6.0.472.55 on OS X
# rows original updated
100 700-2500ms (wildly variant) 5-10ms
50 160ms 5-10ms
15 0-5ms 0-2ms

Here’s What Changed

I got some feedback about using a heap or a priority queue instead of a list (basically keep the open items in sorted order the whole time so we can just grab it off of the top instead of needing to traverse the whole list every time. As predicted, this makes a huge difference as the number of nodes goes up, but may actually be a little slower for a very low number of nodes because of the overhead of keeping them sorted. I found a binary heap implementation from http://eloquentjavascript.net/appendix2.html and put it in.

Also, I realized two places that I was traversing lists that could be avoided. Getting rid of extra traversals (especially ones that ran inside nested loops) made a very big difference in performance.

  1. I was keeping all of the nodes inside a closed list, which required traversing that list every time to see if it was already closed. This was replaced with a simple bit on the node to track if it had been closed.
  2. I was checking to see if a node was visited already by seeing if it was in the open list. This was replaced by a simple bit to see if it had been visited.

Old Pseudocode

  push startNode onto openList
  while(openList is not empty) {
     currentNode = find lowest f in openList
     if currentNode is final, return the successful path
     push currentNode onto closedList and remove from openList
     foreach neighbor of currentNode {
         if neighbor is not in openList {
                save g, h, and f then save the current parent
                add neighbor to openList
         }
         if neighbor is in openList but the current g is better than previous g {
                 save g and f, then save the current parent
         }
     }

Updated Pseudocode (* lines have changed)

  * push startNode onto openHeap
  while(openList is not empty) {
     * currentNode = pop from openHeap
     if currentNode is final, return the successful path
     * set currentNode as closed
     foreach neighbor of currentNode {
         * if neighbor is not set visited {
                * save g, h, and f then save the current parent and set visited
                * add neighbor to openHeap
         }
         if neighbor is in openList but the current g is better than previous g {
                 save g and f, then save the current parent
                 * reset position in openHeap (since f changed) 
         }
     }

I also did some general cleanup, getting rid of dependancies on the demo, and I split out the code so that the search function can be used standalone (taking a two dimensional array as a parameter), so if someone just wanted the optimized astar search, you could call it like so:

<script type='text/javascript' src='graph.js'></script>
<script type='text/javascript' src='astar.js'></script>
<script type='text/javascript'>
var graph = new Graph([
	[0,0,0,0], // 0 is open, 1 is wall
	[1,0,0,1],
	[1,1,0,0]
]);
var start = graph.nodes[0][0];
var end = graph.nodes[1][2];
var result = astar.search(graph.nodes, start, end);
// result is an array containing the shortest path
</script>

What Could Make It Better?

Last time I posted about this, I got a lot of feedback (mostly about how slow it was). If anyone reading this knows of anything that could make it better, let me know.

Source Code (direct links)

astar.js |
graph.js |
demo.js |
astar-list.js (old implementation)

Make Table Rows Sortable Using jQuery UI Sortable

Thursday, July 30th, 2009

I wrote an article, Make Table Rows Sortable Using jQuery UI Sortable on the Foliotek Development Blog about a problem that I ran into when trying to set up a basic sortable table using jQuery UI. The columns would collapse down once the dragging began.

sortable-row-collapsed

This was fixed by adjusting the helper object for the sortable function. Check out the [article][] for all the details, but here is a sample of the code if you are just looking for the solution:

// Return a helper with preserved width of cells
var fixHelper = function(e, ui) {
	ui.children().each(function() {
		$(this).width($(this).width());
	});
	return ui;
};
 
$("#sort tbody").sortable({
	helper: fixHelper
}).disableSelection();

C# Tips – Type Conversion with “as” and “is”

Tuesday, July 21st, 2009

I had used C# for at least a year before I found out a couple of nice shorthand ways to convert types. These are the is and as expressions, and using them can help make your code more readable.

The is Expression

The is expression will return true if the provided expression is non-null and can be cast to the specific type.

private string GetDisplayField(object val)
{
	if (val is string)
	{
		string text = (string)val;
		return "Value was text: " + text;
	}
	else if (val is DateTime)
	{
		DateTime time = (DateTime)val;
		return "Value was date: " + time.ToShortDateString();
	}
 
	return "Could not match type";
}

The as Expression

The as expression will try to cast the object into the given type, and returns an object of that type if the cast was successful, or return null if unsuccessful.

This buys a little bit more functionality for you, as it assigns the variable with an already casted value if successful. It is functionally equivalent to: expression is type ? (type)expression : (type)null

private string GetDisplayField(object val)
{
	string text = val as string;
	DateTime? time = val as DateTime?;
 
	if (text != null)
	{
		return "Value was text: " + text;
	}
	if (time.HasValue)
	{
		return "Value was date: " + time.Value.ToShortDateString();
	}
 
	return "Could not match type";
}

Testing out the functions

Console.WriteLine(GetDisplayField(1));              // Output: "Could not match type"
Console.WriteLine(GetDisplayField("Hello"));        // Output: "Value was text: Hello"
Console.WriteLine(GetDisplayField(DateTime.Now));   // Output: "Value was date: 7/21/2009"

They are both readable ways to perform a type conversion – but pick one or the other! Using both of them is redundant.

jQuery outerHTML Snippet

Tuesday, June 16th, 2009

outerHTML is a property that is provided by Internet Explorer that returns the full HTML of an element (including start and end tags). In jQuery, the html() function returns the innerHTML of an element, which is just the HTML inside the element (not including the start and end tags).

There came a time that I wanted to get the outerHTML of an element, and I found that Brandon Aaron shared a jQuery code snippet that does this exactly. It does the trick for most cases, but there was one problem that I ran into. I wanted to get the outerHTML of an element inside of an iframe, and I got a ‘Permission Denied’ error in Internet Explorer.

The problem was that it was appending an element belonging to the iframes ‘contentDocument’ into an element belonging to the global ‘document’ element.
Using the jQuery(html, ownerDocument) overload of the jQuery core function, this error was fixed:

$.fn.outerHTML = function() {
    var doc = this[0] ? this[0].ownerDocument : document;
    return $('<div>', doc).append(this.eq(0).clone()).html();
};

A* Search Algorithm in JavaScript

Wednesday, June 3rd, 2009

Note that this code has been updated. I have an updated blog post detailing what changed. The full source code is available at https://github.com/bgrins/javascript-astar

View the online demo

I first implemented the A* algorithm for a research group I was in through school (Computer Graphics and Image Understanding). A* is a best-first, graph search algorithm. Some basic information can be found on the Wikipedia page for A* and the external links contained in it. Please refer to that page for general reference about the algorithm, I will simply explain in my own words how it works and how I got it working.

A* algorithm in JavaScript

A* algorithm in JavaScript

Why JavaScript?

Because it was easy to deploy!

Since I know JavaScript pretty well, and most of the examples you can find are in C, Java or a similar language that you cannot run without downloading source code or executables, I thought it would be a good idea to program it on an html page. This way, people could see what was going on and view the source very easily by using the online demo.

My hope was to build a page that could be extended with other search algorithms by separating the UI code (that generates a graph with walls and animates the path that is determined by an algorithm), and the algorithm that finds the path. Maybe I will get around to plugging in some more algorithms sometime and making it into a little resource for graph search algorithms.

How?

search.html

Just a basic html file that includes jQuery, the excellent JavaScript library, main.css, graph.js, and astar.js. Also, I have a JavaScript block that initializes the page.

graph.js

The point of this file is to build the graph, call the search function, and animate the results after the search has returned. It also has an option to show the debugging information created by the search algorithm. I won’t get too into the code here, since it distracts from the search algorithm.

Please take a look at it, be aware that there are some improvements I would make if I was to rewrite this today. There are some performance issues that could be addressed, and It modifies the Array.prototype to add on specific methods (findGraphNode and removeGraphNode) for search algorithms, which may not be ideal for bigger projects. For this little page, I’m not too worried about it, but if I do get around to adding in more algorithms, I will probably improve this code.

astar.js

This is the actual implementation of the algorithm. I will do my best to explain what is going on, but feel free to just look at the source of the example, or just download astar.js.

There are three functions that we keep track of for nodes that we look at:

  • g(x): The total cost of getting to that node (pretty straightforward). If we reach a node for the first time or reach a node in less time than it currently took, then update the g(x) to the cost to reach this node.
  • h(x): The estimated time to reach the finish from the current node. This is also called a heuristic. We online need to update this if it is not set already, since the distance to the finish will not change even if the path we took to arrive at a node changes. Note: There are many different ways to guess how far you are from the end, I use the Manhattan distance in this implementation.
  • f(x): Simply g(x) + h(x). The lower the f(x), the better. Think about it like this: the best node is one that takes the least total amount of time to arrive at and to get to the end. So, a node that took only 1 step to arrive at and 5 to get to the end is more ideal than one that took 10 to arrive and and only 1 to get to the end.

Here is some high level pseudocode of what is happening in the algorithm. Also see the Wikipedia pseudocode for another example.

  push startNode onto openList
  while(openList is not empty) {
     currentNode = find lowest f in openList
     if currentNode is final, return the successful path
     push currentNode onto closedList and remove from openList
     foreach neighbor of currentNode {
         if neighbor is not in openList {
                save g, h, and f then save the current parent
                add neighbor to openList
         }
         if neighbor is in openList but the current g is better than previous g {
                 save g and f, then save the current parent
         }
     }

Here is the JavaScript for the list implementation:

 
var astar = {
	init: function(grid) {
		for(var x = 0; x < grid.length; x++) {
			for(var y = 0; y < grid[x].length; y++) {
				grid[x][y].f = 0;
				grid[x][y].g = 0;
				grid[x][y].h = 0;
				grid[x][y].debug = "";
				grid[x][y].parent = null;
			}	
		}
	},
	search: function(grid, start, end) {
		astar.init(grid);
 
		var openList   = [];
		var closedList = [];
		openList.push(start);
 
		while(openList.length > 0) {
 
			// Grab the lowest f(x) to process next
			var lowInd = 0;
			for(var i=0; i<openList.length; i++) {
				if(openList[i].f < openList[lowInd].f) { lowInd = i; }
			}
			var currentNode = openList[lowInd];
 
			// End case -- result has been found, return the traced path
			if(currentNode.pos == end.pos) {
				var curr = currentNode;
				var ret = [];
				while(curr.parent) {
					ret.push(curr);
					curr = curr.parent;
				}
				return ret.reverse();
			}
 
			// Normal case -- move currentNode from open to closed, process each of its neighbors
			openList.removeGraphNode(currentNode);
			closedList.push(currentNode);
			var neighbors = astar.neighbors(grid, currentNode);
 
			for(var i=0; i<neighbors.length;i++) {
				var neighbor = neighbors[i];
				if(closedList.findGraphNode(neighbor) || neighbor.isWall()) {
					// not a valid node to process, skip to next neighbor
					continue;
				}
 
				// g score is the shortest distance from start to current node, we need to check if
				//	 the path we have arrived at this neighbor is the shortest one we have seen yet
				var gScore = currentNode.g + 1; // 1 is the distance from a node to it's neighbor
				var gScoreIsBest = false;
 
 
				if(!openList.findGraphNode(neighbor)) {
					// This the the first time we have arrived at this node, it must be the best
					// Also, we need to take the h (heuristic) score since we haven't done so yet
 
					gScoreIsBest = true;
					neighbor.h = astar.heuristic(neighbor.pos, end.pos);
					openList.push(neighbor);
				}
				else if(gScore < neighbor.g) {
					// We have already seen the node, but last time it had a worse g (distance from start)
					gScoreIsBest = true;
				}
 
				if(gScoreIsBest) {
					// Found an optimal (so far) path to this node.	 Store info on how we got here and
					//	just how good it really is...
					neighbor.parent = currentNode;
					neighbor.g = gScore;
					neighbor.f = neighbor.g + neighbor.h;
					neighbor.debug = "F: " + neighbor.f + "<br />G: " + neighbor.g + "<br />H: " + neighbor.h;
				}
			}
		}
 
		// No result was found -- empty array signifies failure to find path
		return [];
	},
	heuristic: function(pos0, pos1) {
		// This is the Manhattan distance
		var d1 = Math.abs (pos1.x - pos0.x);
		var d2 = Math.abs (pos1.y - pos0.y);
		return d1 + d2;
	},
	neighbors: function(grid, node) {
		var ret = [];
		var x = node.pos.x;
		var y = node.pos.y;
 
		if(grid[x-1] && grid[x-1][y]) {
			ret.push(grid[x-1][y]);
		}
		if(grid[x+1] && grid[x+1][y]) {
			ret.push(grid[x+1][y]);
		}
		if(grid[x][y-1] && grid[x][y-1]) {
			ret.push(grid[x][y-1]);
		}
		if(grid[x][y+1] && grid[x][y+1]) {
			ret.push(grid[x][y+1]);
		}
		return ret;
	}
};

And here is a faster implementation, using a Binary Heap instead of a list. This is a lot faster and also includes the option to search diagonally – 8 directional movement. Head over to the astar graph search project page to get the latest version of the code!

var astar = {
    init: function(grid) {
        for(var x = 0, xl = grid.length; x < xl; x++) {
            for(var y = 0, yl = grid[x].length; y < yl; y++) {
                var node = grid[x][y];
                node.f = 0;
                node.g = 0;
                node.h = 0;
                node.cost = 1;
                node.visited = false;
                node.closed = false;
                node.parent = null;
            }
        }
    },
    heap: function() {
        return new BinaryHeap(function(node) { 
            return node.f; 
        });
    },
    search: function(grid, start, end, diagonal, heuristic) {
        astar.init(grid);
        heuristic = heuristic || astar.manhattan;
        diagonal = !!diagonal;
 
        var openHeap = astar.heap();
 
        openHeap.push(start);
 
        while(openHeap.size() > 0) {
 
            // Grab the lowest f(x) to process next.  Heap keeps this sorted for us.
            var currentNode = openHeap.pop();
 
            // End case -- result has been found, return the traced path.
            if(currentNode === end) {
                var curr = currentNode;
                var ret = [];
                while(curr.parent) {
                    ret.push(curr);
                    curr = curr.parent;
                }
                return ret.reverse();
            }
 
            // Normal case -- move currentNode from open to closed, process each of its neighbors.
            currentNode.closed = true;
 
            // Find all neighbors for the current node. Optionally find diagonal neighbors as well (false by default).
            var neighbors = astar.neighbors(grid, currentNode, diagonal);
 
            for(var i=0, il = neighbors.length; i < il; i++) {
                var neighbor = neighbors[i];
 
                if(neighbor.closed || neighbor.isWall()) {
                    // Not a valid node to process, skip to next neighbor.
                    continue;
                }
 
                // The g score is the shortest distance from start to current node.
                // We need to check if the path we have arrived at this neighbor is the shortest one we have seen yet.
                var gScore = currentNode.g + neighbor.cost;
                var beenVisited = neighbor.visited;
 
                if(!beenVisited || gScore < neighbor.g) {
 
                    // Found an optimal (so far) path to this node.  Take score for node to see how good it is.
                    neighbor.visited = true;
                    neighbor.parent = currentNode;
                    neighbor.h = neighbor.h || heuristic(neighbor.pos, end.pos);
                    neighbor.g = gScore;
                    neighbor.f = neighbor.g + neighbor.h;
 
                    if (!beenVisited) {
                        // Pushing to heap will put it in proper place based on the 'f' value.
                        openHeap.push(neighbor);
                    }
                    else {
                        // Already seen the node, but since it has been rescored we need to reorder it in the heap
                        openHeap.rescoreElement(neighbor);
                    }
                }
            }
        }
 
        // No result was found - empty array signifies failure to find path.
        return [];
    },
    manhattan: function(pos0, pos1) {
        // See list of heuristics: http://theory.stanford.edu/~amitp/GameProgramming/Heuristics.html
 
        var d1 = Math.abs (pos1.x - pos0.x);
        var d2 = Math.abs (pos1.y - pos0.y);
        return d1 + d2;
    },
    neighbors: function(grid, node, diagonals) {
        var ret = [];
        var x = node.x;
        var y = node.y;
 
        // West
        if(grid[x-1] && grid[x-1][y]) {
            ret.push(grid[x-1][y]);
        }
 
        // East
        if(grid[x+1] && grid[x+1][y]) {
            ret.push(grid[x+1][y]);
        }
 
        // South
        if(grid[x] && grid[x][y-1]) {
            ret.push(grid[x][y-1]);
        }
 
        // North
        if(grid[x] && grid[x][y+1]) {
            ret.push(grid[x][y+1]);
        }
 
        if (diagonals) {
 
            // Southwest
            if(grid[x-1] && grid[x-1][y-1]) {
                ret.push(grid[x-1][y-1]);
            }
 
            // Southeast
            if(grid[x+1] && grid[x+1][y-1]) {
                ret.push(grid[x+1][y-1]);
            }
 
            // Northwest
            if(grid[x-1] && grid[x-1][y+1]) {
                ret.push(grid[x-1][y+1]);
            }
 
            // Northeast
            if(grid[x+1] && grid[x+1][y+1]) {
                ret.push(grid[x+1][y+1]);
            }
 
        }
 
        return ret;
    }
};

Conclusion

This A* search implementation could be used as a component to larger system (like a game – maybe tower defense or puzzle), or just for learning purposes. I have done my best to make the code understandable and to present the concepts in a way that would help someone who has never seen the algorithm before, or someone who is not very familiar with JavaScript.

Feel free to view the demo, or download graph.js and astar.js to mess around with it. Check out the javascript-astar project page on Github for the latest version

Multipart Form Post in C#

Friday, May 8th, 2009

I recently had to access a web API through C Sharp that required a file upload. This is pretty easy if you have an HTML page with a form tag and you want a user to directly upload the file.

<form method="POST" action="http://localhost/" enctype="multipart/form-data">
	File : <input type="file" name="content" size="38" /><br />
	<input type="hidden" name="id" value='fileUpload' />
 </form>

However, this is not always a reasonable path to take. Sometimes you may be wanting to access a file that is already in a system and you don’t want a new upload. If you are accessing an external API, this is probably always the case. Unfortunately, building this post using C# is not quite as straightforward. I first tried using the WebClient UploadFile method, but it didn’t fit my needs because I wanted to upload form values (id, filename, other API specific parameters) in addition to just a file.

So, I needed to roll my own form post. Here is the Multipart Form RFC and the W3C Specification for multipart/form data. After reading these links and searching some forums, here is what I came up with.

Update: This post has gotten a great response from all the readers who have taken the time to comment and contribute. I would like to take this opportunity to promote the best REST Client for .NET, RestSharp. John Sheehan has implemented this technique using code from this post, which can be seen on github (just look for WriteMultipartFormData). He has also done a great job implementing other basic REST operations in a fully tested suite. I would recommend reading the rest of the post to figure out what is going on behind the scenes, but you might consider using RestSharp in a production environment. Thanks for reading!

Note: If anyone is interested in this code in Visual Basic, reader Mike Ferreira converted the code into VB.Net in a comment below.

// Implements multipart/form-data POST in C# http://www.ietf.org/rfc/rfc2388.txt
// http://www.briangrinstead.com/blog/multipart-form-post-in-c
public static class FormUpload
{
    private static readonly Encoding encoding = Encoding.UTF8;
    public static HttpWebResponse MultipartFormDataPost(string postUrl, string userAgent, Dictionary<string, object> postParameters)
    {
        string formDataBoundary = String.Format("----------{0:N}", Guid.NewGuid());
        string contentType = "multipart/form-data; boundary=" + formDataBoundary;
 
        byte[] formData = GetMultipartFormData(postParameters, formDataBoundary);
 
        return PostForm(postUrl, userAgent, contentType, formData);
    }
    private static HttpWebResponse PostForm(string postUrl, string userAgent, string contentType, byte[] formData)
    {
        HttpWebRequest request = WebRequest.Create(postUrl) as HttpWebRequest;
 
        if (request == null)
        {
            throw new NullReferenceException("request is not a http request");
        }
 
        // Set up the request properties.
        request.Method = "POST";
        request.ContentType = contentType;
        request.UserAgent = userAgent;
        request.CookieContainer = new CookieContainer();
        request.ContentLength = formData.Length;
 
        // You could add authentication here as well if needed:
        // request.PreAuthenticate = true;
        // request.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
        // request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(System.Text.Encoding.Default.GetBytes("username" + ":" + "password")));
 
        // Send the form data to the request.
        using (Stream requestStream = request.GetRequestStream())
        {
            requestStream.Write(formData, 0, formData.Length);
            requestStream.Close();
        }
 
        return request.GetResponse() as HttpWebResponse;
    }
 
    private static byte[] GetMultipartFormData(Dictionary<string, object> postParameters, string boundary)
    {
        Stream formDataStream = new System.IO.MemoryStream();
        bool needsCLRF = false;
 
        foreach (var param in postParameters)
        {
            // Thanks to feedback from commenters, add a CRLF to allow multiple parameters to be added.
            // Skip it on the first parameter, add it to subsequent parameters.
            if (needsCLRF)
                formDataStream.Write(encoding.GetBytes("\r\n"), 0, encoding.GetByteCount("\r\n"));
 
            needsCLRF = true;
 
            if (param.Value is FileParameter)
            {
                FileParameter fileToUpload = (FileParameter)param.Value;
 
                // Add just the first part of this param, since we will write the file data directly to the Stream
                string header = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\";\r\nContent-Type: {3}\r\n\r\n",
                    boundary,
                    param.Key,
                    fileToUpload.FileName ?? param.Key,
                    fileToUpload.ContentType ?? "application/octet-stream");
 
                formDataStream.Write(encoding.GetBytes(header), 0, encoding.GetByteCount(header));
 
                // Write the file data directly to the Stream, rather than serializing it to a string.
                formDataStream.Write(fileToUpload.File, 0, fileToUpload.File.Length);
            }
            else
            {
                string postData = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"\r\n\r\n{2}",
                    boundary,
                    param.Key,
                    param.Value);
                formDataStream.Write(encoding.GetBytes(postData), 0, encoding.GetByteCount(postData));
            }
        }
 
        // Add the end of the request.  Start with a newline
        string footer = "\r\n--" + boundary + "--\r\n";
        formDataStream.Write(encoding.GetBytes(footer), 0, encoding.GetByteCount(footer));
 
        // Dump the Stream into a byte[]
        formDataStream.Position = 0;
        byte[] formData = new byte[formDataStream.Length];
        formDataStream.Read(formData, 0, formData.Length);
        formDataStream.Close();
 
        return formData;
    }
 
    public class FileParameter
    {
        public byte[] File { get; set; }
        public string FileName { get; set; }
        public string ContentType { get; set; }
        public FileParameter(byte[] file) : this(file, null) { }
        public FileParameter(byte[] file, string filename) : this(file, filename, null) { }
        public FileParameter(byte[] file, string filename, string contenttype)
        {
            File = file;
            FileName = filename;
            ContentType = contenttype;
        }
    }
}

Here is the code to call the MultipartFormDataPost function with multiple parameters, including a file.

 
// Read file data
FileStream fs = new FileStream("c:\\people.doc", FileMode.Open, FileAccess.Read);
byte[] data = new byte[fs.Length];
fs.Read(data, 0, data.Length);
fs.Close();
 
// Generate post objects
Dictionary<string, object> postParameters = new Dictionary<string, object>();
postParameters.Add("filename", "People.doc");
postParameters.Add("fileformat", "doc");
postParameters.Add("file", new FormUpload.FileParameter(data, "People.doc", "application/msword"));
 
// Create request and receive response
string postURL = "http://localhost";
string userAgent = "Someone";
HttpWebResponse webResponse = FormUpload.MultipartFormDataPost(postURL, userAgent, postParameters);
 
// Process response
StreamReader responseReader = new StreamReader(webResponse.GetResponseStream());
string fullResponse = responseReader.ReadToEnd();
webResponse.Close();
Response.Write(fullResponse);

Hopefully this code can help someone, figuring out exactly where to place the boundary and newlines in between form key-value pairs caused a little bit of grief during development. This is some functionality that would be really nice inside of the language library, but it seems like in most languages this is something you end up coding yourself.

C# Tips – Null Coalescing ??

Tuesday, May 5th, 2009

I have always enjoyed using the logical OR operator in JavaScript (||) as an oppurtunity to check for a null-like value, specifically to provide default parameters for functions or to check for existence of a property on a collection. I think they are a great way to make code more concise, and if used well, more readable.

These two functions do the same thing, but the first is shorter and more readable to people who may have to go back and modify it later.

function contentDocument(frame) {
    // If the frame.contentDocument either doesn't exist or is a null 
    // value, it will skip to the next value down the line 
    return frame.contentDocument || 
        frame.contentWindow.document || 
        frame.document;
}
 
function contentDocumentVerbose(frame) {
    // This works, but is possibly too verbose for such a simple task
    if (frame.contentDocument) {
        return frame.contentDocument;
    }
    else if (frame.contentWindow) {
        return frame.contentWindow.document;
    }
    else if (frame.document) {
        return frame.document;
    }
}

I was happy to see that C# has a similar feature, the Null Coalescing Operator.

This is nice for the same reasons, and is more readable than the normal ternary operator when simply checking null. Of course, if I am not checking for a null value, but a more complex boolean expression, I will usually use the ternary operator or an if-else block it is more than just a variable assignment or return statement. Anytime I use any extra syntactical feature, it is my hope to make the code more readable.

Anyway, here is an example of the ‘??’ (coalescing) operator contrasted with the ‘?:’ (ternary) operator.

string APP_DEFAULT = "application/octet-stream";
 
private string GetContentType(string contentType)
{
    // Nice and readable.  The '??' operator can be very useful.
    string FUNCTION_DEFAULT = null;
 
    return contentType ?? FUNCTION_DEFAULT ?? APP_DEFAULT;
}
 
private string GetContentTypeTernary(string contentType) 
{
    // This is NOT readable.  Please do not use nested ternary
    // operators... They take too much energy to figure out.
    string FUNCTION_DEFAULT = "text/plain";
 
    return (contentType != null) ? contentType :
        ((FUNCTION_DEFAULT != null) ? FUNCTION_DEFAULT : APP_DEFAULT);
}