Friday, May 11, 2007

Flash, Silverlight, and JavaFX all look nice, but what I really want is better file upload

One of the most annoying limitations when writing web apps comes from the <input type="file"> form element, which is the standard way of uploading files.

Here are a few of the problems with <input type="file">:
  • You can only select a single file per input element, and there is no way to select entire folders.
  • Web browsers just pop-open the standard file open dialog, which is often not very good. For example, the dialog probably just shows a list of files, not thumbnails. This can make it difficult to pick the correct file if they all have names like IMG_3485.JPG, and it's therefore easy to upload the wrong file.
  • There is no way to actually read the data locally without first sending it to the server and then having the server send it back.
  • There is no way to do any local processing on the data, such as scaling images to make them smaller.
Flash apparently solves the multiple file problem (kind of), but it's my understanding that it still has all of the other problems (I'd love to be wrong about this though).

Java applets can actually do most of these things if they are signed and the user agrees to allow the applet access to their filesystem. Unfortunately, Java applets don't actually work, and after 11 years of not working, I'm not expecting that to change. Facebook has a nice image uploader applet that lets you browse your photos locally and then scales them before uploading to make the whole thing go fast. The only problem, other than the annoying "do you want to trust this applet" prompt, is that it crashed my browser the second time I tried it.

To me, the great thing about flash is that it fills in basic capabilities missing from the browser, such is video, sound, camera and microphone access, and they did it with a reasonably small and stable browser plugin that I'm not afraid to allow onto my computer. A few months ago, I accidentally upgraded one of my computers to Flash 9 -- I don't remember how exactly, I just remember that it was simple and non-eventful.

Here's my idea: I want to create a small, reliable browser plugin that does for file uploads what flash does for audio and video. It should work with all the modern browser/os combinations, and install with minimal effort and fuss (browser restarts avoided if possible). It would be free and open source, but hosted in a common location for easy install.

Popular Ajax libraries such as jQuery, YUI, and Dojo can add support in such a way that people who don't have the plugin just get the regular browser upload control (and maybe a link to install the plugin), while those who do have the plugin get a better ui. Clearly, getting people to install a new plugin isn't easy, but once any site convinces them, their experience automatically gets better on all other sites that support the plugin. There isn't a chicken&egg problem because the plugin simply provides an enhancement to the already existing (but very poor) browser functionality, so sites can easily add support without fear of losing people who don't have the plugin (unlike building a whole app in silverlight, or whatever).

I'd like to hear from anyone with experience writing browser plugins. How difficult is the technology side of this? (making it small, reliable, and stable -- crashing browsers is not ok) Also, are you interested it writing this plugin? I'm serious about this, and willing to pay quite a bit of money to make it happen.

Here are a few more details on how I imagine the plugin working:

It would not have any in-browser UI -- it would simply be an object that can be called from javascript and that would pop open a nice file browser with thumbnails and all that (similar to windows explorer, perhaps). It would also provide upload-related utility functions, such as image resizing. Since the file access is read-only and limited to what the user selects, security shouldn't be a big problem, assuming of course that the plugin implementation is secure (and it must be).

Here is some JS illustrating how I imagine using this plugin (UploadHelper is the object provided by the plugin):
// Have the user choose files or directories to upload.
// Returns undefined if user clicks Cancel. Doing this with a
// callback instead of a return value would also be ok.
var files = UploadHelper.chooseFiles(
{filter:'*.jpg,*.gif,*.png', allowDirectories:true});
if (files) {
var s = 'You selected: ';
for (var f in files) {
var file = files[f];
s += (file.name + " (" +
file.imageinfo().width + "x" +
file.imageinfo().height + ", original size = " + file.size);

files[f] = file.scaleImage({maxwidth:100, maxheight:100, quality:5});
s += " scaled size = " + files[f].size + ") ";

// If I could somehow insert this image on the current web page
// too, that would be really awsome.
}
alert(s);

// using jquery XMLHTTP wrapper
$.ajax({
type: "POST",
url: "/upload",
data: UploadHelper.encode(files, 'multipart/form-data'),
contentType: 'multipart/form-data'
});
}

var files = UploadHelper.chooseFiles({filter:'*.txt'});
if (files) {
// We can also read the content of the selected file
// The ability to get chunks of a file also means that it's possible
// to restart failed uploads, and to chunk large files into multiple
// POSTs.
for (var f in files) {
var file = files[f];
var d = file.getData(0, 50); // get the first 50 bytes
if (file.size > 50) d += "...";
alert(files[f].name + ": " + d);
}
}


Update:

Several people have suggested that I should instead try to get this feature added to web browsers or (better yet) the web browser standards. That's a fine long-term solution, but realistically it will take years to happen. I would rather have an immediate (months, not years) fix in the form of a plugin, and then let the web browsers add it as a standard feature on their own schedule. This is somewhat similar to the path followed by XMLHTTP.

Sriram Krishnan of Microsoft makes a few good points:
  • The standard file open dialog in windows already has a thumbnail view. Unfortunately, this feature is hidden under a completely unremarkable icon and despite having used windows for many years, I've never seen it before today. This is a great example of why simply adding a feature isn't good enough -- it also needs a good ui or else it may as well not even exist. Furthermore, not existing seems to be the status of the thumbnail view on OSX -- hopefully that will be fixed in 10.5.
  • This feature request could be implemented in Silverlight. That's great news! MS should write and release a little Silverlight control that provides these functions to JS. I would be very hesitant to write (or rewrite) my apps in Silverlight, but if I could simply embed a little Silverlight module the same way that I would a flash mp3 player, then I'd definitely use that. This would be a very effective way for MS to get people started installing and using Silverlight (start with something small and optional).

Silverlight sounds pretty nice. Google or Adobe had better have an open alternative on the way, or else MS may be able to recapture the application platform.

Update Two:
I believe that within a few months we will have a few reasonable solutions to this problem!

20 comments:

Sriram said...

Paul - you can do most of this in Silverlight

- The folder picker dialog shown in Windows can be switched into thumbnail mode. This is not Silverlight specific - this is built into the Windows common dialogs and is popped up with <input type="file"> as well.(I have no idea how this works in other operating systems but I guess it would be similar)

- Silverlight lets you read local files if the user opened them through the above-mentioned file picker common dialog.

- You're free to do whatever you want on those bytes before sending them on their way to the server. This should take care of your image scaling example

- Sriram Krishnan
www.sriramkrishnan.com

Sriram said...

In case you want to find out more, look up OpenFileDialog in Silverlight. Or send me a mail at sriramk@microsof.com and I'll help you out

Btw, do I get to collect some of that money you spoke about? ;-)

David Dahl said...

As far as the code-signing madness is concerned, you can get around all of that with a Firefox extension. The browser re-start is required, but, Firefox returns as it was, with an intact session. I have been doing some Mozilla hacking (on extensions), and am quite impressed with what you can do. There is SQLite support and filesystem access.
The only thing you would have to do to add resizing/scaling is some python code (via py2exe) that you call from your extension.
This would be x-platform but Firefox only.

lee said...

Allowing a website to read the client's files seems like a pretty big security hole, no?

jayshao said...

Is this something that is better handled as part of HTML5? It seems like what we're really asking for is more sophisticated form controls -- why not get it worked into the spec instead of shoehorning it into a plugin?

Anil Philip said...

"Unfortunately, Java applets don't actually work,"

I dont get it - why should file upload be part of the browser?

IMHO, it belongs to, and should be part of the application.

Also, have you looked at Java Web Start?

Ohad said...

if you're serious, i'm willing to write the Internet-Explorer version of such a upload plugin, i think it's a worthy cause.

other than the basic file-selection, resizing, client-side-modification, it can also provide a standard progress bar for file upload.

another important feature is being able to drag files into specific parts of web-pages, in particular file-upload fields in forms. there's a FF extension that does just this (dragdropupload). And also there's a concept my company's developed that allows writing JS file-drop handlers for web-pages (www.3d3r.com/bubbles).

in fact implementing the custom file upload as part of Bubbles would be cool.

Ken said...

"Unfortunately, Java applets don't actually work,"

Paul, I'm a PM from Sun's Java group. I'd like to gain a better understanding of the problems you've encountered with Java applet and plugin. If you're interested, send an email to ken.chen@sun.com.

BTW, have you seen this year JavaOne demo called Iris. This is a cool demo showing "drag and drop" functionality within an applet. Take a look at the video.

http://jasperpotts.com/blog/2007/05/iris-video/

-Ken

Jan said...

Blah. I'd like some improvements too, but most limitations you mention simply arise from:
1) You don't know what os/browser the user is running. Think about it.
2) Security. The whole point is the user having control and handling the file to the browser. The browser should NOT, EVER control file upload in any way. Browsers run 3rd party code, nedd I say more? I understand you might want some more subtle improvements. I hope you understand that any such improvement is a potential security hazard to any of the millions desktop computer online.

Thor Larholm said...

Ken, Iris is not a Java Applet but a signed Java Application requiring that unrestricted non-sandboxed system access is exposed to the website Javascript.

What Paul wants is something small, simple and secure that exposes a functionally expanded and user-initiated file picker and file reader. The only file data exposed to the website is the ones explicitly chosen by the user, which also handles most security implications.

Signed Java in the browser will not be able to do this, the sandbox security is not adequately fine grained and the runtime dependency far outweighs the relatively small functionality scope.

Aside from the fact that Iris just took 5 minutes to launch on my fairly massive system I also agree with Paul that Java applets just don't work. The existing Applet sandbox will only let you draw graphics and produce sounds, anything useful beyond that requires signed applications.

sriram, I think this could be a great showcase for Silverlight. Coincidentally, I am currently reviewing Silverlight for potential security vulnerabilities.

Cheers
Thor Larholm

Marcus said...

This is quite possibly the worst idea I've ever heard... unfortunately, it's one that every single person who has had to deal with file uploads via HTTP has also thought of.

Why is this such a bad idea? Like was mentioned before, this is an amazingly large security hole. We're talking "drive a semi though it" sized hole.

Think about it from the user's point of view. Why would you want the creator of a web site (that you don't trust) to be able to upload any file they wished via javascript?. You don't. You want to have to select the file(s) yourself. Otherwise, you might end up with things (/etc/passwd on a *nix box, for example) moving over the network that you may not want visible.

This unfortunately means that you're limited to only the file selector that comes with the web browser.

This is why Java applets run in a sandbox. Now, you can certainly argue that it is too difficult to get out of the sandbox, but the need for the sandbox remains.

I'm actually a little scared that the person from Microsoft thinks that Silverlight will let you do this. If so, we can look forward to a repeat of the security nightmare that was ActiveX.

Marcus

Mike said...

Paul, I am on-board for this one.

I have been dealing with this very specific issue for years.

I would also like to add to the wish list:

1) ease of use for the semi-tech savvy (eg. autodetect digital cameras or usb storage device with images)

2) batch uploading of multiple selected files

Again, I have been dealing with this for years due to my job. My requirements match yours point-by-point, but also require multiple file uploads(sometimes 100s of images at once), SUPER EASY installation (my users' computer experience varies wildly), and it must be rock solid (or else I lose any gains with constant tech support).

I have tried the following in no particular order:

1) java - some success using jupload which meets most of your criteria above, and nearly all of my requirements, however, like you said, java has issues when you actually have to deploy it to many people. Especially when you can not control security settings or browser types and versions on a user's system. Sometimes it simply doesn't work at all for no reason in particular.

2) flash(version 8 and above) - installation was much easier for users, and much more consistent and solid, however I began to have technical issues with the way the data was being posted to the server and also on larger (many images) uploads, I would also have random quirks and failures similar to java.

3) html - very solid, but was a deal-breaker when the user had to upload many images not to mention the thumbnail issues you touched on above

4) custom apps - this is what I, to date, keep settling on. Drawbacks are installation, debugging for multiple platforms, and support.

5) Using the "web publishing wizard" built into windows xp - this works surprisingly well, even resizes images on the fly. Also is browser agnostic (as you use it from file explorer), and works well for multiple images. (however thumbnail view is tricky, and it is windows specific, and involves changes to the user's registry which is equally or even more difficult to deploy than custom apps) this is my current second favorite behind custom apps.

In my pursuit to provide users with easy file upload, I have written in excess of 10 separate full-blown custom applications just to upload images (and some pdfs)! I have used everything from c# to c++ to ruby and python with varying results.

I have also either tried or am trying apollo (still playing with it), firefox plugin (browser specific), xul app, embedded ftp control(dnd, worked on ie), and a multitude of various other concoctions that in one way or another depended on the user installing a third party app (and generally restricted the app to windows)

Also I have had 2 abortive attempts at creating browser plugins (activex). To date, deadlines have always forced my hand to custom apps before any plugin could be completed.

All that said. I am whole-heartedly on-board for this one. I can donate time or money if someone else is further along.

Please contact me if you wish to form a group to make this happen, it has been my "white whale" for some time.

-Mike

Hanamaru said...

Why to reinvent the wheel when signed Java applets + Javascript neatly solves the problem using the browser's standard UI.

Take a look here
(you need to create a free account) -- you can upload and download directories as well.

We plan to release soon some part of the source code as a free library.

robfelty said...

Paul,

I think you right that there should be a better solution for file upload. You might want to take a look at winkflash.com. They offer 3 different alternatives, including a java applet which doesn't seem to crash (for me), html, and some Windows only option. Maybe it will be inspiring for you.

Looking forward to seeing what you come up with.

Rob

Marc said...

I would be all for developing a "standard" JavaScript that can be guaranteed to work with any browser that has the plug-in, with the added features you suggest. That way, when new features are added to the standard - the plug-in gets updated, the browser can stay the same.

The snag is I have no way near the enough experience required :(

Michael said...

I don't think a browser plug-in would work at all. Most sites that include a file input are expecting one file, not numerous - therefore, it would have to be made available as a flash, javascript, silverlight, [insert newfangled technology here] that the developers could implement in the backend as well as the frontend.

Angelo Gladding said...

The web succeeded because of an open protocol, HTTP, and an open standard, HTML. Creators of user agents have always been free to provide software of varying designs and qualities of service. Thus we have an ecosystem of optionally proprietary software residing atop a layer of free and open architecture.

I use Linux. Therefore I cannot use Silverlight. To propose this as a viable solution is such a MS fanboy's delusion.
Flash has problems. It always has.
Java has problems. It always has. Sign me. Sign me...

All of the above require additional, proprietary software above and beyond the user agent. This might sound natural to most but let's examine the situation from another angle.

The fundamental purposes of the user agent is to browse, download, and upload. That covers read and write which was the original basis of the WWW. Therefore, these three actions should continue to be a part of the previously mentioned open standards.

The point of this post was to investigate a proper intermediary solution to the problem at hand -- a set of plugins that might eventually progress and be embraced by a standards body.

The fact that most deviated entirely from this concept in the comment threads reflects the current state of the industry -- disillusioned and blindly supportive of all the wrong animals.

My opinion.. End of rant.

All things said, Paul, in regards to your second update, and seeing as it's been a couple of months, how are we doing with those solutions?

**crossing my fingers**

ScooterP said...

Looks like I'm late to the party on this thread, but I've been noodling on a similar idea recently. What I'm considering is something a little more involved, but perhaps more powerful.

Rather than just an uploader, this is a control that has it's own local file system space. The trick is that the control supports drag-and-drop in and out from the local system.

Dragging a file into the control gives the control access to a copy of the file, and enables the control to then post it back to the server. Crucially, the control could also download a file from the server and store it locally. Then the user uses DnD to copy the file out to the system.

In this way the control provides a linked file space between my local system and the remote server. Thus it could be generally useful for local doc caching, as well as smart uploads. I'm working on this code now, and interested with connecting with others working on this problem.

Scott Persinger

Robert-Jan said...

How recognizable is this all! The proposed specs that Paul sums up are an exact copy of what I have come up with in an internal memo in our organisation. Only thing missing is the max-file-size barrier, which may be a server-side limitation, but would be very nice to be able to avoid.

Also, I have tried a lot of different solution, of which JUpload is the one we are currently using, with mixed succes. Java-runtime environment seems to be a hassle to install on OSX based systems (don't know if that is still the case), and for some users, it just doesn't seem to work .

A cross-browser, cross-platform solution to this problem, in any form or shape, would be an invalueable addition to my webapplications.

Scott said...

I saw a neat upload on www.facebook.com It uses a activeX/java solution which you can get from http://www.aurigma.com/Products/ImageUploader/default.aspx