Archive for March, 2006
Update: Aug 22, 2006: Download the Code and a working Demo!
Also, read our documentation on the inner workings of the python CGI.
Demo:
The Python File Input CGI
About four months ago, Christopher Bottaro, another developer here at FineTooth, worked on a Python CGI script to handle file uploads and it is mighty ingenious. The CGI uses standard input as a stream, which PHP cannot do. The CGI writes the stream data as it arrives, in blocks, to a file buffer. This enables us to use another PHP server-side script to to poll the size of the file buffer while it’s being written to. With this, we can calculate the parameters necessary for a progress bar user interface (time remaining, speed of upload). We integrated the whole kit and kaboodle into our PHP Form class so that if a form indeed had a file input, the progress bar was all set up to popup automagically, and the form’s action was set to point to the CGI. Upon file upload completion, the CGI would perform a redirect to the ‘original’ action URI for data processing.
The Problem Case – File Inputs & JavaScript
As we’ve moved more and more towards using JavaScript to control our application’s form processing (as opposed to using the standard form action attribute as an URL) one problem situation has been caused by the file input. Since the file input is essentially an OS level object, JavaScript can’t access the information (thank goodness!) in order to send it programmatically to a handler. So what we’ve done in the past has been to use iframes for forms that needed file inputs. I’ve read that the Dojo Toolkit basically does this technique – by detecting if the form has a file input on it, and if so, it uses an iframe mechanism to submit the form. For awhile this was a good workaround. However, for aesthetic and programmatic design reasons, I wanted to come up with another solution.
Why not just use Iframes for those forms?
For one thing, iframes are just kind of annoying. If you want to access your JavaScript framework, you have to program in the scope of window.parent. Hopefully, the browser is caching your stylesheets and javascripts, but I’ve sometimes seen otherwise watching apache logs, so I don’t believe the iframe solution is always efficient. More importantly, though, using the classic form post technique and then rewriting output from the server to the iframe’s window is just not how I want to process forms in our application. I prefer to use Prototype’s Form.serialize and then Ajax.Request to deal with form transactions. Our application doesn’t have page refreshes overall, so introducing them into little iframes if and only when a form needed a file input just feels so cowardly. And, all that *really* needs to post is the file input data. So suddenly, the brain dinger started going off.
Embedding N-Iframes
Instead of using an iframe for the entirety of the form, I wanted to try embedding iframes individually for each file input that needed to be on the form. I could have an iframe that loaded up its own form object into its window.body with only the file input and some session information. Then, via the controlling form, we can use javascript to submit each embedded iframe’s form programmatically. Once the poller running in the main form detects that all of the files are safely on the server, we can submit the rest of the form data using Ajax.Request along with some logic to capture the information about the freshly uploaded files. It sounded funky, but lo-and-behold, it works and it scales.
A diagram might help make more sense of this approach:
Form
*text input
*text input
*Iframe
*form
*file input
*text input
etc...
What’s key here is that instead of having a single CGI try to deal with multiple file inputs and all the other inputs on the form, our CGI is doing only what we originally wanted it for in the first place – the file upload.
Here’s the form processing logic:
* User chooses file(s) to upload in the form.
* User presses submit.
* JavaScript validation runs against the form.
* JavaScript looks inside the form for any embedded iframes.
* If it finds them, it submits the embedded form to the CGI.
* An interval runs again and again until all the files
are reporting 100% upload
* The rest of the form data is at last sent using
Ajax.Request
A few other niceties
Since our CGI is writing data in chunks, and since the juciest information about the file is in the first little bit of data, the CGI can do a little regular expression matching early on in the process to capture the file’s mimetype and name. Our poller can then read this information and we can show a purty little icon for the filetype and also reference the actual file name in the poller window – making it look pretty slick!
The Poller
The return data from the poller comes back to the browser as JSON and looks like this:
({"success":true,
"percent_done":3,
"kps":3,
"timeleft":"21 mins 8 secs",
"filename":"01 Mistakes.mp3",
"mimetype":"audio\/mpeg",
"mimetype_iconsrc":"music.png",
"current_size":126676,
"total_size":4044948})
So Why can’t we do multiple files all at once?
Originally I wanted a display that resembled Firefox’s Dowload Manager. One problem I haven’t found an adequate answer to is, why does it seem that I can’t be sending up multiple CGI requests from the embedded iframes at once? I can see that apache seems to be blocking until all but the last CGI are left in terms of sending responses back to the browser. Maybe it is the browser doing that; I did find one mention of that being the case back in old versions of Netscape, but I can’t seem to find any other validation of this limitation anywhere. So until that issue is resolved, files will be sent and polled sequentially in this version.
March 31st, 2006
Author: Lindsey Simon
As a company that is working on implementing the Agile process to develop a product aimed in the enterprise space, I have been very interested by the series of posts from RedMonk’s Michael Cote called “Agile in the Enterprise”. His comments and the problems of implementing Agile with large teams got me thinking. He mentioned the practice of having “Scrum-of-Scrums”. He mentions that coordinating all those small teams and keeping everyone pointed in the same direction is difficult. I think one of the problems comes from the structure of teams in large software shops.
Traditionally teams are divided by areas that match up to the architecture of the technology. So you would have a “UI” team and a “Repository” team and an “API” team, etc. This kind of division creates silos of expertise and responsibility that are an anathema to the Agile process.
For one, this break up harms the open communication upon which Agile depends. The channels of communication between teams in a large company are not going to be as open and responsive as those within the smaller group. Teams are placed in different areas of the company, sometimes on different floors or buildings, or worst yet cities. If a developer from the UI team has a problem with some new feature that interacts with the reporting engine, how long is it going to take to get an answer/solution when the report engine team in located in a different building instead of the next room?
Also, new features rarely rest in a single part of a system. This forces the work of a story card to be divided among several teams with different priorities and team leads. The coordination for this division is supposed to take place during the Scrum-of-Scrums. But each group will prioritize its own work to meet its own deliverables. This could create delays and conflicts between groups.
Finally, I think the break up of expertise and responsibility amongst teams will harm the clean and refactoring of the system. In Agile you are supposed to constantly be refactoring to clean up cruft in the system and improve its implementation or design. But if you only have access to or knowledge of small segments of a larger system how confident are you on the repercussions of your changes? And if the one team spots a change to be made in another part of the system can they make the change or does it get scheduled for the next sprint?
Ad-Hoc Teams
An idea I was mulling over, and I don’t think it is probably unique, is to create teams in an ad-hoc fashion. At a large development shop there would be a pool of team leads and a pool of developers. At the start of a sprint teams would be formed based on the work to be preformed. Each group would contain the developers and expertise necessary to complete the story cards they were assigned. If the cards require 2 UI guys and 1 DBA, fine. If it requires only developers that know the repository engine, fine.
The processed use to form these groups at the start of a sprint can be as formal as the business requires. You could get all the leads in a room and have them divide the work and then chose which developers would be best for each team. You could also have a more democratic method of all the developers choosing there own teams as you go through the cards. You can try and have people rate which features they would like to work on in the next release and then try and best match those preferences. I think the method used depends a lot on the personalities of the people involved and how they interact. Results for each may vary and it is a process that I think would have to be tuned for the first couple of sprints. The only process I would avoid is the having the team leads pick people in a round robin type fashion. A little too much like the schoolyard playground. Too many painful memories for most programmers about being picked last every time for baseball even though you know you are better then Stevie. But I digress into my own childhood issues.
The important part is that you want people overtime to work with different teams and in different parts of the system. This will improve everyone’s knowledge of the system. It will also give everyone a chance to work with everyone else. I think overtime this will make the teams more productive and estimates more reliable.
These ad-hoc teams could also give a company the opportunity to try a lot of flex space solutions. Do the ad-hoc team members move every sprint to a new location? If so what kind of office space do they have? One war room or cubicle area per team? What kind of IT and building management issues can this cause? If you don’t relocate people for each sprint how do you create open communication channels for the new team? Also, would not moving people create pressure to put people that are already located together in the same team? This would scuttle the idea of having people move through teams. I know some companies that are very successful and have developers living in different cities but still work closely with each other. IM, email, Skype, and monitor sharing software may be the answer for this.
FineTooth currently has a small development team and we haven’t had to worry about these issues yet. But as we grow as a company the problem of keeping a fast, agile small company approach is on my mind. I would like to hear what other people are doing, or not doing and their experience. I would especially like to hear about people who have worked for companies that have transitioned from 5-10 developers to 20+.
March 17th, 2006
Author: ScottD
I figured this would be a good place to record some thoughts that have come up at SXSW as well as to record some notes about the panels I’ve attended this year. So here goes:
Saturday: Daniel Gilbert Presentation: How to Do Precisely the Right Thing at All Possible Times
I accidentally just deleted my entire review of this panel by typing Ctrl-W a la emacs cut. In the end, it must not have been karmically all that useful!
Saturday: Jim Coudal / Jason Fried Opening Remarks
37signals spokesman and his partner in marketing, Jim Coudal, made strong cases for the value of curiosity and the waste of time that is dogma. Curiosity empowers individuals to know how to learn better and more quickly, as opposed to being just purely information libraries. By ditching the dogma (technical specs, documentation, etc.. ) developers can focus on developing itself – trying new ideas, challenging ways of doing things, etc. It was a very uplifting conversation, but I heard a great deal of mumbling, that “Yeah, that works for your little company, but I don’t think it would work for mine”. Mr. Fried had indeed said that their conclusions would not work for everyone, but rather to just be open to the possibilities that there is “no right way”. Start small, fail in obsucrity, iterate, and succeed!
Sunday: What’s Hot in Web Applications
Fix a pain-point for people, and wait patiently for the cash to start rolling in – that was the theme of these Silicon Valley webapp folks. They stressed the importance of solving a real life problem that you are actually having as being the key motivator behind successful startups. If you’re only trying to solve other people’s problems you will both tire of that and come to resent it. The sites demo’d were:
YackPack – an audio-based email webapp. Mostly struck me as being a pain in the butt and slow. I don’t want to use the web to listen to people talk myself, but it was certainly interesting to see how it could be used that way. Maybe when the speed isn’t such of an issue this technology will be cooler.
Meebo – web based IM client integrator. This was really cool – and made some great cases for how web-based apps are really making a web-based OS more possible.
Zimbra – competitor to the Exchange behemoth. Zimbra’s mail client was totally impressive. Really cool integration tools (Blackberry, Calendar, etc.. ) and made me start to wonder if mutt/vim isn’t the ultimate email tool…
Sunday: Making Web 2.0 Accessible
This was an unfortunately useless panel – unfortunate, because it left so much hanging. Short on usable advice and remiss on any truly hard-to-find information, the best way to sum up the theme would be to “get back to basics, and make sure there are ALT tags in your images”. Sure, that’s always good advice. But what the folks in the crowd are hungry for are options and useful examples and advice. There was only one code example in the whole presentation, and yet the conversation centered mostly on code.
If, as Shawn () from the W3C suggested, WCAG 2.0 is was designed from the bottom up for testability as opposed to readability, why was there no demo of a test we could all go replicate on our sites? Maybe the “testability” she meant involved getting users to test and watch them only, and not programmatic testing. Sure, Bank of America can create a team dedicated to doing Accessibility testing (only out of fear of lawsuit, mind you), but what can the developers in the crowd do? It’s easy to shoot down AJAX frameworks and applications(BaseCamp) for NOT being Accessible, but where was the demo of a truly good accessible site? What are the best techniques? What tools are better than others and how should we use them?
No one had a decent suggestion of a framework that had been was more accessibility minded than others. Did anybody “fix” one of these frameworks and post it for download? Now that would have been useful! Until we have the resources to hire less enabled to test our applications or the threat of a lawsuit on our hands, we are left as helpless as we started before the panel. It is indeed noble to code towards Accessibility. Do not mistake my bummed out reaction to this panel as apathy toward the goal of having webapps be truly accessible.
Monday: CSS Problem Solving
This was a fairly fun and useful panel that dealt with a variety of CSS problems and how to solve them. Not a whole lot to say about it really. I didn’t see much going on there that blew my mind, but I don’t think that was the intention. It did seem like everyone wanted to promote their new BSS book however.
Monday: Microformats
Now this was a really great panel. Tantek Celik is a great moderator. He was well prepared and his slides are worth downloading. The panel didn’t ever feel rushed, but still crammed a ton of info and examples into it. Examples of using hCard and vCard translation from technorati, mashups of SXSW speakers and after-parties – mapped on Google maps, Flock recognizing all this extra content.. Man, this was cool! But, I think it’s a technology that’s waiting for the killer app to take advantage of it, and even Flock seemed not to have convinced me that it was it. I’m going to install tails right now though and see if I can make use of these things. My gadget collection is currently pretty weak – my cell phone has trouble holding a charge, nevermind vCards 
March 13th, 2006
Author: Lindsey Simon
Hello Prototype
So I just svn updated my scriptaculous directory and lo and behold there’s a new testing 1.5 version of Prototype in the lib directory. Yum. I’d seen the evalJSON method in the Request before, but not until now did I really want to take advantage of it. In our application, I’m not really using the XmlHttp request to send back large, complex objects to the browser, so 9 times out of 10 some smallish, boolean-oriented response from the server is aok with me, though I’d like to reserve the right to do more sometimes. In other words, the speed-hit from doing an eval on the return string is not something I’m worried about.
Hello JSON
When we inherited the skeleton of our application from Handwire design, it came with two Javascript classes for handling XmlHttpRequests. Their XmlConnector was fairly equivalent to prototype’s Ajax.Request and the HtmlConnector is like Ajax.Updater. The XmlConnector had built in the concept of taking the return (as XML) and parsing it into an object for javascript. Since this return object is both pretty small and pretty simple, it seems like a perfect case to switch over to JSON instead of XML and ditch the 20-line XML DOM parsing routine.
PEAR Services_JSON
Since it exists, I thought I should use Michal Migurski’s Services_JSON PEAR class. I created my own wrapper class called ArrayToJSON which pretty much just adds some headers and spits out the string of JSON.
evalJSON() in Ajax.Request
That JSON string is cool as responseText, but prototype can do something even better with it – if the returned page contains the X-JSON header with the string as its value too. So in my ArrayToJSON class, when I get to the part where I pass out the headers via PHP, I have one extra line:
header("X-JSON: $output");
So I spit out the JSON both in the headers and in the page content. I left it in the page content for readability. This of course means the returned packet is that much bigger, but I’m not concerned. It’s still really quite small and this could always become conditional later.
Note: I had to stick some parentheses around the $json->encode() line since prototype’s evalJSON method doesn’t enclose the value in parentheses the way many recommend for evalling. So now, my return looks like this:
({“success”:true})
and then when I call Ajax.Request, I can set onComplete to a function which will then get two parameters:
exampleOnComplete: function( transport, json )
{
alert( 'yeeha: '+ json.success );
}
Error Handling
Well, I tried to set up what looks like a very elegant way of setting global called functions via prototype, by doing (I’m not showing them, but I’d created some functions called Ajax.onException, Ajax.onFailure etc.. ):
Ajax.Responders.register({
onException: Ajax.onException,
onFailure: Ajax.onFailure,
onSuccess: Ajax.onSuccess
});
But I don’t seem to be getting the call order I expect, so instead I’ve created separate objects which I use when I want to initiate Ajax request or updater calls and then set those in that object, like so:
var AjaxRequest = function( obj )
{
if ( !(obj.parameters && obj.onComplete) ){
top.location.href = "?module=error&error_code=2031&" + "error_message=" + escape("AjaxRequest failed to initialize");
return;
}
// we add on a parameter to break cache and so we know we're dealing with AJAX on the serverside
var random_num = Math.round( ( Math.random() * 666 ) );
new Ajax.Request( '?', {
method: obj.method ? obj.method : "get",
parameters: obj.parameters + "&AJAX=Ajax.Request_"+random_num,
onSuccess: Ajax.onSuccess,
onComplete: obj.onComplete
onFailure: Ajax.onFailure,
onException: Ajax.onException
} );
}
You’ll see there’s an additional parameter as well to all outbound Ajax requests(Ajax.Request=randomnum) – that is so that on the server side, I’ll know which requests are coming via Ajax, and the random number ensures a cache break. I thought about using that strange _= parameter that prototype lops onto the end of every request, but found a blog post that mentioned that the underscore thing is actually a fix for a Safari bug and I expect it’ll disappear at some point anyhow.
My plan is to take advantage of the order of the callbacks to use onSuccess to test for an error on the serverside, and at last onComplete to pass to an onEvent function that actually can do the secondary work, assuming a successful transaction. This is nice because then I’m writing code without having to always write in a conditional to check for errors and consequently call an error function; error checking is now handled exactly the same way for every call.
Conclusion
I’d love to see how other people are implementing prototype’s JSON and error handling mechanisms into their applications. There’s not a great deal of good examples or documentation online about how one should go about doing this, and what’s on this page represents my first crack at it and a few hours of research. I’ve updated the DataGrid code to version 0.2 from an earlier post – now the grid uses this functionality instead of the XmlConnector and HtmlConnector , and you can see that in action here:
Wherefore art thou DataGrid
March 1st, 2006
Author: Lindsey Simon