Java Web App automatic hot deploy in 50 msec

April 9th, 2010

Java web development is notorious for its long deployment times. Java’s modify-build-deploy-test cycle is a huge disappoint to developers used to TurboGears, Ruby-on-Rails, or even PHP’s “save-and-see-changes-immediately” work flow.
One important goal of SlimWeb is to shorten these cycles, so that you can spend more time writing and testing your application, and less time waiting for your web app to be redeployed. Using SlimWeb’s default configuration you can already achieve redeployment times of under a second, which is OK by Java standards, but still slow compared to the competition.
Fortunately, SlimWeb’s redeployment times can be shortened in many, although not necessarily trivial, ways.

In this post I’ll try and describe how I got my application to redeploy in less than a hundred milliseconds!

What is involved in a SlimWeb app redeployment?

  1. Checking and gathering of external dependencies
  2. Compiling your code
  3. Preparing the Java Web Application folder tree
  4. Zipping the web application tree into a .WAR file
  5. Deploying the app to a web container (Tomcat, Jetty, etc)

This is what more or less happens when you type ‘ant deploy’ in your project’s folder.
The default SlimWeb configuration already optimizes this process to a significant extent.

  1. The external jars gathering happens only the first time you build your app. Once you have all the external dependencies and they are up-to-date, the first step of the build process does nothing.
  2. Building is incremental, so only the modified files are rebuilt as you would expect. However, if you use Eclipse, you can use it’s built-in compiler that compiles code as you type. When you hit ctrl+s to save your source file, Eclipse has already compiled it and simply saves it to disk, eliminating the need for the ant task to do any compilation itself.
  3. Every SlimWeb project has a ‘/WebContent’ folder that serves as the root of the web app tree. When you edit JSP, TAG, JS, CSS or image files, you edit them directly in this folder, so no copying is required, but, more importantly, no redeployment is required, when these files are modified. Redeployment only happens when external dependencies (which are automatically copied to ‘/WebContent/WEB-INF/lib’) and/or resource and .class files (copied to ‘/WebContent/WEB-INF/classes’) are modified.
  4. During development you can skip the .WAR archive. You can deploy the exploded WebContent folder directly in your web container. On Linux and MacOS you can create a symlink in TOMCAT_HOME/webapps. On non-Unix platforms you’ll need to edit your web container’s XML configuration files. Refer to your container’s manual for details.
  5. Your web container will by default reload a web app if a special file’s modified date changes. For Tomcat that special file is ‘/WebContent/WEB-INF/web.xml’. For Jetty it’s the context file for your app, something like ‘JETTY_HOME/contexts/yourapp.xml’.
    SlimWeb will automatically determine when a redeployment is needed (only if resource, class or jar files were modified) and touch the special file, triggering a redeployment in your web container.
    This step takes the longest time in the entire modify-build-deploy cycle. See below for some hacks to speed it up.

That’s how the default setup works and achieves redeployment times of a second or less (that’s when resource, .class or .jar files are modified).
Note that when you modify stuff outside of ‘/WebContent/WEB-INF/’ no redeployment occurs and changes are visible immediately.

Now let’s do some hacking

DISCLAIMER: The hints that follow may cut your redeployment times dramatically, but are not for the faint of heart! Unless you’re able and willing to modify your web container configuration, your app’s build scripts, and your Eclipse project’s configuration, you should wait for SlimWeb 2.0 where some of these ideas may find their way into the framework itself, and hopefully become a little bit more developer-friendly.

You can greatly speed up reload times within your web container by moving some .jar files from ‘/WebContent/WEB-INF/lib’ to your web container’s lib folder. For Tomcat that’s ‘TOMCAT_HOME/lib’, and for Jetty it’s ‘JETTY_HOME/lib’
The more jars you move this way, the faster your app will redeploy. When you move a file to your web container’s lib folder, make sure your app’s ant script no longer copies it to ‘/WebContainer/WEB-INF/lib’. You’ll need some ant knowledge to do this. The smaller your WEB-INF/lib folder, the faster your app will redeploy.

WARNING Only do this for external jar files. Do not attempt this trick for your own .jar files.

HINT This works best for “runtime dependencies”, i.e. jars that are only needed at run-time, and not at compile-time. The best candidates are your JDBC driver, and the JPA implementation jars. You can also do this for your “compile-time” deps, but you’ll need to further edit your ant script, so that it can find the compile-time jars which are no longer in ‘WEB-INF/lib’, but also you will need to configure one or more User Libraries in Eclipse, so Eclipse itself can find them. By default both ant and Eclipse look for compile-time jars in ‘WEB-INF/lib’ so be careful when you move stuff away from that folder.

The Magic redeployment hack

If you’re using Linux (the reload.py script requires pyinotify, which is only available on Linux!) you can go one step further and disable the ant part of the build process altogether, relying on the Eclipse compiler only.
To do this you can follow these steps:

  1. Run ant build at least one so that dependencies are initialized correctly, resource files are converted as needed, etc.
  2. Go to ‘your_project/build’ and rename the ‘classes’ folder to ‘classes.bak’
  3. In the same folder create the following symlink: ln -s ../WebContent/WEB-INF/classes
  4. Launch the magic reload script: SLIMWEB_HOME/bin/reload.py
  5. Edit a source file in Eclipse, and hit ctrl+s.

On a Core2 machine, your app should automatically reload and be ready for testing 50 msec later :-)

Quote of the Day

March 11th, 2010

To err is human. To really mess things up you need a computer.

- Unknown

(Not quite!) Anonymous Web Browsing

February 19th, 2010

http://panopticlick.eff.org/

“Your browser fingerprintĀ appears to be unique among the 651,356 tested so far.”

Lucky me :-)

Announcing SlimWeb 0.9.10

February 19th, 2010

The latest build of SlimWeb is available here:

http://download.slimweb.net/devel/slimweb-0.9.10.tar.bz2

Release highlights:

  • More complete samples
  • A basic JSON-to-Java marshaller
  • Fixed the response.getOutputStream() bug
  • Various minor bug fixes

KDE vs GNOME 2010

January 30th, 2010

I’ve been an enthusiastic KDE user since the Red Hat Linux 5.0 / KDE 1.x days. The best Linux desktop experience ever was provided IMHO by the KDE 3.5 series.

KDE 4.x was a complete disaster for me. I do realize that the 4.0 series were an experiment on their loyal user base, sorry, I mean an early release aimed to gather user feedback rather than provide the ultimate Linux desktop as we’ve come to expect after the excellent 3.5 series. I know that, I did read all the right blogs, etc. So I’m willing to forgive KDE 4.0’s countless bugs and tons of missing features.

What I can’t forgive is that while there have been gradual improvements, KDE4 remains trouble-ridden to this day. Even two years later the current stable release (4.3.5 – the last in the 4.3.x series according to kde.org) still has some glitches that annoy the hell out of me.

My number one pain that nags me every day is that the damn taskbar rearranges itself every time you switch virtual desktops. I’m using the Manual task arrangement option, and I expect when I return to a virtual desktop to find the same tasks in the same order I left them. Come on, guys, WTF?

Amarok is the best audio player application that I’ve ever seen. On all platforms. But it took forever to port it to KDE4. And the bugs I’ve encountered with it over the last two years have frustrated me to a point where I’m using rhythmbox under KDE :-(

A few days ago Amarok would not even output any sound under Fedora 12 at all. It used to, but some update broke it. Fortunately, right now the latest update works. Still, I don’t remember such brutal issues with KDE3/Amarok1. An audio player that fails to output any sound… come on, guys, WTF?

And even now with the latest Amarok 2.2.2 and KDE 4.3.5 I’m having the ages old problem where playback will sometimes stop after the current track. I have really tried but I can’t figure out how to reproduce this reliably.

But the good thing about Linux (and other Unix-alikes) is that there are plenty of desktop environments to choose from :-)

BTW, when I’m talking about a desktop environment, I’m including applications which are not strictly part of the respective desktop environment. Basically I’m counting all QT/KDE apps on KDE’s side, and everything that is GTK on GNOME’s.

GNOME has always felt too simplified, rigid and plain, compared to KDE’s excellent customizability and countless bells and whistles. But to my amazement after using it for a while it grows on you. The various GNOME apps (terminal, audio player, rss reader, etc) have always seemed inferior to me compared to their KDE counterparts. But less so in more recent versions. For example, rhythmbox now has a context plugin to display info about the currently playing band, song lyrics integrated right into the player window, and while it’s not an Amarok yet, it is inching closer and closer.

I’m just getting started and it already seems this rant is getting a bit too long :-)

Here’s a brief summary about the things I like/dislike about both environments:

KDE niceness:

  • Many KDE applications are still better imho compared to their GNOME counterparts, e.g. amarok, konsole, akregator, ktorrent, kdesvn, kompare, konqueror (file manager), etc.
  • Applications like konsole remember open tabs after logging out and logging back in.
  • The start menu is searchable (the only undoubtedly useful feature of the great KDE 4 re-imagining)
  • (minor) The plasma widgets are nice, but not a decisive feature for my choice of a desktop environment. I usually have plenty of windows open on all virtual desktops so I rarely see my desktop :-)
  • (minor) I like developing QT UIs a lot better than developing GTK/Glade.

KDE brain damage:

  • (major gripe) The tasks on your taskbar rearrange themselves when you switch virtual desktops (even when the ‘Arrange Manually’ option is selected!) – update: this is fixed in KDE 4.4, see Rex Dieter’s “Fedora 12 KDE 4.4.0 Remix
  • Amarok still has issues.
  • (minor) Starting a KDE app is noticeably slower under non-KDE desktop environments. Starting additional apps is better though.

GNOME advantages:

  • It clearly shows that significant amounts of thought and effort were spent on cleaning up and streamlining its UI. GNOME provides the Mac experience on Unix systems :-)
  • GNOME’s 3D desktop is the best
  • I really like the GNOME weather applet :-)
  • (minor) GTK is the native widget library for a number of critical apps for me: Firefox, Google Chrome, Miro, Pidgin, Eclipse. No significant performance hit is observable when running all those under KDE though.

GNOME disadvantages:

  • (minor) I could neverĀ get used to the spatial Nautilus feature. I usually have too many Nautilus windows open with too many items in them. Luckily the file browser mode (it has tabs!) saves the day.
  • (minor) The default configuration with two panels – one on the top (with start menu, systray, but mostly empty) and one at the bottom (task bar, virtual desktops, etc) is a HUGE waste of screen space, especially on a wide screen. Luckily it’s quite easy to merge the two panels into one and still have a comfortable experience, even on a 4:3 display, so this is hardly a showstopper.

Final Verdict

I’m still a huge KDE fan…

But there’s no way I’m using KDE until the taskbar rearrange bug is fixed (4.4 is due in ten days…)

In the mean time I will feel comfortable in my customized GNOME environment :-)

Electric biking

January 13th, 2010

Finally my childhood dream has come true.

I rode an electric bicycle today :-)

It was an eZee 400 watt, 36V, 9.6Ah LiMn model, not too different than the 500 watt, 10Ah LiFePO4 model I’ve been planning to get :-)

The ride itself was pretty short (huge thank you to the kind rider, who trusted a complete stranger with a $1000+ bike!), it felt less powerful than I expected, but then again more power will likely lead to stability problems. Also my own bike will have 25% more power :-)

It was hard for me to use the throttle while pedaling, I was going either on the electric motor alone, or on pedals alone, or at least this is what it felt like… Perhaps I’m mistaken, or maybe some practice with the throttle is required to be able to get just the right push so that you’re not pedaling slower than you’re going or vice versa)

Anyways, the good thing is that even with 400W you can still ride without ever using the pedals :-)

The bad news is that still nothing like this is on sale around here, so it seems I will still need to smuggle it from another continent, but the important thing is the concept is now officially validated, fiscal plans are in place and the electric upgrade to my bike is now inevitable!

I’m finally free

January 4th, 2010

“This feeling inside me
Finally paid my dues, I’ve finally broke free
No longer torn in two
Living my own life by learning from me”

P.S. My Current Financial Status:

Bank account balance right now: 3.41 EUR
Total personal debt as of Jan 4th, 2010: 0.00 EUR
Overall Status FILTHY RICH

Superstars

December 16th, 2009

At the Gay Bar…

Gay Bar

Electric Two

Fedora 12: Unite.

November 18th, 2009

Welcome to Fort Constantine, everyone!

Announcing SlimWeb 0.9.6

October 29th, 2009

The latest build of SlimWeb is available here:

http://download.slimweb.net/devel/slimweb-0.9.6.tar.bz2

Release highlights:

  • Painless JSON output
  • Tiles2 integration
  • Fixed the bad-property-kills-entire-instance bug
  • More sample projects