Archive for the ‘Projects’ Category

Announcing SlimWeb 0.9.13

Saturday, July 24th, 2010

The SlimWeb 0.9.13 release comes after 4 months of tweaking, bugfixing and improving documentation.

Get it at the usual place:

http://download.slimweb.net/devel/slimweb-0.9.13-full.tar.bz2

The coolest news is that SlimWeb is now used in a growing number of applications, including internally at one of the largest virtualization providers in the world!

Release Highlights:

  • The framework core servlet is now a filter, check your WEB-INF/web.xml!
  • Some validation API cleanups, check your JSPs!
  • Routing convention addition: “/package/controller:action” is now a shortcut for “/package/controller.onAction” (or “/package/controller?onAction=1″).
  • Some internal API cleanups (more consistent naming of stuff)
  • Various minor bug and javadoc fixes

Java Web App automatic hot deploy in 50 msec

Friday, April 9th, 2010

Java web development is notorious for its long deployment times. Java’s modify-build-deploy-test cycle is a huge disappoint to developers used to TurboGears, Ruby-on-Rails, or even PHP’s “save-and-see-changes-immediately” work flow.
One important goal of SlimWeb is to shorten these cycles, so that you can spend more time writing and testing your application, and less time waiting for your web app to be redeployed. Using SlimWeb’s default configuration you can already achieve redeployment times of under a second, which is OK by Java standards, but still slow compared to the competition.
Fortunately, SlimWeb’s redeployment times can be shortened in many, although not necessarily trivial, ways.

In this post I’ll try and describe how I got my application to redeploy in less than a hundred milliseconds!

What is involved in a SlimWeb app redeployment?

  1. Checking and gathering of external dependencies
  2. Compiling your code
  3. Preparing the Java Web Application folder tree
  4. Zipping the web application tree into a .WAR file
  5. Deploying the app to a web container (Tomcat, Jetty, etc)

This is what more or less happens when you type ‘ant deploy’ in your project’s folder.
The default SlimWeb configuration already optimizes this process to a significant extent.

  1. The external jars gathering happens only the first time you build your app. Once you have all the external dependencies and they are up-to-date, the first step of the build process does nothing.
  2. Building is incremental, so only the modified files are rebuilt as you would expect. However, if you use Eclipse, you can use it’s built-in compiler that compiles code as you type. When you hit ctrl+s to save your source file, Eclipse has already compiled it and simply saves it to disk, eliminating the need for the ant task to do any compilation itself.
  3. Every SlimWeb project has a ‘/WebContent’ folder that serves as the root of the web app tree. When you edit JSP, TAG, JS, CSS or image files, you edit them directly in this folder, so no copying is required, but, more importantly, no redeployment is required, when these files are modified. Redeployment only happens when external dependencies (which are automatically copied to ‘/WebContent/WEB-INF/lib’) and/or resource and .class files (copied to ‘/WebContent/WEB-INF/classes’) are modified.
  4. During development you can skip the .WAR archive. You can deploy the exploded WebContent folder directly in your web container. On Linux and MacOS you can create a symlink in TOMCAT_HOME/webapps. On non-Unix platforms you’ll need to edit your web container’s XML configuration files. Refer to your container’s manual for details.
  5. Your web container will by default reload a web app if a special file’s modified date changes. For Tomcat that special file is ‘/WebContent/WEB-INF/web.xml’. For Jetty it’s the context file for your app, something like ‘JETTY_HOME/contexts/yourapp.xml’.
    SlimWeb will automatically determine when a redeployment is needed (only if resource, class or jar files were modified) and touch the special file, triggering a redeployment in your web container.
    This step takes the longest time in the entire modify-build-deploy cycle. See below for some hacks to speed it up.

That’s how the default setup works and achieves redeployment times of a second or less (that’s when resource, .class or .jar files are modified).
Note that when you modify stuff outside of ‘/WebContent/WEB-INF/’ no redeployment occurs and changes are visible immediately.

Now let’s do some hacking

DISCLAIMER: The hints that follow may cut your redeployment times dramatically, but are not for the faint of heart! Unless you’re able and willing to modify your web container configuration, your app’s build scripts, and your Eclipse project’s configuration, you should wait for SlimWeb 2.0 where some of these ideas may find their way into the framework itself, and hopefully become a little bit more developer-friendly.

You can greatly speed up reload times within your web container by moving some .jar files from ‘/WebContent/WEB-INF/lib’ to your web container’s lib folder. For Tomcat that’s ‘TOMCAT_HOME/lib’, and for Jetty it’s ‘JETTY_HOME/lib’
The more jars you move this way, the faster your app will redeploy. When you move a file to your web container’s lib folder, make sure your app’s ant script no longer copies it to ‘/WebContainer/WEB-INF/lib’. You’ll need some ant knowledge to do this. The smaller your WEB-INF/lib folder, the faster your app will redeploy.

WARNING Only do this for external jar files. Do not attempt this trick for your own .jar files.

HINT This works best for “runtime dependencies”, i.e. jars that are only needed at run-time, and not at compile-time. The best candidates are your JDBC driver, and the JPA implementation jars. You can also do this for your “compile-time” deps, but you’ll need to further edit your ant script, so that it can find the compile-time jars which are no longer in ‘WEB-INF/lib’, but also you will need to configure one or more User Libraries in Eclipse, so Eclipse itself can find them. By default both ant and Eclipse look for compile-time jars in ‘WEB-INF/lib’ so be careful when you move stuff away from that folder.

The Magic redeployment hack

If you’re using Linux (the reload.py script requires pyinotify, which is only available on Linux!) you can go one step further and disable the ant part of the build process altogether, relying on the Eclipse compiler only.
To do this you can follow these steps:

  1. Run ant build at least one so that dependencies are initialized correctly, resource files are converted as needed, etc.
  2. Go to ‘your_project/build’ and rename the ‘classes’ folder to ‘classes.bak’
  3. In the same folder create the following symlink: ln -s ../WebContent/WEB-INF/classes
  4. Launch the magic reload script: SLIMWEB_HOME/bin/reload.py
  5. Edit a source file in Eclipse, and hit ctrl+s.

On a Core2 machine, your app should automatically reload and be ready for testing 50 msec later :-)

Announcing SlimWeb 0.9.10

Friday, February 19th, 2010

The latest build of SlimWeb is available here:

http://download.slimweb.net/devel/slimweb-0.9.10.tar.bz2

Release highlights:

  • More complete samples
  • A basic JSON-to-Java marshaller
  • Fixed the response.getOutputStream() bug
  • Various minor bug fixes

Announcing SlimWeb 0.9.6

Thursday, October 29th, 2009

The latest build of SlimWeb is available here:

http://download.slimweb.net/devel/slimweb-0.9.6.tar.bz2

Release highlights:

  • Painless JSON output
  • Tiles2 integration
  • Fixed the bad-property-kills-entire-instance bug
  • More sample projects

Announcing SlimWeb 0.9.3-beta

Wednesday, August 5th, 2009

SlimWeb 0.9.3, codenamed “Simple things should be simple. Complex things should be possible.”, is available for download:

http://download.slimweb.net/devel/slimweb-0.9.3.tar.bz2

Introducing SlimWeb

Monday, July 13th, 2009

J2EE development has always been painful. The tedious plumbing required at every step, the error-prone maintaining of countless XML files, the slow build times, and even slower deployment times, the JAR hell, the portability issues between the myriad of J2EE implementations… have often made me wonder if J2EE is worth the time I have invested in it.

Enterprise Java Beans suck. The 3.0 version has improved things a lot, but they still suck. JSF is complex, and ridden with its own problems. Some JSF implementations (like ICEFaces) are immature, and others (like Woodstock) are dead. Whatever productivity you gain by drawing pre-packaged components using a visual designer are absolutely lost in deployment, performance, compatibility and stability problems, not to mention the pain involved when you need to do something outside of the component author’s original intentions. You can’t win. JSF is a good idea in theory, but a huge disappointment in my experience.

It is amazing how easy and straightforward the development of the same web-based, AJAX-enabled, relational database-backed applications is with TurboGears or Ruby-on-Rails. How did the creators of these frameworks solve all the painful problems, without sacrificing flexibility at the least? Why is it so easy to use any third party JavaScript library to add client-side functionality to a TurboGears web application, but it’s virtually impossible to do the same with an ICEFaces one?

Why do we poor J2EE developers need to suffer the full J2EE stack, while rubyist and pythonistas enjoy their cool high-productivity frameworks?

Here’s my answer: we don’t!

Not anymore :-)

Checkout my new project which will hopefully provide a better alternative to good old J2EE.

Less pain, more productivity.

I can only hope than JSF version 3.0 will be more like SlimWeb than JSF 1.2! That is, if SlimWeb (or another framework in the same vein) hasn’t made JSF and hopefully EJBs completely irrelevant by the time the next versions of their specs are finalized and implemented ;-)

Gapless Playback of Multiple Videos with MPlayer

Saturday, January 10th, 2009

MPlayer has the reputation of a stable, full-featured, flexible and portable audio/video playback software. The greatest thing invented since the wheel.

But I encountered an annoying issue on my very first day of my new project while trying to play a sequence of video files with MPlayer on a Linux box. There would always be flicker between the videos.

Fortunately, there’s a nice solution to this. Unfortunately I tried tons of options until I discovered it. The solution is the -fixed-vo argument. The best you could do without it, is something like:

In a fresh X instance (without a desktop environment):

xsetroot -solid black

mplayer -fs -rootwin -playlist some.m3u

This almost works. It only leaves a single black frame between videos. You can even restart MPlayer (to avoid a long running MPlayer process) between videos and the black frame is still hardly noticeable.

Of course -fixed-vo looks even better, as there is absolutely no flicker between the videos, and it works great with or without a desktop environment. That’s any desktop environment. The disadvantage, of course, is that at this point I’m not sure how a few weeks old Mplayer process that has played thousands of short videos will behave. Let’s hope MPlayer/XOrg can handle this…

P.S.

Here’s how my app invokes mplayer: it opens a pipe to the following command:

/usr/bin/mplayer -fs -fixed-vo -slave -idle -nolirc -msgmodule -msglevel demuxer=9:statusline=0

you can then write to the stdin of the pipe commands like: loadfile ‘/path/file.avi’

you can watch the pipe’s stdout for a message like “DEMUXER: ds_fill_buffer: EOF reached (stream: video)”

Now you know when the file you began to play finished.

The Vista UAC (mis)feature

Friday, November 7th, 2008

User Account Countrol. Advertised by Microsoft as the greatest security improvement since the switch to NT from Windows 9x.

But is it?

And great for whom? The users? The independent software vendors?

Let’s see.

A user is presented with two extra clicks when they need to copy a file to a protected folder with Windows’ own explorer. Not one, two! I guess this is usability – Microsoft-style!

A user wants to run a new program. He’s not quite sure how safe this new program from this new vendor is. It says it needs privileges. The user has to either give their consent or the program will not run at all. In my observations 99% of the users click continue 99% of the time. What protection does the UAC offer then to most users? Not much. But it sure as hell takes its countless click toll on your productivity.

But, hey, we all know that UAC was never meant to protect the poor old end users in the first place. It was in fact introduced to punish sloppy software vendors, which have created their programs to run with administrative privileges all the time. The Microsoft plan, apparently, is to persuade them to redesign their mess in a more secure manner, by splitting the sensitive parts that really need admin privileges from the rest of the software. A sufficiently annoying prompt, it seems, is the Microsoft way to achieve this.

But the product I’m working on is not a mess. It was developed and runs fine on pre-Vista platforms. It accesses files and registry keys carefully secured via ACLs. Only for me to one day wake up and find out that the respective components run with a ‘virtualized’ registry on Vista and changes to the registry are lost as soon as the process completes and are never seen by other users. You need full admin privileges to touch HKLM in the registry. What was all of my ACL pain for then?

The product I’m working on also keeps the password to a privileged account on the local system. It manages that password in a secure manner, very careful not to expose it as plain text in the process address space (e.g. in the .net parts of the system the password is always kept in a System.Security.SecureString). When it needs to perform a sensitive task it impersonates the privileged account, performs the task, then drops the privileges immediately. Simple. Flexible. Secure.

Doesn’t work on Vista!

The quick solution was of course to run parts of the system via the “runas” mechanism. Which immediately reduces security significantly, because now entire components run privileged, instead of a few lines within the component which actually need those privileges. But wait, it gets worse.

Due to the spectacular lack of proper software installation and updating mechanisms in all Windows platforms in existance, our product (like so many others) has its own. Our updates system will detect that a new version is available, will copy a few assemblies into a temporary folder and will execute the “updater” from there. This is needed because of another glaring Windows shortcoming – you can’t overwrite files which are currently in use (without a restart). What’s worse our update mechanism is MSI-based (yet another great Microsoft technology!) and MSI will ask you to close the “updater” and then try again. Whatever. Throughout the development of the product we’ve worked around all of this brain damage. But here’s the cool part: once the updater has run (after being elevated thru an UAC prompt of course) it wants to restart the calling application, whichever that was, as the standard (non-elevated) user that originally launched the initial app. How do you do that in Vista? You wouldn’t believe. But what if the privileged account was different than the original unprivileged user that started the application in the first place? You’re out of luck!

So all I want to say is,

“Thank you, Microsoft! That you for your wonderful products, and for the way you’re treating me as a developer!”

Fedora 9 Sulphur

Tuesday, May 13th, 2008

I’m downloading the Fedora 9 Sulphur release official DVD via bittorrent even though the official release is still a few hours away :-)

The official tracker says “Requested download is not authorized for use with this tracker” but any decent bittorrent client should be able to find enough peers through DHT.

Here’s the .torrent file.

libshout on Win32

Friday, February 8th, 2008

The official libshout-2.2.2 release doesn’t build out-of-the-box on Windows.

However, with some effort you can get it to build as a static DLL on Cygwin (if you undefine _WIN32 everywhere and unset some options in config.h)

With even more effort you can build a native Windows .DLL with VisualStudio 2005 but you’ll need to make (at least) the following changes to the project:

  1. Get all the dependencies: ogg & vorbis from xiph.org (they should build in VS2005 after migration from VS6 without problems). You’ll also need Win32 pthreads – look it up on Google (the distribution already contains pre-built VS DLLs).
  2. Configure the VS2005 solution so that it can find the header and .lib files for ogg, vorbis and pthreads.
  3. You need to manually add vorbis.c to the project after the migration to VS2005.
  4. Fix compilation errors (i.e. define size_t in vorbis.c, replace va_copy with memcpy, fix void pointer arithmetic)
  5. Mark the public API functions as __declspec(dllexport)

I’ve successfully used the so produced DLL with my ctypes-python-shout bindings to connect to a Windows Icecast instance – therefore having the whole stack running entirely on Win32.

I’ll post more details as well as a both binary and source (VS2005 solution) snapshots of libshout 2.2.2 (and all dependencies and required fixes), as soon as I can.