Invariant Properties

Here’s something I’ve been thinking about for the last year or so. No answers, just questions.

What does the typical small development team’s resources look like? I’m thinking of anything from a single person working on side projects at home to teams with a dozen or more members in an office.

There’s typically few overworked servers grudgingly provided by the IT group. The physical servers host numerous services:

• continuous integration build server. We require one build server per supported OS flavor if we distribute the software – this can add up quickly. (Windows XP, Vista or 7? Redhat, Ubuntu or CentOS? Mac? 32-bit or 64-bit? What about smart phones?)
• database server (for CI results)
• web server for CI build
• web server for daily build (testing)
• web server for regression tests (where appropriate)
• database server(s) (for web servers)
• subversion server
• bug/issue tracking
• database server (for bug/issue tracking)
• local maven repository (if you use maven)

Some of these servers are resource-intensive, others aren’t. E.g., subversion just doesn’t require that many CPU cycles or disk space. The maven repository might have a database backend but otherwise it’s also a low load.

The first approach everyone takes is to merge services. One database server shared by all applications. One appserver hosting multiple applications. That drops the requirements substantially, right?

In practice the services start bumping into each other. You can lose all services while doing maintenance on one. Occasionally a critical fix on one service will require a database bump that breaks another service.

The second approach is to migrate the less resource-intensive servers onto virtual machines. This gives good separation but comes with its own costs. Fortunately most of these costs are rapidly dropping (better processors, cheaper memory, better software) but you still have to establish requirements, set up the virtual servers, etc.

The third approach that I’ve recommended but never seen put into practice is to move some of these tasks into the cloud. Do we really need to have build servers on local hardware? This is a cheap way to quickly expand capacity – we don’t have to fight for capital budget since the cost of a cloud server isn’t that high if we only run it during working hours.

There’s a secondary benefit of standardizing our virtual servers. We don’t have to think about how much space to allocate, we don’t have to load the operating system ourselves, we just select one of a dozen standard configurations, grab an image, and we’re running in minutes.

This approach runs into a management roadblock that reminds me of the earliest days of the web. This is new and the risks are unknown from non-technical management’s perspective. We see large companies with thousands of clients, they see “TO THE CLOUD!” ads by Microsoft. (I still have no clue what they mean by that.)

This brings us to my current thoughts. Running virtual servers isn’t the only thing that’s gotten easier in the last few years – you can make your own private cloud with open.eucalyptus.com and other tools. You can also create hybrid clouds.

(Some quick definitions – a ‘public’ cloud is hosted by a company that offers services to the public – Amazon, Rackspace, etc. A ‘private’ cloud resides on a company’s own hardware. A ‘hybrid’ cloud is hosted on both types of hardware.)

This means it’s now easy to manage your development servers. It’s a few clicks to move a server from one node to another node if the first node gets too busy. It’s a few more clicks to move services to a new server if you get new hardware.

You can expand this idea – make every developer’s workstation a node. It has no available slots during the workday but can be used in the evening and weekend for additional tasks. E.g., perhaps use them for load testing from midnight until 6 AM. This would be time-consuming if you do it manually but with a cloud it’s a few quick cron entries.

Does this mean the servers will no longer be overworked?  Of course not – you’ve moved everything into virtual servers and added another service. Does that mean this is a pointless exercise? I don’t think so – I think it will be easier to manage standard virtual instances in a cloud than to manage the mishmash that usually develops. It will definitely be easier to bring up and shutdown servers as necessary, e.g., to bring up a legacy server for regression tests.

But as I said I only have questions, not answers. I think it’s worth pursing but don’t have hard numbers to prove how good or bad it is.