Choosing an app Server

Choosing a J2EE app server is one of the most important IT decisions an organization will make. Adopting J2EE is a strategic decision, so how it's done will have a lasting effect. Many app servers are expensive, requiring significant outlay on licenses for large installations. However, the cost of the server is only a part of the story. Even when choosing a free server such as JBoss, we are making a significant commitment. Expenses over and above license costs are likely to include:

The virtue of having a marketplace for servers is that different servers fit different requirements. Comparisons rapidly become outdated and misleading, and can't be kept up-to-date in tutorial form. Hence we'll discuss the issues to be considered while making a choice here, rather than recommend or compare products. In this section we look at:

Note that the ideal is to standardize on a single app server throughout an organization. Maintaining multiple app servers will prove expensive throughout the software lifecycle, and should be avoided if possible. Usually, the use of several different app servers in an organization reflects historical accident or lack of a coherent strategy. One of the few valid technical reasons for running more than one app server in an organization is when a few apps depend on a particular feature that is unavailable in the app server normally used: for example, when one app requires features from the latest release of J2EE and therefore must run on a recently released app server, while most apps run on a proven app server. In this case, the long-term aim will usually be to return to the use of a single server.


In many cases, the app server will have been chosen before the project begins. For example, there may be an organization-wide policy regarding app server choice. In this case, try to work with the chosen server. Don't introduce another based on your or your team's preferences, complicating the organization's technology mix. If the chosen product is demonstrably inferior, and it's still realistic to abandon it, lobby to have the choice changed.

When to Choose an app Server

It's wisest to choose an app server early in the project lifecycle. This has the following advantages:

It's important to recognize that the potential to port J2EE apps between servers does not mean that a J2EE server can be chosen at leisure. The same considerations apply as to other important enterprise products such as databases. The longer a choice is deferred, the more familiarization time is wasted and the less potential there is to make use of valuable features of the eventual choice. The only advantage in choosing a server late is that the choice may be made after your team has gained J2EE experience. This is only a consideration if all team members are unfamiliar with J2EE, which itself is an unacceptable risk. Address this problem by hiring some J2EE specialists early in the project. A small pilot project is often a good idea to familiarize a team with both J2EE and a candidate server. Implementing a "vertical slice" of the planned app is also a good way of mitigating many other risks, such as inadequate performance.


Choose an app server and other important software products such as a database as early as you can. Attempting to defer a choice because "J2EE apps are portable" is a recipe for wasting time and money.

While we should choose an app server early, we should still try to ensure that code is portable between app servers wherever possible. Working with a single server from early in the project lifecycle does mean that it is easy to write code that violates the J2EE specifications without realizing it. For example, many EJB containers allow violations of the EJB specification. However, undetected violations are a major cause of portability problems. There are good reasons why total code portability isn't always possible, but ignorance and carelessness aren't among them. One strategy to avoid such inadvertent violations of the J2EE specifications is to deploy and test the app regularly on another server (it is much less likely that two servers will fail to detect violations). However, this strategy demands that resources are devoted to configuring the other server and mastering its deployment process, not merely testing the compliance of code. The J2EE Reference Implementation server is sometimes used for this purpose. However, as the Reference Implementation server is slow to start up and buggy, the open source JBoss server is probably a better choice if adopting this strategy. JBoss 3.0 does tight compliance checking and offers particularly simple app deployment. A better strategy is to use the verifier tool that also ships with the J2EE Reference Implementation (available from to verify the compliance of all J2EE deployment units - WARs, EJB JARs, and EAR files. This tool gives excellent and helpful output. It can be invoked with commands like the following:

 cd J2EE_SDK_HOME/bin
 verifier -v /path/to/ejb-jar-file.jar

The -v flag causes verbose output of all tests. The output will indicate either compliance (no failed tests) or list those tests that failed. A log file will be generated. The following is an example of log output on a failed test: Test Name: tests.ejb.session.ejbcreatemethod.EjbCreateMethodFinal
Test Assertion: Each session Bean must have at least one non-final ejbCreate method test
Detailed Messages:

For EJB Class [ facade.ejb.JdbcReportEJB ] method [ ejbCreate ]
 Error: A final [ ejbCreate ] method was found, but [ ejbCreate ] cannot be declared as final.

Verification can-and should-be integrated into the app build process, for example using the Jakarta Ant build tool (discussed below).


Regularly test the compliance of J2EE deployment units with the J2EE 1.3 Reference Implementation verifier tool. Integrate such verification into the app build process.

Defining the Requirements

It's important that the process of server selection should be planned and transparent. Let's consider some of the steps and decisive criteria. The first goal is to define the requirements of an app server. Unfortunately, few organizations do this. Many allow their selection process to be driven by vendor presentations and what they perceive as industry buzz. It's important to get past the marketing hype and to base the decision on the present and probable future needs of your organization. Key criteria include:

We'll take a closer look at some of these below.

Evaluation Criteria

Let's take a look at some of the factors that should determine server choice.

Supported Specifications

Policy on specification versions (discussed above) will dictate how important a requirement is support of the latest J2EE specification (remember, however, that J2EE specifications are normally backward compatible - in the case of J2EE 1.3, wholly so):

Sun Resources

Sun, as the custodian of the J2EE specifications, runs two app server evaluation programs: the Compatibility Test Suite and ECPerf. While the existence of these programs is important, as we'll see we cannot rely on them to guide our choice of server. Let's look at the aims and practical value of these programs in turn.

The Compatibility Test Suite (CTS)

Sun publishes a J2EE CTS and maintains a list of products that have satisfied its requirements at Products on this list may be marketed as "J2EE Compatible". The J2EE 1.3 CTS includes over 15,000 tests covering all classes and methods in the J2EE APIs, and interaction between J2EE services. The CTS is a welcome development. In the early days of J2EE, no such guarantee was available, and servers often failed to deliver workable implementations of core J2EE features. The existence of the CTS motivates vendors to implement the J2EE specifications faithfully. It increases our confidence in our ability to port standards-compliant apps between servers. However, the CTS is of limited help in choosing a server and cannot be used as a definitive guide, for the following reasons:


Performance and scalability issues are addressed by ECPerf, another initiative from Sun developed under the JCP. ECPerf is a benchmark app that tries to replicate real-world issues, for which performance results are independently certified. The ECPerf app is based on a relational database and uses CMP entity beans. The main ECPerf home page is at ECPerf results are hosted at The ECPerf app can be downloaded from the ECPerf home page. As with the CTS, however, many servers have not submitted ECPerf results. As of August 2002, only five servers-WebLogic, WebSphere, Oracle, Sybase, and Pramati-are represented. ECPerf is useful to a point, but it's geared towards demonstrating performance on particular hardware configuration and with a specific RDBMS. For example, it's possible to compare the performance of WebLogic on a cluster of HP servers using Oracle 8.1.7 with that of WebSphere on an IBM cluster using DB2. But this is no help if we want to predict the performance of WebLogic and WebSphere on the same hardware using the same database (typically hardware availability is largely predetermined and the RDBMS will already have been selected before we select a J2EE app server, so this is what we really need to know in practice). Also, ECPerf results have limited relevance if we use different app architecture such as session beans with JDBC or JDO instead of entity beans. The Australian Commonwealth Scientific and Industrial Research Organization (CSIRO) provides a commercial performance comparison of some leading app servers, available at Unlike ECPerf, this does compare app servers on the same hardware.


Beyond their use in comparing app servers, ECPerf results are a valuable resource for performance tuning information. ECPerf results must fully document any modifications to default server configuration, such as JVM options, number of EJB containers used, and operating system settings.


J2EE app servers vary widely in license cost-from zero for JBoss to hundreds of thousands of dollars for a large installation of leading commercial products-and in how license cost are calculated (such as per CPU or per physical server). In one episode of the wonderful 1980s British TV comedy "Yes, Prime Minister", the scheming civil servant Sir Humphrey Appleby attempts to persuade Prime Minister Jim Hacker to buy a particular type of nuclear missile irrespective of cost (and strategic requirement) because it is simply the best. Sir Humphrey's argument concludes with the stirring phrase "it is the nuclear missile Harrods would sell". In boom times like the dotcom frenzy of the late 1990s, such arguments carry great weight with senior management. Sir Humphrey as a CTO or CIO (roles in which he would be at home in many companies) will argue that XYZ server is the most expensive, and therefore the best, and that it would reflect lack of confidence for the company not to buy it. He might add, with more justification, that the market expects his company to invest at least as much as its competitors in technology products. The board would probably be more skeptical about an open source solution. This (admittedly cynical) analysis, based on experience, is borne out by industry analysts such as the Gartner Group, who estimated that by late 2001 companies had wasted well over $1 billion on high-end app server technology they didn't really require (see In the leaner times of 2001/2, companies are loath to spend money on software products, even to make a strategic investment. There's a tendency to spend as little money as possible initially, while leaving open the possibility of migrating to a more expensive platform, should it become necessary. This can prove even more expensive than overspending up front. app server costs will only account for a fraction of a project's total budget; development teams cost much more to run. The cost-in customer perception, and staff time and morale-of limping along with an inadequate solution may soon outweigh any initial savings. Also, the uncertainty of a possible migration is a recipe for frustration and paralysis.


Cost should not be used as the measure of an app server's worth. The true test is whether it meets the organization's needs. Remember to consider total cost of ownership. Outlay on an app server should be only a small proportion of total cost.

Vendor Relationship

Committing to a particular J2EE server means committing to a continuing relationship with the server vendor. The quality of presales advice may indicate the quality of the vendor's experts, but the organization's level of commitment to its customers is important. Some points to consider here are:

Open source products offer a different mix of support positives and negatives from commercial products. On the positive side:

On the other hand, no company can be forced to take responsibility for fixing a problem. Open source developers are normally proud of the product they've contributed to and are often very helpful in fixing bugs and resolving users' problems. If your posting of a problem results in you being flamed rather than helped, you'll have no one to complain to. The archives of the JBoss mailing list display both these positives and negatives.

Open source support also tends to be more democratic than that provided by commercial products. Postings are likely to be given equal attention, whether they come from a single developer, a small or a large organization. Commercial server vendors, on the other hand, are likely to pay more attention to their relationships with key purchasers, meaning that large organizations that have downloaded multiple or enterprise-wide licenses are likely to receive priority attention. This is one of the reasons that large enterprises rarely choose open source products.

Vendor Viability

Since its peak in 1999/2000, the app server market has experienced a shakeout due to an excessive number of players and difficult trading conditions. Several smaller players have withdrawn from the app server market to concentrate on their areas of special expertise, and this trend seems set to continue. Thus, it's important to consider the viability of an app server when making a choice. In the worst case, what would happen if the product were no longer supported? This is especially dangerous if you have relied heavily on a particular capability of that server.

On the positive side, a small company may prove to be more responsive and more concerned about the needs of individual customers, and especially small individual customers.

Development and Deployment

Another important criterion in choosing a server is how easy it will prove to develop and deploy apps on. Ideally the server should not impose onerous proprietary requirements when developing and deploying apps, and should be quick to start up. It's essential that new releases of software be rolled out without major impact. Ideally, it should be possible to do a release without shutting down the server. Development and deployment considerations include:

Value-added Features

As more servers become J2EE compliant, reliability, scalability, and value-added features increasingly differentiate app servers. As the J2EE specifications don't meet all common requirements, this is inevitable. Value-added features that may prove important include:

Quality of Documentation

Most vendors provide excellent free online documentation. This is a valuable resource during server evaluation. It's also a good idea to consider the availability of tutorials and other independently authored resources relevant to the server. This consideration tends to favor the market-leading servers.

Availability of Skills

It's important to ensure that a significant subset of a project team has hands-on experience with the chosen server (rather than just J2EE), and that it's possible to hire more developers with experience using it. It's possible to measure the availability of server-specific skills by checking job sites for requirements from other companies; checking the popularity of any certification program run by the server vendor; and looking for local user groups devoted to the server (however, a developer with a deep understanding of J2EE will soon get up to speed with any server, and will ultimately outperform a developer with knowledge of a particular server but a shallow grasp of J2EE concepts).

Again, this consideration tends to work in favor of the market leading products. Remember that, because agencies and employers tend to look for products (such as WebLogic) rather than technologies (such as EJB) on resumes, many developers are wary about working on projects using obscure products they don't expect to have a big future.

User Experience

It's vital to talk to staff at organizations using each server you evaluate. Vendors should be able to provide reference sites. If possible, it's good to have informal conversation, without any salespeople present. It may be more revealing to talk to individual developers than to organize formal conference calls. Frustrating problems are more likely to be mentioned when talking off the record. Most server vendors operate newsgroups or mailing lists. These are usually freely accessible, and provide a valuable insight into the experience of working with the products.

The following links to newsgroups may prove a useful starting point:

Choice Process

I recommend having a semi-formal process for app server choice to ensure that it is transparent and justified by the available evidence. Too much formality will result in endless committee meetings and the production of high-level documents divorced from reality. Too little formality will result in an ad hoc decision and inadequate documentation in case the decision is questioned in future (future deliberations may inadvertently cover the same ground). The development team should be heavily involved in the final choice. I suggest the following inputs for consideration:

Top Mistakes in Choosing an app Server

In my experience, the following are some of the commonest (and costliest) mistakes in choosing an app server: