The Standish Group

Sample Research Paper

SUN Also Rises: Solaris Vs. NT

"A zeal for different opinions concerning religion, concerning government, and many other points,… have, in turn, divided mankind into parties, inflamed them with mutual animosity, and rendered them much more disposed to vex and oppress each other than to cooperate for their common good…"

James Madison

It's a political issue; it's a personal issue; it's an issue that plagues the minds of IS executives everywhere. Which operating system should I choose? Recently The Standish Group was asked by a CIO of a large publishing firm, "Can you help get beyond the politics within my IS staff to choose the correct operating system?" This CIO was caught like a fly in multiple spider webs. He needed an answer, but each time he tried to talk to his staff, he realized the question was slanted by prejudices beyond his control. It was no longer a question of which operating system could perform best; it became a war between the UNIX group (proclaiming themselves the "open team") and the Windows group (referred to as "Microsoft groupies").

Rather than expound the benefits of the operating system they were using, the UNIX group often proclaimed freedom and openness. Meanwhile the "Microsoft groupies" argued, out with the old and in with the "new operating system."

This user is not alone. Fighting through the alternative "religions" to find an answer has become difficult at best. Recently Microsoft pinpointed one UNIX vendor in particular to target for competition; that vendor is Sun Microsystems.

In the LAN workgroup server market, the Intranet/Internet Web server market, and increasingly in the departmental database server market, Microsoft Windows NT, and Sun Solaris systems are competing head-on. Sun Solaris has only competed in the enterprise (primarily database) server market with HP, DEC and IBM systems without worrying about serious consideration by IT organizations of Microsoft Windows NT servers. However, this grace period may be short-lived, since Microsoft is currently pushing NT Server Enterprise Edition with Wolfpack.

Sun and Microsoft actively compete in several markets. Sun is the only major UNIX vendor which lacks a relationship with Windows NT. On the desktop they have widely different strategies - with Sun favoring the thin-client model, the Web and Java; and Microsoft developing for a Windows-centric fat-client PC (despite a great deal of lip service to Java, the Internet and the Web).

Sun is synonymous with the Internet and TCP/IP ("The Network is the computer"). Sun produces servers sold primarily for Internet use (the Netra Internet Server), including many of the first web servers. And Sun developed the Java language which is currently the subject of much hype in the IT community.

In addition, after some initial confusion, Sun has vowed again its strong support for the Object Management Group's (OMG) CORBA initiative. This support for both CORBA (and its Internet Inter-ORB Protocol - IIOP) and support for Java has highlighted Sun as the "open" choice.

Microsoft is a recent convert to the Internet and its standards processes. It has a desktop strategy still very much tied to ActiveX technology, which is descended from the proprietary and older Microsoft OLE and COM technologies.

Both Sun and Microsoft have their own network services for file service (NFS in the case of Sun, and SMB/CIFS in Microsoft's) and remote procedure call (RPC) mechanisms (Sun RPC vs. the Microsoft NT DCE RPC variant). However, it is possible to obtain third party products which allow them to interoperate as both clients and servers to each other. At the layer below these application protocols, both suites utilize TCP/IP transport and network protocols.

Microsoft and Sun have both adopted and proclaimed adherence to Internet standards. Both have also created standards which have been adopted by the IETF and other vendors. Both also continue to bump heads with each other: Sun has proposed a modified version of NFS running on TCP, called WebNFS as a file system protocol for use by Web browsers. Microsoft, on the other hand, is pushing the network file system protocol served by Windows NT as a primary means of accessing files across the Internet, and has renamed it CIFS (Common Internet File System protocol).

Solaris machines are still a choice of ISPs (Internet Service Providers) for providing Internet services to their customers, especially where "shell" access accounts with "telnet" login is required. To counter this, Microsoft has embarked on a program to prove an ISP can be successful with a 100% pure NT shop. They have touted case studies to demonstrate that a pure NT strategy works for an ISP in the hopes of replacing Sun at many ISPs.

UNIX (especially Solaris) machines are still the mainstay of the firewall market for securely connecting companies to the Internet. Many popular firewall packages (Checkpoint Firewall-1, SunScreen, Borderware) originally came out on the Solaris platform. But most proxy and "stateful inspection"' (a.k.a. dynamic, a.k.a. intelligent) packet filtering firewall vendors have hurriedly ported their products to NT. Being chased at the low end and middle tier, Sun had to move Solaris up into the strata formerly occupied only by mainframe-class machines and operating systems. Sun proved agile at developing hardware that was both reasonably fast (the UltraSparc) and scalable in SMP configurations.

In the enterprise database server market, as well as in the specialized MDDB (multi-dimensional database) server market, Sun reaps profits (and market share) with high-end multi-processor systems running Solaris. Sun combines hardware technology and solid software support to compete with DEC and HP in terms of processor speeds, and with IBM in terms of scalability.

With no legacy baggage to support, Sun has been able to vigorously undersell more traditional vendors. It can engineer hardware solutions at a lower cost than the others, and its cost and profit structure is based on a much leaner, smaller, strictly client/server and Internet oriented computer company.

Today companies are changing the way systems and people talk to each other on the network. It's no longer about LAN vs. WAN. It's now Intranet vs. Internet. Concurrently, there is no "normal" way of moving to the Internet; just as many companies start with an Internet application as do those starting from an Intranet perspective. The real difference appears in how applications and systems are managed. With an Intranet environment there is still a reasonable sense of "the norm." Systems managers know when peak levels are reached, and what those levels will be. As companies move to an Internet environment, this is far more difficult to forecast. The key to competition on the Web is to deliver new services, as many as possible, and faster than ever before. The key to remaining competitive is grounded in critical systems services support (such as availability and security).

The Standish Group's server operating system selection criteria are a weighted summarization of approximately 250 individual criteria grouped into 12 main areas. The individual criteria and main areas are weighted to properly reveal optimal performance and selection for a server operating system. More than simply a feature/function comparison, The Standish Group combined feature/function reviews with the results of thirty (30) personal interviews for our final scores. All interviewees were using both NT and Solaris. The companies interviewed ranged from small to large corporations covering a sampling from multiple industries and geographic areas.

TOP TEN CRITERIA

An operating system's ability to gracefully support explosive growth while remaining stable and available is crucial to the success of a company's Internet applications. The Standish Group has identified ten selection criteria when considering an enterprise/Internet/Intranet operating system, and will compare Solaris vs. NT using these ten as a guide:

Reliability: Reliability is the measure of how consistently the machine runs the applications and services that are its purpose. The integrity of the data in memory, as well as that written to disk is extremely important. A file, disk or database system which allows even a small percentage of corruption would cause severe adverse effects on business functions; and a system with chronic problems in this regard would quickly become worthless in any corporation. Basically an operating system's reliability is measured in two ways: the integrity of the information, and its ability to recover and repair damaged information. These things certainly include integrity checks, correction algorithms, both synchronous and asynchronous rights, and the ability to perform lock management.

Security: Security is the ability of the system to protect itself from denial of service attacks as well as intrusions. After protecting data from accidental damage in the reliability criteria, how well data is shielded from willful destruction or modification is paramount. Modern computer security must also be concerned with protection of the privacy and integrity of data traversing networks (LANs, WANs, Intranets and the Internet). The security of an operating system is the major barrier preventing all-out commercialization of web transactions. This means, by necessity, any operating system today will have cryptographic functionality added to it for encrypting, check summing, and "signing" data and control messages. The Standish Group looked at 16 discrete security items to measure the security levels of the two operating systems in this paper.

Scalability: There are two ways for an ISP to go to bankruptcy: he can drive in a Volkswagen Bug or a Mercedes. The guy in the Volkswagen will go because he didn't have enough capacity; the guy in the Mercedes will go because he had too much. By having a system that can scale, one can prevent the too little or too much syndrome. We measure scalability on several fronts, such as the number of processors supported, the number of file sizes supported, bit structures supported, and the logical file systems supported. More importantly, we assess the ability to adapt to a growing environment without having to prepay for over capacity.

Availability: In electronic commerce more and more applications require 24 x 7 availability. The ability to recover or to prevent a failure is becoming more and more important for an ISP. So availability is measured on several fronts including automatic restart, failover to another processor, or fault-tolerant operations.

Manageability: Manageability represents the set of features which automate the operation of the server, as well as leverage the few skilled personnel responsible for administering the machine and the OS. Installing new system and application software and maintaining and monitoring it are prime factors requiring human intervention once a server is set up. The Standish Group measured the manageability features and functions using 19 different points. We included things like system monitoring tools, GUI based administration, SNMP support, and available systems management products. Another issue which must be considered is the environment in which the systems need to reside (can one use a large machine to do several things, vs. using multiple smaller machines).

Interoperability: We live in a world with the old, the present, and the future. Interoperability must be viewed from applications and environments which have been our legacy, as well as environments we are currently deploying. However we can not lose sight of the fact that the world is changing, and we need systems that can work with new environments. In order to measure the present and past we looked at almost 100 different discrete interoperability requirements. More importantly, we need to consider the philosophy behind the individual companies regarding their interoperability strategy.

Adaptability: Adaptability is the measure of how well the operating system "plays with others" in today's heterogeneous environment of legacy terminals, traditional client applications, new "browser" user interfaces, as well as devices not yet fully accepted nor widely implemented (Network Computers, PDAs, Java terminals, etc.) Adaptability is also judged by the software programming packages that can be used to develop client applications that will interface with processes running on the server, as well as the number of popular database server software implementations that run on the OS platform. The Standish Group measured adaptability by looking at almost 60 discrete products, items or standards.

Ease of Development: Ease of Development looks at several things such as the flexibility and extensibility of development tools, debuggers, testing and other types of development environments.

Affordability: Affordability is looked at in two different ways. Pricing and support are major parts, but the ability to have proven performance is paramount to any decision for a mission-critical application. It is important not to be fooled by a low initial price tag, no matter how attractive, but attend to the on-going cost to support the entire environment.

Ease of Use: When considering ease of use, one is tempted to look only at the initial installation and the GUI features. Such a superficial evaluation can lead to real issues in the daily operation of the environment. When considering an operating system, look not only at ease of installation, but also at managing the environment, ease of growth, ease of integrating different elements to the environment, keeping it secure, and so on.

SUN SOLARIS

Solaris for Intranet was released in the summer of 1997. Solaris for Intranet runs on the 32 bit and 64 bit Sparc CPUs as well as the Intel x86 chips.

Reliability Solaris 2.6 was rated very high with respect to reliability features (such as robust file system recovery and file locking). In addition, user interviewees rated Solaris as highly reliable. User case studies strongly support a high grade in this category.

GRADE: A-

Security Solaris 2.6 improves upon the security in 2.5.1 with PAM (Plug-in Authentication Modules) and GSS-API. Solaris provides multiple security options and provides SunScreen SKIP for Virtual Private Network security. Still the operating systems security could be improved through support for restricted hour of operation and transport layer security, and through support for multiple VPN security measures. We were hard pressed to find any Solaris users making broad use of the provided security for the product.

GRADE: B

Scalability As one user commented, "Boy that thing can scale!" With support for multiple SMP processors, network clustering, large file support (terabyte), etc. Sun yielded a perfect score. Solaris 2.6 begins to take advantage of the 64 bit architecture of the UltraSparc CPU to break past traditional UNIX limitations.

GRADE: A

Availability Availability is certainly a strong point for Solaris. When users were asked about downtime, many suffered loss of memory very few were able to remember if the system had ever failed. One user informed us that the last time he remembered, the system had run consecutively for 248 days. This user told us, however, that he was prepared due to a "known bug" in the system that will shut the system down after exactly 248 days. (Please note Sun has assured us this bug has been fixed in version 2.6.) This user was quite sure that had there been no Solaris bug, the system would not have shut down. He further estimated it took less than a minute to bring the system back up. The only desirable availability feature Solaris 2.6 lacks is the ability to perform an OS upgrade online.

GRADE: B+

Manageability Solaris manageability is clearly a strong suit, and the differences between Solaris and NT are striking. Solaris users can repair broken applications while other services run on the server. The repair facility provides both a local and remote repair capability. When repairs are completed, the application simply comes back online, and no reboot is required.

An equivalent NT experience would probably take the server and its users down during the repair phase, and certainly prior to returning the application to service, because NT would automatically reboot. Additional manageability considerations include the fact that the Solaris administrator need not be physically present in order to maintain the system: it can be accessed through any terminal.

GRADE: A

Interoperability Solaris 2.6 doesn't support the number of devices, interfaces and peripherals that NT does on PC hardware. Sparc hardware and drivers are much more limited for Solaris than for NT on the PC. However, Solaris operates on a wide range of Network Protocols and ODBC support is fairly strong. In addition, in terms of available enterprise software product support (such as middleware) Solaris certainly has it covered. And of course no one works better with Java than Sun.

GRADE: B

Adaptability Solaris had average marks in terms of adaptability. A good number of clients are supported, and there is strong database coverage (minus SQL server). Solaris supports CGI and Java interfaces, and has high marks for Job/Task scheduling. However, hardware platform support is limited (Intel and Sun).

GRADE: C+

Ease of Development UNIX (and especially Solaris) are still premiere development environments with superb toolsets, IDEs and CASE tools. The GUI Integrated Development environment and GUI debugger and support for multiple procedural programming languages and system-level scripting upped the score here as well.

GRADE: A

Affordability Solaris for Intranet is very competitively priced. A five user license cost $1295, but the big bonus is the unlimited license costing a mere $1295 extra. For large environments the cost savings gained using the low unlimited pricing model can be extraordinary.

GRADE: A

Ease of Use Sun improved on their installation procedures and system administration GUIs through the use of the Netra web server interfaces, by using the HotJava browser and creating Java-based installation utilities.

Solaris also provides an automated remote installation and upgrade capability, and a replicated installation capability (where one can take classes of users and do installations by class). With full remote control, an administrator need not be physically at the machine. No, Solaris is not your father's UNIX! The browser interface is slick and easy to use yet when it really counts a company can still fall back on familiar low-level stuff.

GRADE: A

WINDOWS NT 4.0 SERVER

Released in August 1996, Windows NT 4.0 server is touted by Microsoft as scalable from small systems to the largest multiprocessor PC configurations. Though Windows NT is portable (it is written in high-level programming language) among several processor chips, it generally runs just on Intel x86 family CPUs (Pentiums over 100Mhz, Pentium Pro, and now Pentium II machines).

Reliability Customers considering the introduction of NT 4.0 may look to a simple feature/function chart and consider the product high in the category; however, we looked further. We'll start with the goodness of NT in this area: the disk and file subsystems almost never result in data corruption; the OS itself performs a sanity self-check upon boot by comparing a checksum for each critical system software file loaded with a stored list of checksums for the same files (if there is any mismatch, NT will not continue the boot, and will require human interaction to resolve the problem).

However, there is a bad side to this story. When we spoke to users they reported general protection faults and memory leaks which required regular rebooting of the NT system. Therefore it is difficult to rate NT high in this category.

GRADE: C

Security NT has benefited from advances in operating systems and software engineering to avoid the legacy security problems of most vendor UNIX systems. However NT recently achieved some notoriety for the number of new security bugs discovered in it. It has particularly attracted a number of successful methods for denial-of-service attacks for which Microsoft has had to release patches repeatedly.

Compared with earlier PC operating systems (which had zero security built in) NT is reasonably secure, and offers a degree of corporate security just below that of most commercial UNIX versions (which are also able to obtain a DOD C2 rating). But it is still not on a par with the level of mainframe security NT and UNIX are replacing.

NT has come a long way from Windows, but still has a way to go. (It now has more security bugs reported each week than UNIX.) Also, the default file permissions are lax, designed not to get in the way of normal users accustomed to the typical Windows method of installing software. DLL and other files are written all over the place, especially in the system software directory.

Despite Microsoft's security efforts, the users we spoke with were uncomfortable using NT for any type of application requiring strong security, such as electronic commerce applications.

GRADE: D

Scalability Microsoft recently announced that it will be supporting NT server in eight CPU configurations, where the standard version typically had only run on dual and quad processor systems (special OEM versions from vendors such as DEC had permitted more than 4 processors). However, there appear to be a few bottlenecks on NT which prevent it from scaling linearly in performance when the number of processors exceeds eight. It remains to be seen whether or not NT 5.0 can remove this roadblock to NT SMP performance.

In addition, the current version of NT is strictly a 32-bit operating system. The Alpha NT port by DEC is the first stab at using a 64-bit microprocessor, but does not take full advantage of the wider word size. Microsoft is working on creating a 64-bit version of NT; but it will not be released for quite a while. The UNIX server manufacturers are taking advantage of this time window to flaunt the performance benefits (higher data throughput, larger addresses spaces, larger file sizes, etc.) that 64-bit architectures can provide (for example, a 64-bit Solaris architecture gives 4 billion times more data space than a 32.) Users we spoke with stated that it would take three times as many NT servers and administrators compared with a Solaris installation.

GRADE: F

Availability NT has not yet achieved the Holy Grail of high availability, and the OS still machine checks and results in a "blue screen of death." No Solaris users we spoke to experienced this type of failure. While the capability to restart processes exists and works, NT servers often crash and stay down until morning when the early staff finds them. In addition, the lack of a standard disk quota mechanism often means that the system runs out of disk space, which can also stop the operation of some services.

While Wolfpack is a great promise, in the past these types of failover systems have caused more failure than they've prevented. Also, failover scenarios only work when the underlying technology is reliable.

GRADE: F

Manageability NT Systems are easy to set up as well as to install additional software on. Tools exist to track, audit and debug NT services and applications. The system administration tools and visual interfaces are very nice. If all any company ever had to do is install an operating system, NT would be a gift from heaven. But of course life is not that easy, and neither is NT.

Our users complained of a serious Catch-22: while they loved the GUI interface's ability to get a system up and running quickly, they all complained that when problems occurred, trying to find the necessary information for a solution was often elusive. The Standish Group heard stories of endless searches through multiple screens, horrendous file systems and annoying graphics. These users longed for a simple command line interface to default on when the going gets tough. While NT does provide a simple interface for problem resolution, none of our users felt it was strong enough to be useful.

GRADE: D

Interoperability While NT doesn't support as wide a range of devices and boards on PC hardware as Windows 95, it certainly supports a wide range of standard PC hardware and peripherals. NT natively supports more compatible hardware than most UNIX OSs and drivers are usually supplied by hardware vendors for NT. NT supports most network (LAN and WAN) interfaces; however, ODBC support is very weak. Of course availability of enterprise software for NT is very strong it should be noted, however, that this support is not a two-way street. For example, NT does not inherently support DCE or CORBA this support is available only through third party products.

GRADE: B-

Adaptability NT servers support a wide range of remote clients. Though most NT applications don't provide a traditional textual interface, it is still possible to develop and run applications which do, and NT still provides the DOS command line interface with some enhancements.

NT provides some hooks for server tuning (background batch vs. foreground interactive, etc.) and priority setting. All of the popular UNIX relational databases have been ported to NT, and most of the popular multi-dimensional databases should be soon. NT is portable, although the number of microprocessor CPUs other than Intel that Microsoft officially supports on NT is dwindling.

GRADE: C+

Ease of Development The Windows NT environment gets a perfect (100) score in this area. NT benefits from the large number of tools in the earlier Windows environment as well as those written for 32 bit Windows 95. In addition, all the UNIX programming tools and scripting languages have been ported to NT.

GRADE: A

Affordability When compared with Solaris, NT often has a higher price tag, much higher for large numbers of users. Particularly when you consider the possible increase in number of machines and personnel required for operations. For Intranet environments it appears NT can be much more expensive.

That said, NT is certainly not expensive. When compared with other offerings, NT is very competitively priced. In addition, most software available to run on both NT and Solaris is invariably always cheaper on NT.

GRADE: A-

Ease of Use Installation for NT is a painless operation a couple clicks of a mouse and you're on your way. In addition, when performing normal day-to-day operations, NT is said to be as easy to use as Windows 95. This is not surprising, as designing installation procedures and configuration setup "wizards" is one of Microsoft's core software strengths. However, when compared to Solaris, Microsoft lost out on remote administration capabilities. Also, any maintenance and repair to a system requires a reboot.

GRADE: B+

SCORE CARD The Standish Group's server operating system selection criteria are a weighted summarization of approximately 250 individual criteria grouped into 12 main areas.

SUMMARY

All the users we spoke to had prior use of Solaris when they brought in an NT system. To the question, "If you were so happy with Solaris, why did you bring in NT?" we always received the same answer. The original reason for purchase of NT was two-fold: first as a file and print server to replace the existing system (typically NetWare); second with the potential of using NT for both a print and file server, and as an application or enterprise server. The thinking was, "If I can have one system for both, my life will be easier." While all the users we spoke to were content with NT as a print and file server, the system just isn't ready for anything more.

For Intranet/Internet services, NT could not touch Solaris. As part of our research for this project The Standish Group spoke with several ISPs ranging in size from a large national ISP provider to a tiny ISP reseller. Within this segment there was overwhelming support for the Solaris system. Many of these companies were currently using NT for internal LAN systems, and had researched the possibility of using NT for Internet services. They quickly dismissed the idea. In some cases providers were forced to use NT due to customer requests and noticed a significant downside for these sites.

The strongest differentiators we found were in the areas of scalability and availability. Many users commented that NT is a "resource hog," confiding they believe this is due to Microsoft's overuse of GUI interfaces and design work. It's interesting that the one thing these users liked most about NT, the GUI interface, was invariably the thing they liked least as well.

In terms of availability, the NT system was considered very faulty. One user said NT crashed about as often as his Windows 95 desktop. Obviously this was an exaggeration, but apparently not by much. As NT improves, undoubtedly these complaints will diminish. Sun must take advantage of the current window of opportunity to push Solaris as the Internet and Intranet server of choice.