The Inside Story:
How Linux Helps Make Smart Companies Smarter

Tale of Two Stories

There are two perspectives on Linux' success in the enterprise: One from the inside and the other from the outside. Both agree that Linux and open source are Good Things that are changing enterprises for the better. The main difference is in where they place the credit.

Insiders credit the organization's intelligence, resourcefulness and expertise. Outsiders credit vendors with the same qualities. Credit is due to both, of course; and to some degree both sides give some credit to the other. But the the press generally takes the outside view, giving full credit to vendors and little credit to customers. As a result, approximately zero attention is paid to developments that not only account for Linux' success in the enterprise, but also for Linux' creation in the first place.

One problem with inside stories is that no two of them are the same. Every vendor-side story is basically about how products solve problems. That's why vendors love to call their products "solutions". But Linux isn't about "solutions". Linux is about resourcefulness and intelligence. Linux succeeds in the enterprise by helping smart companies make themselves smarter. With Linux and open source, the primary supply and the demand sides of the marketplace are both inside the company. Vendors are optional. And when they're involved, customers set the terms.

This is a very hard development for many of us to understand. It's hard for vendors because they're used to a marketplace where the supply side is in control and wide profit margins are the ideal, even if they are no longer the norm. It's hard for the press to understand, because so many of the news stories they write are vendor-vs.-vendor sportscasst in which the marketplace is a "playing field" and customers are prizes . It's hard for corporate leadership to understand, because they're so used to playing the prize role in vendor playoffs.

The people who don't find it hard to understand are the employees who use Linux and open source to make their companies smarter, more efficient, more reliable and more effective. These the people are not only leading Linux and open source to success in the enterprise, but are helping smart vendors participate in the same success.

Take the case of LSI Logic, the semiconductor maker. Roland Smith is the Director of Global Operations there, running one of four groups that report to the company's CIO. The company's big switch to Linux came a while back, when Smith's team recommended replacing HP-UX with Linux, increasing performance by 5-6x, and cutting costs by two thirds. The team overcame internal resistance, and ultimately won support throughout the whole IT department. Now, Smith says, "Whenever somebody comes in and wants to talk to the CIO about a new application, or a project, or anything new, the CIO's question is, 'Does it run on Linux?'"

This has some interesting effects not only on the flow of demand, but the flow of help. Roland Smith provides an excellent real-world case in which the flow goes from customer to vendor:

In respect to vendors, help actually runs kind of the other way around. Let me give you a current example. In our SAP environment we recently decided we wanted to begin inserting Linux application servers. So we went out and talked to both Dell and HP. We told them what we wanted from each of them was an eval server -- one they think is best suited for running Linux. We wanted four processors and a gig of memory.

A couple of interesting things happened. One was that HP fumbled around and took two weeks to come back and say "What do you really need?" The other was that Dell took three days and shipped us a server. They said "Here. You've got it for three months. Go tell us what you can do with it." That was kind of wonderful.

So then we went to SAP and said "What do you recommend?" And SAP said, "Well, we think SuSE 8 runs okay. And we think that Red Hat 7.1 runs okay. But we don't really know." Then we went and talked to Red Hat and they said "We think it'll run nicely on our Advanced Server 2.1, but we don't really know. So how about you go try it out and tell us?"

So we built it. We bought the Red Hat application server, built it, loaded SAP on it, and put it in our test environment. And it's running incredibly well. We're very happy with it. But it's interesting to me that, of all the vendors out there, only Dell said "We'll happily give you hardware to try it out on, but we don't know anything more than that."

In other words, most of the intelligence and leadership is located on the customer side. This may not be true for every product category in the industry, but it's certainly true for Linux in particular and open source development in general. Bill Joy famously said, ""Let's be truthful, most of the bright people don't work for you -- no matter who you are. You need a strategy that allows for innovation occurring elsewhere." Roland Smith's corollary might be, The most important smart people are working for your customers.

One of those smart customers is Orbitz, the online travel company owned by five major airlines. Leon Chism, Chief Internet Architect for Orbitz, says this about how the company applies its own intelligence internally:

Our developers are very good at leveraging open source tools to improve our design, build, and software management process. On their own they have built a number of tools that parse logs, store them in a PostgreSQL database, and do ex post facto analysis to find errors and opportunities for improvement. The first I had heard of one tool was after it had been used to do analysis and improve some sections of the code successfully several times. We also have a number of developers and engineers using tools like Perl, Python, and Jython (we employ an official Jython developer) for their own ad hoc tools to help them monitor site behavior. All of this is in addition to the official production management infrastructure. Generally, after a tool has been developed, tested, and proved useful, we migrate it into the rest of the management infrastructure our Network Operations Center uses.

We also use a number of Java open source tools, many of which are or were Jakarta projects. We use Ant to do builds, we use Cruise Control for testing the build several times a day, and we use JUnit and Cactus in our daily test processes. All of these tools were brought in by developers and tuned for our specific needs without any direction by management. Generally, when a developer has an itch they look around for others with the same itch and find a way to scratch it quickly. That usually means finding an open source tool.

He also sees Linux and open source as a competitive advantages for Orbitz:

Orbitz is a company that leverages open source projects in its production environment and isn't afraid to be public about it. We currently use over 750 Linux machines in our production environment. Our web servers are Apache running on Linux, our application servers are proprietary servlet engines running on Linux, and the back end "booking engine" is comprised entirely of Java services running on Linux. Lastly, the software that does the low fare searching that is one of the key differentiators between Orbitz and our rivals is also running on Linux.

That's a lot of applied intelligence at work, and it speaks volumes about power relationships with vendors in the real IT marketplace. So does "Fear the Penguin," a document published earlier this year by Goldman-Sachs. It says

...the technical developments and market forces are in place for [Linux] also to become the dominant O/S on the higher-end servers of the enterprise data center, where mission-critical functions are run and the lion's share of IT spending occurs. In particular, we believe that enterprise customers will use Linux primarily to take advantage of lower-cost, higher-performance Intel-based servers and avoid technology lock-in situations. As a result, we believe the Linux-on-Intel model will displace the existing paradigm of premium-priced proprietary systems based on Unix operating systems and RISC processors.

With those three examples we get a pretty good idea about how Linux and open source smarts are applied, both within the enterprise, and between the enterprise and its outside suppliers.

Now what about the outside view? An excellent example is provided by William M. Bulkeley, in "Out of the Shadows: Open-source software is not only becoming acceptable -- It's also becoming a big business" -- a feature that ran in the March 31, 2003 Technology section of The Wall Street Journal:

Free software is going mainstream.

Open-source software, developed outside the corporate framework and available at no cost to anyone who wants to download it, has long been considered by some to be somehow untrustworthy or inadequate. But that perception is giving way to an appreciation of its improving quality and increasing versatility.

And, perhaps most important, companies have found ways to make money by providing services to open-source-software users, or by packaging this free software with products they sell. With the profit motive driving its promotion, free software is cropping up all over corporate and government computers.

When I first saw that piece, I thought Cool! The Wall Street Journal says "Free software is going mainstream". But then I began to ask some questions:

• What does he mean, "outside the corporate framework?" Take away all the GNU/Linux code written by programmers working inside corporate frameworks and there isn't a lot left to the right of the GNU.
• What's this "long been considered" stuff? By whom, exactly?
• Why is it "most important" that "companies have found ways to make money"? Is that more important than the actual work that gets done?
• Why does he say "companies" when he means "vendors"? Aren't customers companies too?

I wondered, What about the customer's need to save money as well as time, effort and capital investment? Don't those account for Linux' success too?

Well, not if you only look at what vendors are doing. And not if you assume nothing significant happens unless a big vendor pushes it.

Ask yourself: Is IBM responsible for Linux' success in the enterprise? Michael Totty thinks so. He wrote the lead article -- "Ten Technologies You Need to Know About Now: From Wi-Fi to Open Source" -- in that same Wall Street Journal section. Says Totty,

It helps to have heavyweight backers. Linux, the open-source computer operating system, might have remained a tool of hobbyists and software rebels had it not been embraced by giant International Business Machines Corp.

In fact, IBM's official enthusiasm for Linux came only after company brass discovered a critical mass of Linux usage and know-how among its own rank and file technologists. This was explained to me by a roomful of freshly-minted IBM Linux advocates at a meeting during the August 1999 LinuxWorld Expo. When I asked why IBM was suddenly all rah-rah about Linux (this was long before IBM launched its famous "$1 billion" Linux campaign), I was told that the company had surveyed for six levels of "Linux awareness" among six hundred workers in one technical division. The bottom level of awareness was "can spell Linux". The top level was "hacks kernel code". All six hundred could spell Linux. One hundred and twenty were hacking the kernel. Thus IBM, being a smart company, leveraged a decision already made by its own engineers. The notorious "Peace, Love & Linux" campaign" was never about mainstreaming hobbyists and rebels. It was about applying its own resourcefulness and intelligence -- and appealing to the same in its customers.

And let's face it: Linux users are tough customers. Here's Leon Chism again:

On the relationship front, there is a vendor that we have and are still considering replacing with an open source solution. For the most part, the application works, but when we find bugs in it, the resolution process leaves a lot to be desired. Escalation procedures, arguments about root cause, long discussions about "supported configurations" all lead to muddling the process rather than focusing on solution. When we have issues or questions with Linux, or Apache, these things aren't at issue. We simply use the customer service app to end all customer service apps -- http://www.google.com/ -- and start researching. Or we start perusing source code. Orbitz is a customer that typically pushes the limits of the products we use. Some vendors are up for the challenge and are willing to make efforts to support that need. Some aren't. And we don't know who is who until, as they say, push comes to shove. Using open source products removes that issue. We have an engineering and software development staff that are fully capable of paddling their own canoe, and the support we get from authors, list servs, and Google is more than enough in most cases.

Yet the tough vendor story gets played while the tough customer story does not. One example is "Larry Ellison's Sober Vision", by Mylene Mangalindan, in the April 8, 2003 edition of The Wall Street Journal. Citing "Industrial Revolution concepts" including specialization of labor" and "economies of scale", Ellison predicts "the end of Silicon Valley as we know it". According to the story, Ellison "paints a dark vision of the computer industry's future: Standardized products with little distinguishing technologym thin profit margins, sweeping consolidation, fewer start-ups... and a thousand dead companies in the midst of which stand a precioius few category-dominating winners, all controlling innovation. All because "...cheaper computers that use the free Linux software and other technical efficiencies will drive down prices."

But is the future really this dark? Let's take an inside view of the same trends.

The Intelligence Imperative

Dell may be the top-selling PC brand, but the majority of PCs sold have no brand at all. The real best-sellers are no-name "white" boxes are either bought by the ton or built out of piece parts on an as-needed basis. Customers love them, because they're cheap to buy, remarkably reliable (especially at the price), and easy to replace.

But we don't hear much about them One reason is because they don't spend much money on advertising and PR. But a bigger reason is that the computer industry has a long-standing prejudice against the word "commodity", and had a huge fear of "commodification".

Of course commoditizatoin has long since become a fact of life in the hardware business; but software is still widely considered a big margin category. With good reason. Microsoft has a bad quarter when gross profit margins slip below 80%. Oracle's worst quarter in the last five (at this writing) was still over 75%.

But the software industry will have to face the fact that customers love commodities. In fact, they love software commodities so much that they make their own and improve them continuously. This do-it-yourself (DIY) process deserves enormous credit for the successes of Linux, of open source development and of all the standards and protocols on which the Net and the Web were built. Had it been up to the large high-margin software vendors of the world, none of that good stuff would have ever been contemplated, much less invented.

Yet they did succeed, and continue to succeed, because smart companies are doing whatever it takes to get smarter. Since smart companies care deeply about saving money, getting smarter will naturally involve efforts to pay even less money than before.

One good example is the University Corporation for Atmospheric Research (UCAR) in Boulder, Colorado. In spite of its public sector name, UCAR is a business with substantial customers that include the national air traffic control system. Greg Thompson, a scientist with UCAR, explains the economics involved on the customer's side:

We're very anti-pay-for-anything around here. Cheap hardware is what got Linux going in the first place for us. That was back around '95. Before then we were a Sun and SGI shop. Now we're all-Dell, except for occasional iBooks, which we use because they run a flavor of UNIX. All the rest of our laptops are Linux -- mostly Debian with a little Red Hat. Some of them are dual-boot with Windows, so we can relate to customers. That's why we run a lot of VMware too.

He explains how the company makes itself smarter this way:

Everybody here has a shelf full of O'Reilly books. We like to figure things out for ourselves. For example, I'm a scientist, not a programmer. But awhile back I saw a need to get past manipulating data with Perl and textfiles. So I started reading up on databases and learning MySQL. I wouldn't have bothered if MySQL hadn't been free. We don't have five digit figures around here to run Oracle. We don't need hard-core database stuff. We don't need transaction support. But we do need results that are Web-accessible. So MySQL made sense. Building internal expertise around something like MySQL is straightforward and easy. I do a little research on my own, see if something I need is already installed -- and if it's not I send a help desk email to a sysadmin-- and then the next day I just download a Debian package, run apt-get and I'm in business. Without spending a dime."

It's amazing how small your software approval bureaucracy can get when you're dealing with freeware rather than costware. When friction is low and costs are zero, free enterprise lives up to its name. Literally.

"Our IT budget is zero," says Elliot Noss, President & CEO of Tucows in Toronto. Tucows' business is hosting one of the largest software download sites in the world, plus running one of the three top-level domain name registries (OpenSRS), managing around 3.5 million domain names. "We've built everything from the ground up with Linux, Apache, MySQL, PostgreSQL and other LAMP stuff that costs us nothing," Noss says. "I can't even imagine what it would cost to work with a big Sun system, or to run a big Oracle database. If it weren't for Linux and LAMP, we wouldn't be in business."

Resistance is Futile

Of course, not every company has yielded to the intelligence of their most resourceful IT workers. Some have absolute resistance. Chuck Swoboda, President and CEO of Cree, Inc., a $200 million semiconductor company, told me last year that he had made an executive decision to run his company entirely on Microsoft software -- just for the sake of simplfying matters as much as possible. Several months ago, Jerry Hale, CIO of Eastman Chemical, a $5.3 billion company, told me they were still almost all-Microsoft, but would "be open" to alternatives. Meanwhile, I'm hearing a somewhat different message from rank & file technologists inside large enterprises. Resistance is futile, they say. Take, for example, this email:

The Construction Model

To see what a grown-up software industry looks like, just look to where it borrows its vocabulary: The construction industry. It's no coincidence that the software world has "architects", "designers", "builders", "tools" and "platforms". Nor that the Web has "sites" called "homes" with "locations" and "addresses". Code is construction material. Programmers are builders. This suggests that the software industry has a model in the construction industry, and is very likely to mature into something that resembles that industry.

This would be a good thing. Construction the oldest and largest industry in the world. Sales estimates run into trillions of dollars, worldwide. The construction industry is filled with commodities, and with companies making a good living selling, installing and servicing commodities. It has many huge vendors, including some as large as Microsoft. Yet it has no Microsoft. No one company dominates or defines the industry, or conversation about the industry's concerns.

Yes, profits are lower than Microsoft's and Oracle's, but they're nothing shameful. Fluor Corporation, for example, has a gross profit margin of 5.5% on revenues of about $10 billion. Georgia-Pacific has a 22.2% on $23.3 billion. Jacobs Engineering has 13.1% on $4.9 billion. Nobody in construction is saying "Gee, they're doing so much better over there in the software business." Instead they know 70+% margins are impossible to sustain. When that realization becomes obvious to everybody (including Microsoft and Oracle) the construction business will provide a handy working model.

Open source, for example, is standard in construction. If somebody thinks of a better way to hang a door, pour concrete or put up sheetrock, that know-how is passed along to everybody else. Sure, there are still plenty of patents, and a thriving business in pre-fabricated materials that are full of secret stuff. But the nuts & bolts of the trade are exactly that: nuts & bolts. It's the public stuff, not the secret stuff, that keep the business going.

A mature regard for openness-where-it-counts has been coming for a long time to the software business. Maybe that's because it was there in the first place. Avery Lyford, President and CEO of Linuxcare, puts it this way:

I would say that Oracle is probably the best vendor in the marketplace, in respect to Linux. When we told our Oracle account team that we wanted to put up an Oracle database on Linux, they were all over it. They really knew what they were doing. Within a week they had us in touch with their development folks back in Redwood Shores. They had good documentation. It was easy to put up. They were patient. It was easy to run, easy to connect to. We're very happy about it.

Smart companies naturally want smart relationships. Oracle seems to be doing a good job of meeting that market demand. It should help them continue to adapt to the successes of MySQL and PostgreSQL.

Doc Searls is Senior Editor of Linux Journal.