Updated: Let's ask some Smarter Questions about DB2 LUW Performance Tools

Borrowing words from Texas Governor Perry, it is time for some "provocative language." I am an IBM DB2 GOLD Consultant and Information Management Champion. I've earned these merit badges over the years for my advocacy of DB2 and the DB2 community. But, I am not a puppet for IBM and I am concerned for the DB2 community regarding recent trends and events. The rant that follows is my opinion based on information, belief, and observations...
Too many times these last few months I've heard "We're switching to (another database)". As a DB2 LUW advocate since 1992, this makes me sick.

UPDATE 6 OCT 2011: IMHO, one customer switching is TOO MANY.


Why would an IBM DB2 LUW customer abandon its investment in DB2 and head for the competitors? Lower price? I doubt that. Better features or functionality? Unlikely. I suspect it boils down to performance disappointment and glossy brochures.

UPDATE 6 OCT 2011: During The DB2Night Show™ Episode #59 on Index Design, Best Practices, Myths, Realities, and Magic with special guest Martin Hubel, we asked our studio audience polling and survey questions to ascertain their opinions on why organizations might switch from one database to another. In order of popularity (most likely reason to least), our audience indicated these reasons:

  1. Cost Savings - seeking lower overall IT costs
  2. Performance - seeking improved performance
  3. Application Mandates or Preferences - the application drives the database
  4. Skills - the availability of talent in the market
  5. Security and Compliance
If you missed this show, Martin shared many pearls of wisdom, tips, and techniques. WATCH THE REPLAY

The original premise of this blog presumed that organizations switched databases largely because of performance. As it turns out, performance was only the second most popular response after COST. I now have heightened awareness, although I assert there remains a very tight correlation between Performance and COST. A well tuned database will operate with less CPU and I/O consumption whilst delivering higher performance satisfaction to its user community. If you throw money at hardware to achieve performance, your costs will be much higher than they need to be. It doesn't matter what database your organization operates for its business - databases must be tuned to deliver maximum value (highest performance at the lowest possible costs).


Here's the rub - DB2's Achilles' heel - if you will: DB2 license bundling. IBM includes IBM Optim Performance Manager (affectionately known as OPM) with Advanced Enterprise Server Edition (AESE), Workload Manager Performance Optimization Feature (POF), the Database Partitioning Feature (DPF), Smart Analytics Systems, and more. It seems OPM is pretty hard to avoid unless you are a small enough organization to run DB2 Express. Inasmuch as IBM OPM is freely bundled into so many DB2 licensing packages, it is, for all intents and purposes, virtually a free part of DB2.

It is bad manners in the software industry to not say nice things about your worthy competitors. But IBM OPM isn't a competitor - it is part of DB2 - so we can, and should, have a conversation about it.

You can get a free toy inside a box of Cracker Jacks. Also check the box for cautionary wording to the effect of "choking hazard".

So, inside your box of DB2, you get a "free" performance tool. Terrific. Another free tool to augment "db2pd", "db2top", "db2mon", SQL Snapshots, and all the other "free" stuff out there. There are two important axioms to remember:

  1. That which is free has no value
  2. You get what you pay for
IBM OPM has been on the market now for about a year and a half. I haven't yet met an end user of it that actually likes it - and I talk to a lot of people. But maybe, just maybe, there are a handful of happy customers out there. Are you one? Email me or call my mobile phone - I'd like to meet you.

UPDATE 6 OCT 2011: Since the original posting of this blog a few weeks ago, no one has contacted me to express their happiness with OPM. To the contrary, discussions in various LinkedIn groups have been poignant. Visit the DB2 UDB DBA LinkedIn Group and see what people are saying.

To the contrary, what we hear is that IBM OPM takes weeks or months to install and implement - that is if you can get it working with or without help from the Germany lab. I'm also told by several sources that the overhead on the monitored database server is extremely high - in the range of 20-30%. Further, IBM OPM users are encouraged to purchase a separate machine for the IBM OPM server - but, no worries, you can get a free limited use license of DB2 ESE for THAT machine.

Well, there's the first three strikes against IBM OPM:

  • Takes a long time to install and implement
  • High overhead on the monitored DB2 server (hey, just turn on some more CPUs to cover it? )
  • Need to buy a big beefy machine to support the OPM server
But, let's just imagine or assume that you have all the time in the world and unlimited budget. This makes the first three strikes a nit. Here's the real, fundamental, root cause problem with IBM OPM:
  • Incomplete and possibly inaccurate information
It doesn't tell you everything you need to know to fully optimize your DB2 workloads. It might miss important information. It can distort performance data leading you to think that what is a root cause problem indeed is not really a core root cause problem.

I'm not making this crap up. It's in the IBM OPM documentation. I've verified the analysis flaws with my own testing. And I have a pretty good understanding of why OPM's overhead is so high.

I'll show you. Join us for The DB2Night Show™ episode #58 on 23 September 2011 at 10am CDT/11am EDT: Details and Registration Did you miss it? WATCH THE REPLAY

In fairness, as free tools go, even I can find a few things that I like about IBM OPM - if I could get past the first three strikes.

Why does IBM package OPM into the great majority of DB2 license packages?

Your IBM professional will tell you that the bundling is to provide DB2 customers with better value. Really? Is that free toy inside the box really free?

  1. You're going to spend weeks or months trying to get it to work, probably with Germany's help. Estimated Cost: 25-50% of a FTE.
  2. You'll need to add CPUs to the monitored database server to offset the high overhead. Estimated Cost: $50,000 - $80,000 at retail including additional DB2 licenses
  3. You'll need to procure a big beefy multi-CPU machine to support the OPM Repository Server. Estimated Cost: $30,000 - $50,000 or more
These are your direct costs. In addition to these costs, you have forfeited opportunity costs such as:
  • Forfeited savings that you'd otherwise achieve by reducing CPUs and license costs
  • Forfeited savings from server consolidations and reduced energy consumption
  • Forfeited savings from lost DBA productivity - spending hours trying to solve (perhaps the wrong) problems instead of minutes
  • Forfeited savings from lost organization productivity - hey, if transactions run twice as fast you might not need so many customer service representatives and data entry people
  • Forfeited revenue from customers that abandon their shopping carts

While these direct and indirect opportunity costs are painful, the ultimate cost is the cost of failure.

I know of one large company in the US that was working on rehosting a CICS/COBOL/DB2 z/OS application to the DB2 LUW distributed platform. They spent, they said, about six months working with Germany to get OPM working. Finally, one day last month, they flipped the big switch and cut over to the new distributed DB2 LUW system. And they fell flat on their faces with a system that was too slow and frequently crashed. After being in Sev 1 Crit Sits for 3 days while their business was out of business, the company called DBI and I had a chance to get involved and help. We had their system stabilized and performing better in just a couple of hours.

Why didn't IBM OPM help this customer avoid this catastrophic failure? INCOMPLETE and INACCURATE INFORMATION.

As long as we have this wound opened up, let's rub some more salt into it. Because this new system was too slow and unresponsive, the first thing this organization did was what many companies do - they threw hardware at the problem - they doubled the number of CPU cores from 12 to 24 - incurring, of course, not only hardware costs but also DB2 license costs. The hardware upgrade didn't work.

After I worked with this customer for a few days, response times were markedly improved and CPU utilization dropped to 30-40%. The customer asked IBM for a refund on the hardware upgrade. Once again, we were all reminded that CPUs are like underwear - not returnable. IBM declined to refund the upgrade costs. IBM also declined to refund the customer money it paid (separately) for IBM OPM.

Lesson learned: Tune (with a tool that actually provides results) BEFORE upgrading hardware.


So, why are DB2 LUW customers switching to other databases?

IMHO, because IBM provides a free toy, sorry - tool, called OPM inside the DB2 box. Executives are excited because they just "got a value deal" from IBM. Database Administrators are encouraged to use the free tool. The free tool adds overhead to the DB2 server and fails to reveal true root cause performance problems and tuning opportunities. Performance mediocrity prevails, at best. Applications and Data Warehouses grow, and performance deteriorates. Performance dissatisfaction sets in. People complain. Competitive database sales people show up with glossy brochures promising the moon. A multi-million dollar deal gets inked with (another database vendor) and the migration journey begins. Why? Oh yeah, IBM provided those free tools in the box with DB2 that virtually assured DB2's performance failure.

UPDATE 6 OCT 2011: In fairness, per the earlier update, organizations might switch FROM other databases TO DB2 for a number of reasons with "COST" being the most likely driver. IBM states that over 1,000 customers switched from (another database) to DB2 last year. I haven't been introduced to all 1,000 yet, but DBI has had the opportunity to work with some of the customers that switched. And, yes, we helped tune those migrated databases up! READ THIS PRESS RELEASE.


  • Why does a database vendor provide "free" performance tools in license deals?
  • What does the database vendor gain?
  • Is it possible the database vendor wants to keep other ISVs out that might actually help customers lower CPU consumption and database license costs?
  • Is there a conflict of interest here?
  • Is the fox watching the hen house?
  • Do "free" tools actually, in reality, help the database vendor assure higher hardware and software revenues?
  • What doesn't your database vendor want you to know?

Do your Due Diligence

That's what I'm saying. It used to be that people would say "no one ever got fired from buying from IBM." IBM OPM is so riddled with direct costs, indirect costs, and business risks that I worry this notion may longer be true. Never mind the anti-competitive nature of IBM's licensing practices, I think the bigger issue borders on corporate negligence for ignoring the conflicts of interest and failing to do reasonable due diligence.

Think outside the box. The IBM DB2 box, that is.

Suggested Next Steps

  1. Compare DB2 LUW Tools
  2. See that "Mail this" link at the bottom? Click it and send this blog to your CIO, IT Director, and DBA peers. Go ahead. It won't hurt. Sharing is caring.
  3. Contact DBI. Yell, rant, rave, or simply inquire about a DBI Proof of Concept. Put the word DISASTER in the comment text box along with your poetic thoughts and DBI will donate $50 to the American Red Cross to assist with FIRE, FLOOD, TORNADO, and HURRICANE relief efforts (limited to next 20 inquiries during September 2011). Our business is booming - this is just one way we can give back to the communities we serve. UPDATE: We'll extend this offer to cover October 2011 as well.

I wish you many happy, successful days with DB2.

Kind regards,