| Next Page | Prev Page | Main Index | Home |
When faced with a potential new paradigm the first thing you must
consider is by what criterion will you judge it. It is frequently
the case that using the tried-and-true technological yardsticks
will give you false or highly misleading results. This is due
to how technology yardsticks are created. Lets take a look at
this process.
The typical development of yardstick criteria follows a predictable
pattern. The criteria start as real world numbers, for example,
this job runs in X hours, or this engine has Y horsepower. But
these are hard numbers to generalize; they do not compare well
across multiple systems and companies. They depend on too many
variables to be reliable and repeatable. The solution to this
problem leads to the next step. This is to find "low level"
numbers that seem to be good predictors of the "real"
numbers.
Here is where the yardsticks tend to get paradigm-specific. These
"low level" numbers can be excellent predictors in one
paradigm and terrible predictors in another. Numbers like, this
job moves Xk per second from disk, or this engine has Y cubic
inches. These numbers are NOT direct performance numbers. They
are predictors of performance. In the case of the cubic inch,
this predictor works fairly well until you look at the rotary
engine. It fails completely when used with turbine engines. Both
rotary and turbine engines represent different engine paradigms.
It is very important to understand the difference between these
"real numbers" and the "predictors." Here
lies the major problem in evaluating any potential new paradigm.
Given the invalidation of the prevailing yardsticks, how can anyone
ever evaluate a truly new technology? How can someone hope to
develop a fair, generic, and paradigm-free yardstick? The first
step is to separate the "real numbers" from the "predictors."
Toward that end, I will now present several common computing predictors.
For each I will provide an explanation as to why they are truly
predictors and not always valid.
1: Percent of CPU utilization,
In the current paradigm a low percentage equals higher total throughput. This is a good thing. In other paradigms, high percentages equal better throughput. The "real number" for this is total CPU time consumed, and the lower the better. Be careful because this is referring to total system CPU consumed. It is a very hard number to quantify on a system running more then one task. There is a lot of CPU overhead at the system level. A significant part of this is not reported at the single-task level.
2: I/O latency,
I/O latency, commonly called "disk wait." In the current
paradigm, the lower the number the better. In other paradigms
this number can very from having no effect, to being in the higher-the-better
category. This wait represents the only free time the system has
to work on other tasks. In some environments this is a good and
necessary thing. In a parallel environment, the disk waits can
be "free" as they do not consume CPU time.
3: Communication latency,
This is the wait associated with any communication task. In the
current paradigm, lower is better. However, several of the current
paradigm tricks used to produce lower communication latency will
increase the system overhead. This has the effect of reducing
total system throughput. In short, there are times where increasing
this wait time will increase total throughput. In other paradigms,
increasing this value's maximum can have the effect of reducing
the average wait.
The only true measure of computer performance is total system
throughput. Numbers like: this job, on this hardware, runs in
X minutes. Or results like: this system, on this hardware, can
support Y thousand users before response time goes over 1.5 seconds.
Both of these examples produces an apples-to-oranges situation
when comparing to any other system. This unfortunately makes any
comparison tricky at best.
So how can any meaningful comparison be done at all?
To some extent this needs to be evaluated on the basis of what
has meaning to you. The strategy I would recommend is as follows:
1: Identify your business make or breaks.
This will allow you to "fix" some of the variables.
Each number you can fix reduces the complexity of the comparison.
Be careful that you use only real world business numbers here.
System numbers, "predictors" or otherwise, have no business
here.
2: Clean up the fixed business values.
Once you have a set of fixed numbers, it is time to make them
paradigm-free. Numbers like: from the time we receive the last
update, this processing must complete in X hours, are good. Numbers
like: this batch run must complete in Y hours to allow time for
the index-build, are bad. Real world business numbers only.
3: Equate values that are not fixed.
Once the fixed values are set the next step is to equate as many other performance values as possible. A good way to equate hardware platforms is to use processing cost. Once the fixed-business-numbers are met the best system has the least total cost.
Figuring the cost of a CPU second can be eye-opening. The cost
of a "CPU second" = ("Total system cost, including
supporting sub-systems" / "Useful system life in seconds")
+ ("Total operational cost per year" / "Number
of seconds in a year"). Once you know the cost of a CPU second
you can derive the cost of running your task on each system. If
you do this for both of the systems being compared, you may find
one system is less expensive even when it is slower and occupies
more physical boxes.
4: Remove issues that are not business based.
Internal system architectures have no place in the evaluation. What internal model a system uses IS NOT RELEVANT to how it will perform the needed business functions.
If you think that this is the case you are by definition stuck
in the old paradigm. A new paradigm is by definition a new way
of accomplishing some task. No large benefit can ever be found
by doing the same old thing "a little better."
When you use this approach to comparing systems, you are in a position to take advantage of the really big improvements. Paradigm changes can be evaluated using this simple and sane approach. You can decide based on business facts which system can really meet your requirements best. You will finally have truly the best system for the business need.
| Next Page | Prev Page | Main Index | Home | © 1998-2004 NPSI |