Next Page Prev Page Main Index Home

White Paper & Overview

Next Paradigm Systems has developed a method by which databases can be created using the native operating system as the core of a database management system (DBMS). Such a database architecture when using a massively parallel cluster of I/O processors, has no hard limits to speed, size, maximum users, reliability, or recovery options. No architectural limits to when or how much capacity and/or performance can be added to an I/O cluster. It is now possible to start considering your corporate information based on business need rather than on computing or traditional database constraints.



In traditional DBMS's, a single set of difficult tradeoffs is the inevitable result of any database design. This means that design considerations have been driven by the database's limitations. The Next Paradigm Systems approach puts the focus on the system goals, not on the limits. Each new component builds on, or in parallel with, the rest of the system. Each MPbase is created to grow and change without the normal pain factor. The method used to access the data is neither fixed nor limited to a single choice. The MPbases developed to date have used three or more parallel access methods to a single copy of the same data at the same time. Each individual access method can then be tuned and optimized for the applications it is supporting.

DBMS Design Questionnaires

Select one point in each range
Indicate importance of each
Online . . . . . . . . . . . . Batch
0 . . 1 . . 2 . . 3 . . 4 . . 5

Recovery . . . . . . . Through-put
0 . . 1 . . 2 . . 3 . . 4 . . 5
. . .
All five's OK
Online . . . . (0,5) ______
Batch . . . . .(0,5) ______
Recovery . . (0,5) ______
Through-put (0,5) ______
. . .


With most clustered architectures, linear scaling is a much sought after, yet never achieved, computational Holy Grail. MPbase generally exhibits better than linear scaling. The more I/O processors an MPbase cluster contains, the faster each processor will run. This is only possible with an architecture driven from the bottom up (pull) not the top down. As the number of I/O processors continues to increase, the scaling may eventually fall back to linear, but never less.

Performance of an MPbase I/O cluster is primarily determined by the amount of disk space given to each I/O processor node (disk/processor ratio). This factor is determined independently of data capacity and access capacity, and can be changed independently as well. This means that a performance level can be set that will not degrade as more capacity is added to the MPbase. This ratio determines the performance for the worst case query. All other cases will show improvement with size. With MPbase the bigger it is, the faster it runs!


There is no performance penalty for adding hardware to an MPbase I/O cluster. There is no architectural upper limit to the size of a table in an MPbase. After the initial disk-to-processor ratio is set, the number of disk/processor sets (nodes) needed is then determined by the amount of data to be stored. This means that to add more data, add more nodes. A single query will never be slower after adding more nodes and will most often run faster.

The two-part Achilles heel of most conventional large databases is fragmentation and the reorganization required to correct it (or, for that matter any process requiring a full sequential scan). With MPbase, once the disk/processor ratio is determined, the time needed to perform any full database process will be no more than the time needed at a single node. This time will NOT increase as the database grows. A table of 100 billion rows will reorganize as fast as the number of rows found on a single node. Once again, as MPbase grows, the worst-case-processing times will not increase. All other processing cases just keep getting faster with size.


The architecture of MPbase allows a true many-to-many network between the nodes holding the data and the nodes providing user access. This means MPbase has no single choke point. This lack of a single access/choke point allows the I/O cluster to be scaled to ANY level of parallel access. This also allows parallel access to scale independently of data scaling and performance scaling.

The architecture allows a single database image with no single choke point. This means creating a database with NO LIMITS. A database which will start at any requested level of performance. A database to which you can predictably add any amount of storage, parallel access and/or performance. A database which can start small and gracefully grow to ANY size providing ANY desired performance level with ANY degree of parallel access.


MPbase can be configured to be fault-tolerant at the node level. With a massively parallel platform, fault tolerance is not an option. MPbase can be implemented with no single failure point. The loss of ANY one user or data node in the MPbase I/O cluster need not stop it from functioning nor cause any data to become inaccessible.

The architecture is such that an MPbase need never go down. Backups, restores and even re-organizational unload/reloads, do not require queries or transaction processing to be stopped. Depending on requirements and MPbase's I/O cluster design, even updates can continue during such processing.


All data and data types stored in MPbase are kept internally in a highly compressed common format. The compression method, which is lossless in nature and unique to MPbase, is called "Multidimensional Data-Intelligent Run Length Encoding." Think of this as computer-readable shorthand. This allows even faster access to the compressed data than would be possible for the uncompressed data. Typical compression rates can be 60% to 99% over a flat ASCII (text) file. The resulting physical format is directly usable on any hardware capable of running MPbase. The compression format by itself provides a good level of both security and data integrity. If required, the low-level access routines can include varying additional levels of encryption parallelized across multiple nodes. The function handling the compression also provides access or update monitoring and/or control through a single low-level routine. Access to the database can then be as open or as controlled as required.


In MPbase, the data is transformed inside the low level objects into information. This can best be described as a formless format. From this state, the data can be recreated in any required output format. There is no performance penalty for this activity. As a matter of fact, this reformatting is used to increase performance while at the same time significantly decreasing the physical storage requirements.

This data storage technique allows the direct linking of any tables containing the same "information." Internally, the external format is irrelevant. The key values are meaningless until converted by each table's access code. This puts all of the information stored in MPbase into the same information space. The only time the external data format is used is in the final result set to be sent to the user, without regard to EBCDIC vs. ASCII vs. proprietary formats. On the inbound side, the external format is lost just as quickly. It is converted to an internal transfer format before being passed to the data nodes for storage in the formless common format.


The storage media model changes significantly with the implementation of MPbase. It is possible to view optical or tape media in a much improved light as they relate closer to the performance of magnetic disk and memory.

In comparison with conventional databases, memory is faster, disk is as fast as memory was, and optical is faster than disk was.


The loosely coupled architecture lends itself to a unique set of implementations. For instance, there is no requirement that the nodes making up an I/O cluster be in the same room or even at the same site. This same freedom exists in relation to second or even third copies of the same data when running mirrored. Think of the possible benefits to having a single consistent database image based on hardware placed all over the world. Each branch could maintain its own portion of the corporate data, while at the same time corporate headquarters would have a company-wide view. The possible configurations are as endless as real world business requirements dictate.

For extreme reliability, MPbase can be run with distributed mirrored copies at multiple sites with each site kept in update synchronization in real time. The queries can be distributed to the site currently running with the least load. This distribution can occur at the sub-query level allowing all sites to work on parts of a single query as the load permits. In this configuration the failure of any one node or even one entire site need not affect the ability to continue to run queries.


Security in a networked environment is a major concern. MPbase has two operational models to address this. The open model which is used with non-secure data and/or on a protected internal network. Or, at the other end of the spectrum, the secure model can be configured to meet any possible requirements.

In the open model, the access routines can be stored and backed up with the data requiring no keys and operating from the command line. In this mode, the compression functions as just that: compression. The only side benefit is data integrity. Think of this model in the same light as a self-extracting archive. If it is corrupt, it will tell you. If it is not corrupt, it is usable on its own.

At the high-security end, the compression can include multiple layers of encryption. The access routines can be kept on a single high-security server within the cluster. Access can require external keys which can be passed into the cluster encrypted or found only on a second high-security server within the cluster. The cluster can be configured so that the security servers are not accessible or even visible outside of the cluster. The way the database functions, no user logins need ever be allowed directly on the cluster. All access can be through secure Remote Procedure Calls (RPCs). In this mode, the user access nodes function as firewalls (secure access points). Backups can be done in sections, such that each section is useless without the others. The backup sections can then be stored at different sites. MPbase can search, total and analyze the data without first producing the plain text.


And now, really good news! MPbase loves small inexpensive workstation class machines. Because of the fault-tolerance at the node level, there is no need for special high-reliability fault-tolerant equipment. Also, because there is no penalty for adding nodes, there is no need for the premium-priced, high-end SMP platforms. The main criterion for an efficient hardware selection is dollars per unit of useful work.

When redundancy is required, the combination of the compression and the ability to use less expensive hardware really adds up. A configuration using two copies of the data will typically have an aggregate compression (including the space used by both copies) of 60% to 80% or better. Based on this type of compression multiple mirrored copies become a very cost effective option.


The "magic" that makes all of this possible is a massively parallel, object-oriented architecture that can best be described as inside-out. The database exists, not "inside" a black box, but "outside" in the operating system's environment. The MPbase architecture is much like the operating system (OS) itself. It is made up of lots of little independent pieces. Each of these pieces communicates to the others through the file system interface. The result is a system that can take full advantage of all of the OS features and functions.

What Next Paradigm Systems has developed is a method by which such a database system, spread out across several machines can act as if it were a single black box. At the same time allowing the system to run as fast as if it were an embedded or stand alone system. When used with "Multidimensional Data-Intelligent Run Length Encoding," the resulting performance is nearly unbelievable.


Due to the "inside-out" architecture this database is directly accessible by any program or system that can use the OS file system interface. It has been accessed by using: Structured Query Language (SQL), World Wide Web (HTML), shell scripts, command line and custom programs of any language. Cross platform access is accomplished using HTML, RPCs or sockets. All of these are available on virtually any computing platform.


And now, some details for those into numbers. One MPbase, built for less than $500,000.00 on thirty workstation nodes, contains over 50 billion 84-byte rows. It is designed to grow to over 150 billion. This database runs 24 hours a day, seven days a week, with no downtime. The access rate is over 90 thousand rows, selected, sorted, and returned per second. This rate involves a key hit rate of only 25%. So the database is handling 360 thousand keys per second. This extract rate is maintained in parallel with a 150-row-per-second insert rate, and a 50-row-per-second update rate.

The internal processing rate, available to "next generation" applications, is over 5 million rows per second, or 18 billion rows per hour. The compression achieved in this database is 95% for a single copy. Counting both the primary and mirror copies the compression is still 90%. This database system lacks any single failure point.

Next Page Prev Page Main Index Home © 1998-2004 NPSI