Real World Numbers #1 "Data Transfer"
A 7/24 batch data transfer system

A good example of MPbase's parallel fault-tolerant architecture is a data transfer system set up by MPbase's author. This was done long before MPbase, but is a good example of this type of paradigm.

This system made use of "spare bandwidth" on a T1 to move data from an IBM ES9000 to a cluster of UNIX servers. It replaced the need to run a van with a load of 9 track tapes over 300 miles twice a day. In addition, these tapes could not be reused after leaving the data center, due to operations' policy. The other bonus was that files larger than one tape could now be used.

Statistics:

1.5+ terabyte each year,
7/24 darkroom operations,
Self pacing to 50% of the available bandwidth during day,
Used 98% of the available bandwidth at night,
Highly fault-tolerant multi-path transfer,
Automatically supplied one BBS and three departments with data,
Controlled by mainframe naming-convention.

After being asked several times an hour about link status, a console was set up facing the window in the server room. After this, there was frequently a good size group watching their files move through the system.

An amusing note:

Most of the users of the system were not fully aware of just how robust the link was. During one large and critical transfer I was asked by the user to take an early lunch. He was concerned that other work might somehow interfere with the transfer. My response was to walk over and shut down the primary machine performing the network routing into the server.

After a moment of total panic the transfer resumed by routing through a secondary network connection. I explained to the user that there were three such links and all would need to fail at the same time to interrupt his transfer. For some strange reason I was never again asked to take an early lunch due to an in-progress transfer.

Real World Numbers #1 "Data Transfer" A 7/24 batch data transfer system

Real World Numbers #1 "Data Transfer"
A 7/24 batch data transfer system