Mitopia® to this day still contains a built-in and tightly integrated robotic mass storage subsystem known as MitoStore™ which was expressly designed to meet the needs of a diverse range of multimedia formats in a fully configurable and extensible architectural manner. Our initial attempts starting in the early 90’s to use a commercial mass storage software package met with little success due to the inflexibility of such packages and the inability to customize package behavior based on the needs of each different media type. All such mass storage suites appear to be predicated on the assumption that there is a central authority (vendor supplied) that is in control of robotic autoloader hardware, and this is of course incompatible with our distributed scaleable database requirements. As a consequence of these experiences, Mitopia® was designed such that mass storage is handled explicitly within the standard server model through an abstraction called the “Logical Mass Storage Layer”. The logical MSS layer allows registration of standardized custom device drivers for new media types and robots (it is these custom drivers that form the bulk of the MitoStore™ subsystem), as well as complete control over all the various loading and routing strategies necessary to make optimal use of mass storage based on media type. Within the logical MSS abstraction, storage is broken into four categories viz:
- Disk caches and buffers for media content held in the client machines.
- Disk and RAM caches for media content distributed throughout the drones of the server cluster that relates to the media type involved. This includes RAID disks and sharing techniques such as Storage Area Networks (SANs.) based for example on fiber channel.
- Robotic storage based on removable media such as CD-ROMs, DVDs, tapes or other devices. The model supports redundant media copies if necessary.
- Off-line archival storage of the removable media volumes in a storage facility requiring explicit human activity to retrieve and re-load off-line volumes if required.
Mitopia’s abstraction for a robotic storage device that can participate in the mass storage infrastructure consists of the following:
In this model, near-line storage is organized into a set of possibly heterogeneous robotic autoloaders where each robot/autoloader is comprised of a set of storage slots possibly organized into shelves, a set of drives connected to the drones of the server cluster via some kind of data connection (e.g., SCSI), a controller which can be commanded via a control connection of some sort, and a transport which can be commanded to move media from and between slots and drives and in addition to/from a mailbox slot (which allows media to be added to and removed from the robot). In some robots, the shelves themselves are removable as a block. While the details vary considerably from robot to robot, by registration of suitable autoloader specific plug-ins, all robotic storage systems can be represented in this model. For any given server incorporating robotic storage, the <ConfigString> XML tag supplied during MitoPlex™ server initialization (discussed elsewhere) can be used to specify the symbolic name of the autoloader(s) involved and the connectivity of the data connections. Once the necessary driver for a given robot has been registered, the logical MSS layer is able to fully automatically integrate the use of near-line storage farms into the basic operation of any multimedia MitoPlex™ server.
The Mitopia® server code contains sophisticated strategies for routing requests based on where the required media actually resides and which drives can be used to access the media (and thus which drones are involved). In all operations Mitopia® attempts to cache results in the hard disk storage of the various drones that are used to access the robots, and thus an automatic most-recently-used caching scheme is in effect at all times. Additionally as new multimedia information arrives at the server, the logical MSS layer utilizes knowledge of the media format in the attached autoloaders to create archive ‘chunks’ in drone disk storage which are automatically moved to robotic media when complete, thus ensuring a smooth migration of multimedia data from cache to near-line storage.
Many media types (e.g., DVD) require specialized ‘burning’ activities in order to create them and an entire abstraction for media burning is also in place. Essentially all that is necessary is to fill the robot(s) with blank media and Mitopia® takes care of the rest. This philosophy is extended to handle off-line storage so that Mitopia® supports the concept of system ‘operators’ (for large installations) who can take media off-line (under software control via the mailbox) and who will be automatically notified by the system if an attempt is made to access the off-line media. In this case the operator(s) is prompted for which media to retrieve and which robot to put it in (the system always checks that he did it right!). The entire storage abstraction all the way from caches in the clients through to off-line warehouse storage is completely handled by the Mitopia® architecture in a tightly integrated manner which allows a far higher degree of control over system load balancing and performance than is possible with off-the-shelf mass storage solutions. Mitopia® already contains drivers for many leading autoloader devices and the specification of a driver for new autoloaders is a relatively simple matter.
But it is now the year 2014 and Moore’s law has continued its inevitable march as far as disk drive densities are concerned.
A one terabyte drive is now commonplace and can be had for around $50! Back in the early 90’s our robotic storage consisted of CD-ROMs (and later DVDs). Because disk drives were tiny and outrageously expensive, it made sense to store large data sets in a robotic autoloader. In different systems and configurations we supported a wide variety of autoloaders including: Cygnet Model D480, Pioneer DRM 7000, Plasmon D Series, InfiniDISC, Pioneer 1804 and 5004, TiltRac CD and DVD models, and others. Indeed when the capacities of these models proved too low in some cases we also had two specialized custom robotic autoloaders designed and built to our needs, one by CyberKinetics, and another even larger model taking up its own small room using pick-and-place robots of the kind found in car assembly lines. I fondly remember watching that bad boy pick up a video tape, shove it into a drive that already contained a tape, and because we’d temporarily disabled the force sensor (so it didn’t know there was a problem), continue shoving the tape, and the drive, through the back wall of the room. This of course happened during a client demo! Ah…those were the days! Some of these venerable devices are shown above, others regrettably I can no longer find pictures of.
One can fit 1,500 CD-ROMs into a single $50 1TB drive, 200 DVDs fit in the same space. The economics of these robotic monsters has long since ceased to make any sense, they have been killed off by Moore’s law. Of course those 1TB disk drives can still be picked-and-placed so there is still a few dark corners where the robots can eek out a living; places like the NSA spring to mind. But storage technology hasn’t stopped and even those systems are already relics.
A new breed of robots has emerged that can walk, fly, and go places that humans can’t. The storage dinosaurs are gone and the Mitopia® code that supports them is an echo of a bygone era. Long live the new warmblooded robots, extinction events create new opportunities.