In previous posts we have examined how the Carmot ontology for the system is leveraged by Mitopia® to auto-generate and handle persistent storage and query, thereby eliminating the ‘database’ portion of the Bermuda Triangle molecule discussed earlier. We have also discussed how the ontology is used to automatically generate and handle virtually all user interfaces (an adaptive UI) in a Mitopia®-based system, thereby eliminating the last remaining ‘GUI’ portion of the the triangle and creating an architecture that is fully adaptive to change (as required to solve any large problem in an unconstrained environment).
In this post I want to briefly examine Mitopia’s fundamental low level approach to the adaptive UI abstraction/isolation layer, particularly as it relates to adaptability issues. We should perhaps begin with a brief overview of the world of GUI frameworks and the current state of the art in this arena.
The list above shows some of the common high level GUI toolkits, for more details see the Wikipedia article on ‘Widget tookits‘. As can be seen, there are a vast number of commercial and free GUI development frameworks that are available. The list of free GUI toolkits contains around 120 distinct entries, while the commercial list has nearly 40. These figures are for C/C++ toolkits only. If other languages (particularly Java), are included, the lists are vastly larger. It is perhaps safe to say that more GUI toolkits are created than in any other functional area of software architectures, since it is one of the most enjoyable areas of software to dabble in, and also one of the few areas where OOP languages can be shown to best effect.
Historically, all operating systems tend to provide a low level (non-OOP) GUI toolkit which allows creation of arbitrary GUIs regardless of how the application code is organized or designed. This is of course essential since most applications have no formalism for the underlying data model and simply graft a custom UI onto the data in an ad hoc manner as required by the particular problem. For small to medium problem, this approach takes considerably less work than other approaches but of course it is highly susceptible to breaking when any aspect of system operation or data changes. This fact notwithstanding, the overwhelming majority of all GUI approaches are of this ad-hoc form, including many of the earlier GUI frameworks themselves.
Most modern GUI toolkits use the Model View Controller (MVC) design technique. In MVC, the Model represents the information (the data) of the application and the business rules used to manipulate the data; the View corresponds to elements of the user interface such as text, checkbox items, and so forth; and the Controller manages details involving the communication to the Model of user actions such as keystrokes and mouse movements.
Of course many applications use a persistent storage mechanism (such as a database) to store data. MVC does not specifically mention the data access layer because it is understood to be underneath or encapsulated by the Model, that is there is custom code in the Model that adapts the needs of the data repository to those of the application and the GUI framework API. In other words, the MVC approach makes no attempt to solve the Bermuda Triangle problem, it simply encourages isolating it into the ‘Model’. In reality, the MVC approach is another 3-atom molecule, identical to the molecule described for the Bermuda Triangle discussion, but labelled differently. Attempting to build a truly complex unconstrained application using the MVC technique will result in the same problems as all other existing techniques.
If we attempt to narrow our internet search to GUI toolkits that are in some way truly data-driven, that is the GUI framework discovers the ‘type’ of actual binary program data to be displayed at run-time and generates a suitable user interface on the fly (i.e., an adaptive UI), we find nothing. The closest matches to this kind of capability are the GUI generators available with high end databases for converting database schemas to UI, and the current efforts to generate GUI based on XML schema to allow XML data editing. Both of these technologies rely on a formalized schema to generate GUI, but this schema is unrelated to the actual types and structures in use by the application code, and is instead tied to the ‘database’ used to implement persistent storage. In other words, these approaches are ‘disjoint’ in the same manner that semantic ontologies are disjoint in that they do not address unifying the programming model for application code with that of the GUI or database. Without addressing this issue, we cannot truly solve the triangle problem.
The problem is that GUI frameworks tend to be created out of an interest in user interfaces and how to make them more functional or elaborate, not from an interest in making applications more adaptive and getting rid of UI related code entirely.
Mitopia’s takes a plugin approach to the GUI layer where specific implementations and delivery platforms (known as ‘renderers’) are registered with a ‘universal’ abstraction layer which is itself hidden behind an interpreted GUI language, based on Carmot-E so that direct access to system data is automatic. It performs this trick through a plug-in adapter to Mitopia’s data model and type management system. Virtually any conventional GUI framework could be registered behind these abstraction layers in order to target different GUI devices or environments. Indeed at this time within a standard Mitopia® local GUI environment, there are currently two very different renderers running, and communicating through, the same abstracted windowing environment. Which renderer is used by any given Mitopia® widget is a matter of widget and/or user choice.
The first such plugin is called Genesis. Genesis is a large and elaborate C++ cross-platform GUI framework and encapsulates all of the View and Controller logic within itself while providing a data-model interface for run-time data discovery (GUI-to-application). In other words from a GUI perspective, Genesis supports the discovery of the ‘Model’ in the DMV paradigm rather than just the registration of custom Model code. This is a more sophisticated approach to data connection than that used in conventional GUI toolkits. The interface between the ‘renderer’ and data residing in Mitopia® is provided by Mitopia’s low level DataModels.c package which supports the registration of all the necessary plug-in functions required to form a complete GUI/application connection. Genesis is the oldest of Mitopia’s current GUI plugins, predating Mitopia’s ‘Universal Renderer’ abstraction, and so it connects the data model directly to the GUI interpreter layer. All other renderers plug into the ‘Universal Renderer’ (UR) layer.
The ‘Universal’ renderer layer describes and manipulates the GUI through extensions to the ontology found in “K.DEF”. The universal renderer plugs into Mitopia® at the same level as does Genesis. The UR provides all the the data-discovery logic and most of the abstract GUI behaviors. The universal renderer requires two target-specific plugins, a ‘Canvas’ plugin (think Window handler) and a ‘Painter’ plugin (think Control handler), in order to direct rendered output to/from any given UI environment. This approach dramatically lowers the hurdles necessary to repurpose the GUI to any other target platform (e.g., a web page), since the canvas and painter plugins can be relatively ‘dumb’ and are easy to implement. Mitopia® itself provides platform specific canvas and painter plugins, so that most Mitopia® GUI is now handled through the UR. Currently a few visualizers remain to be implemented under the UR. The eventual goal is 100% UR layered GUI renderers since any GUI handled by the UR can be more easily repurposed to other GUI environments including target screens that are remote and elsewhere on the network.
The application-to-GUI interface between Mitopia® and it’s renderers (e.g., Universal and Genesis) is implemented though a single Mitopia® API function call UI_Command(). This API function is implemented as a parser, for which the interpreted language parsed is in fact Carmot-E, that is the run-time version of Carmot. The actual functional UI operations in GUI command scripts are implemented as a series of custom GUI-related Carmot-E procedures and $functions, and the link to the underlying data which is held in a Mitopia® collection and described by Carmot, is established by designating the collection node to be used at any given time. The node of course allows the interpreter to use Mitopia’s Carmot APIs to discover types, fields and all other information it needs to interpret the commands and data references within the GUI script. This use of the Carmot language to bridge the gap between Mitopia® code/data and the GUI framework, not only accomplishes the required degree of separation between GUI and application code, but also extends this separation all the way to the ‘discovered’ ontology, thereby further immunizing the compiled code from system-specific behaviors.
From the Mitopia® perspective, the ‘local’ renderers are simply the principal members of a number of potential ‘renderers’ which are dynamically registered with the architecture during application startup. Other renderers (via Universal Renderer plugins) might generate web pages, or might use completely different GUI frameworks without in any way impacting core architectural Mitopia® code. A Mitopia® widget generally does not know or care if its GUI is local or not – in any given Mitopia® process there may be a mixture of threads, some using local and some remote GUIs.
The Mitopia® ‘RendererPlugin’ package implements all the required renderer registries and resolution logic. In GUI discussions here and elsewhere, we give examples, and use screen shots, generated by the local renderers, however, it is important to bear in mind that alternate renderers can be registered simultaneously and might be used for different delivery platforms (e.g., a web browser interface) without impacting the functional thrust of the discussions.
Mitopia® requires and enforces a small number of functional assumptions and behaviors on any GUI environment provided by a renderer, and any framework that violates these requirements cannot be used in conjunction with Mitopia®. The requirements are made for the purpose of allowing Mitopia® to define an abstract UI model and registration interface that is capable of implementation by multiple renderers and does not allow renderer-specific assumptions to creep across the interface or to violate the assumptions of Mitopia’s cooperative threading model.
One of the most critical aspects of Mitopia’s programming model is the threaded nature of the code components known as widgets that are combined in order to create a complete Mitopia® application that can run effectively based on the laws of data-flow. This means that every widget must be functionally isolated from all others in its own thread. Each widget is allocated scheduling time by the Mitopia® scheduler according to data-flow and other scheduler-specific considerations. What this means for the GUI environment is that each widget’s GUI environment must be 100% isolated from that of any other widget. To fail to do this raises the potential for deadly embraces and lock/race conditions between widgets mediated through thread conflicts in the GUI layer.
The overwhelming bulk of all GUI frameworks tightly integrate the ‘Controller’ and ‘View’ aspects of the framework by providing their own internal event delivery mechanism. Unless the necessary plug-ins and callbacks are provided by the GUI framework to allow event delivery and all other GUI behaviors to be completely isolated so that Mitopia® retains control of which thread runs at any given time, the GUI framework is hard to adapt for use with Mitopia®.
At the fundamental level, this means that the GUI framework must perform all of these operations using an ‘application GUI context’ which can be switched by Mitopia™ as desired. This also effectively means no non-constant static global variables are allowed in the GUI framework, that the framework cannot assume a single thread or any framework control over thread switches, cannot assume a direct connection to underlying OS event delivery mechanisms, and cannot assume it has complete control of the entire screen or window layout. Most GUI frameworks do not support this level of customization and thus are unsuitable for Mitopia’s highly threaded and data-flow driven environment. These requirements placed on GUI frameworks by Mitopia® are in reality no more than requiring good framework design and coding principles, however, few actual frameworks implement this level of design or code rigor in practice.
Some frameworks attempt to address these issues by moving all GUI operations off to a separate GUI thread and passing messages to/from that thread (Java is an example). Unfortunately, such a strategy is rife with potential problems when combined with a data-flow based scheduling algorithm, since the GUI thread becomes an uncontrolled means of ‘entangling’ the operation of other widget threads and creating undetectable lock and race conditions between them. These ‘separate’ GUI thread frameworks can thus be eliminated from consideration, in fact GUI environments that assume a single application thread but follow good code design principles (like avoiding static globals), are far more easily adapted (by context switching) than those that attempt to address threading themselves. To illustrate how easily separate GUI threads can bring a highly threaded system to its knees, consider the following scenario if implemented using a separate GUI thread.
A user is in the process of using the File-Browser as initiated from UI from client widget ‘A’. For simplicity, most GUI frameworks and OS toolbox libraries would handle this as a ‘modal’ situation, that is they would hang in a ‘modal’ loop waiting for the user to make his pick and dismiss the dialog. Now suppose that running in another thread of the same application is a server which makes a harmless call to the GUI layer to ‘read’ the value in a UI control associated with the server’s UI. This ‘read’ must be sent to the GUI thread for implementation, however, the GUI thread is currently looping waiting for the user to pick a file (or any other modal situation) and is unable to service the request immediately. What this means is that the server is now locked up waiting for the user to pick his/her file. Perhaps the user has gone to get some coffee! This of course means the server is no longer able to process incoming server requests from other clients. This in turn will lead to communications timeouts and other errors with the clients. In other words, the GUI thread has managed to entangle multiple processes across the network and cause the entire application to malfunction by any modal behaviors. If fact, there are a number of other means of entangling external threads through a separate GUI thread that do not even involve modality, this is particularly true when ‘remote’ GUIs are involved.
Mitopia® provides its own internal event delivery system which extends the common UI style events (e.g., mouse down, key press etc.) to add a wide range of other specialized event types that are used by Mitopia® to perform various communications and housekeeping activities and are completely unrelated to but integrated with the UI events. For this reason, Mitopia® allows custom event plug-ins to be registered through the ‘platform’ registry API provided by the ‘RendererPlugin’ package. These platform plug-ins are responsible for capturing events from the OS event mechanism and inserting them into the internal Mitopia® event queues targeted for the appropriate widget(s), and retrieving them in the widget context. Mitopia® pre-registers a built-in platform plugin to capture platform events. It is desirable that all events, including GUI events, go through the Mitopia® event queues in this manner since the Mitopia® scheduler examines the internal queues for pending events in order to adjust its scheduling priorities to ensure that widgets with pending events get scheduling slots rapidly. It is therefore required that any registered GUI renderer is capable of operating normally when events have been routed in this manner rather than pulling events directly from the OS event queues.
Mitopia® logically breaks the executable widget code into a set of ‘views’. Each view can have a single ‘view window’ associated with it. Each view can contain one or more widgets within it, and each widget may have an associated ‘pane’ which is a nominally rectangular region of the view window. In the local GUI environment, panes within view windows in the platform context are implemented by Mitopia® using borderless windows that are locked in position relative to the view window and scale with it. This gives the appearance of a single window on the screen whereas in reality each widget has its own exclusive borderless pane window in which it displays its UI. From the perspective of the GUI framework, each pane window and widget represents a completely distinct application with no connections whatsoever to any other GUI in the Mitopia® application. Mitopia® is responsible for manipulating and ordering the view windows and panes, the GUI framework is responsible only for displaying and handling the content within a pane. Widgets can create daughter windows and dialogs which appear floating above the view window and widget/GUI code is responsible for handling these windows even though the events for these windows are routed through the Mitopia® event queues just like all other events. Only the daughter windows for the frontmost or ‘focus’ widget are showing at any given time, they are hidden by Mitopia® as soon as the widget loses focus.
Within any given widget pane or window, Mitopia® code assumes that the GUI renderer can provide a GUI environment that meets the following minimum criteria:
By contrast, in larger programs it has been shown that over 90% of the cost and effort is involved in the maintenance phase, not in development as most people think. Big complex programs, rather than being built from the ground up (an intimidating and usually impractical strategy) tend to be assembled by ‘integrating’ a variety of building-block technologies or (commercial off the shelf software – COTS). This means that the developer is free to focus on the unique ‘glue’ that is the specific application program being developed, while ignoring the complexity of building complex underpinnings like a relational database, GUI frameworks, functional libraries, and the many other familiar subsystems that make up any large program. This COTS approach is favored by all the large systems integrators. Unfortunately even though the US government (and others) have favored this approach over proprietary development for decades, people now realize that it comes with a serious downside:
|Figure 2 -Evolution of the Mitopia® code base 1991-2013 (click to enlarge)|
The graph shows the total size of the code base over the 22 year period, broken down by the various subsystems that make up the entire Mitopia® environment. Various critical ‘change’ events are annotated on the diagram. Overall, a key thing to note is that excluding the early code mountain ranges, long since eroded, the rate of new code creation has remained roughly constant over the entire life-cycle (Lehman’s 4th law). This is despite the fact that during initial development there were approximately 20 times as many full-time developers working on the code as there are today. The fraction of the code base implemented as fundamental application-independent abstractions within core Mitopia® code (the purple area of the graph) has risen from an initial value of around 5%, to just under 50% when MitoSystems itself took over in 1998; it is around 85% today. In that same time frame, application-specific code (i.e., not abstracted and generalized) has dropped from 95% at the outset to just 0.3% percent today. This is a key indicator of a mature and adaptive ‘architecture’ rather than a domain-specific application. This universal use of, and familiarity with, a few very powerful abstractions to do virtually everything has been responsible for the constant increase in programmer productivity (up to 30K LOC/yr. today from just 7.5K LOC/yr. prior to ATP) – Lehman’s 5th law. This change has allowed software team size and costs to drop by such a large factor (20x) while total productivity remains unchanged or even increased.
“The entropy of a system increases with time unless specific work is executed to maintain or reduce it.”
The gradual consumption of other subsystem code by the core abstractions within Mitopia® depicted in Figure 2 shows that MitoSystems approach to software evolution has always been driven largely by the desire to reduce software entropy. It is this fact more than any other that explains program longevity. Mitopia@ is what Lehman defines as an E-program, and as such is governed by all 8 of Lehman’s laws. Many of the other posts on this site address the core Mitopia® abstractions and how their expressive power allows broad applicability and removal of domain-specific code edifices.
At this point it must be said that the design documents produced by the end of the two year design phase that started in 1991, already specified many of the most basic abstractions in great detail, so one would expect that there would be little or no wasted or duplicative code developed in the other subsystems. It is the ultimate elimination of this waste/duplication that partially accounts for the jagged peaks that characterize the first half of the graph (i.e., the first decade of software life – by which time most programs have already died). The truth of the matter is that initial development of low level Mitopia® abstractions themselves took nearly four years to complete, and in the mean time other subsystem developers were forced to create alternate piecemeal solutions themselves, sometimes by incorporating COTS, in order to move their code bases forward. The pressure to deliver concrete functional capabilities to the customer, despite prior warnings of how long such a system development might take, had much to do with this early mountain formation. I myself got pulled into implementing the core abstractions despite my planned intent to stay out of the code entirely. This in turn caused a lack of competent technical supervision elsewhere during this period. As we all know programmers are infinitely harder to herd than cats, so the ultimate result was a COTS cobbling approach early on. It was never the plan, but it happened anyway. Mea somewhat culpa.
As the chart shows, by around the year 2000, the code base had swollen so much it was larger even than it is today. Worse than that, because of the large diversity of approaches in the code, and particularly the large number of COTS and external libraries involved in the non-core subsystems, programmer time was almost entirely consumed by maintenance activity for the reasons already discussed. As a result, little or no actual forward progress in terms of non-core functionality was occurring. Meanwhile the core Mitopia® code which was (and remains) wholly independent of any other technology (except the underlying OS) continued to expand in capabilities. This dichotomy prompted a detailed examination of the underlying causes, and resulted in identifying ‘COTS cobbling’ as the fundamental issue and enemy of productivity over the long term.
In 2000 we decided we had to tackle this issue, since we could no longer tolerate the entropy and maintenance workload caused by so many diverse technologies. First to go was the custom database library (InsideOut) and wrappers used internally by the audio/video subsystem (pink area labelled MitoMovie™) to track external equipment and allocation for video capture and streaming. This was integrated into the Oracle database used elsewhere at the time. The continued development and maintenance of our own video CODEC, ATM network driver, and streaming technology (known as DVS) also no longer made sense, so we re-engineered to use QuickTime. Moreover, for a couple of years (starting in ’99) we had been forced to run in 68K emulation on Apple’s new PPC machines (there were heterogeneous installations at the time) because we used a purchased ‘stemming’ library that was implemented using the 68K processor; a PPC replacement was not on the cards. We were thus forced to develop our own replacement stemming technology. The first precipitous drop in code size caused by these changes can be seen in the chart during the year 2000.
In 2002, our ability to continue running under Mac OS-9 came to an end, as we knew it ultimately would. Apple had moved entirely to the OS-X architecture, and while the ‘Carbon’ APIs from OS-9 were still supported, three of the largest and most heavily used external COTS chunks were dead in the water with the change in OS. The Oracle database, while supported (somewhat) under OS-9, was at the time unsupported on OS-X with no plans to change this in the foreseeable future. The same was true for the Personal Librarian Software (PLS) we had incorporated into our database federation to handle inverted file text searching across languages. Equally disastrous was the fact that our internal MitoWorks™ GUI framework no longer made sense in an OS-X environment, and a replacement would be needed at the same time as all the other changes. 2002 constituted the single most disastrous illustration of the danger of COTS to life expectancy. If these components had only been our own code, all we’d have had to do would be re-compile them. It was a harsh but useful lesson.
For most projects, such a combination would have been instantly fatal, however, due to the unique nature of our primary customer, we were able to embark on a 1.5 year re-implementation of all three obsolete components which was responsible for the single largest drop in non-core code (see chart from 2002-2003). In this one transition we shed around a third of the entire code base while coming out of it at the end of the transition with enhanced capabilities, vastly improved performance, and a code structure now almost entirely implemented as core abstractions. We were now dependent on just 3 external COTS components (down from closer to 10 at the outset). The remaining dependencies being the underlying OS (now OS-X), the replacement GUI framework (known as Genesis), and a GIS library (which itself had replaced an earlier choice when the original vendor went out of business). Best of all, after the transition, programmer productivity increased dramatically and the time spent treading water in maintenance dropped proportionately, so allowing a subsequent up-tick in the rate of introducing new functionality into the architecture.
Throwing out the relational database and inverted text engine turned out to be one of the best things we ever did since it lead to the MitoPlex™ and MitoQuest™ core technologies, both built directly on the core Mitopia® extractions. The resulting code simplifications were only outweighed by the massive increases in performance and scaling we were able to realize at the same time. The same was true of switching to the new GUI platform which ultimately delivered improved visualizers, and enabled rapid development of the data-driven GUI approach to previously unthinkable levels. The combination made the architecture ‘adaptable’ to new domains in time frames that were fractions of those required earlier.
The whole experience, while traumatic at the time, made it clear that COTS cobbling is the enemy not only of productivity, but also adaptability, functionality, performance, generality, cost flexibility, and a whole host of other metrics that we had previously though to be immutable facts of software life. Not so, poor scores on these dimensions over time turn out to be largely driven by the compromises that COTS dictate. It became clear that we should not stop there, but should re-examine the three remaining COTS items to see if even they could be eliminated to achieve greater performance. COTS programs are S or P programs (in Lehman’s language), not E-programs. Depending on S or P programs will usually be the undoing of an E-program. I would propose a 9th law to add to Lehman’s which is as follows:
Proposed 9th Law: “Each significant external software component that an E-program relies on for internal functionality, will incrementally reduce that E-program’s life expectancy.”
In other words, its not just about how you structure and maintain your own code, longevity is also just as much about the number and type of the external technologies you rely on. External reliances introduce as much if not more entropy than sloppy maintenance changes ever could.
Figure 2 above from 2003 onwards shows a smooth process of increased programmer productivity, core abstraction functionality, and gradual independence from the remaining COTS components. Such independence remains our top technical goal. It took 20+ years of experience on a single code base to realize how critically important this is. Alas, few programmers ever work on one large code base for so long, mainly because few projects survive this long, which in turn is because the developers rarely get to see how damaging external COTS can be over a multi-decade software lifespan. It is a vicious cycle that few can even perceive, and which by its very scale prohibits serious academic investigation. Trust me though, it is real.
Over the years following 2003, we implemented our own ontology-based GIS subsystem entirely built upon core Mitopia® abstractions. One more COTS component bites the dust. Other minor dependencies have also been systematically eliminated. Starting in 2008, a new core-abstraction based universal renderer approach has incrementally replaced the use of the 3rd party Genesis GUI framework through a kind of background coding task when time permits. At present, both renderers remain simultaneously active within a single instance of Mitopia® (through the new GUI interpreter abstraction). Different widgets can pick different preferred renderers (there are now others) so that at this point of time Genesis is used only in a few places (mostly visualizers). Soon it too will be optional, and the last COTS dependency, other than the underlying OS, will be gone.
The OS will undoubtably be next as Apple’s tendency to deprecate C APIs in favor of Objective-C replacements becomes sufficiently irritating that it will ultimately force a full scale elimination of OS dependencies. MitoSystems sees Objective-C as just another ‘fad’ language (i.e., COTS) to be avoided, even if platform mandated.
The moral of the story? If you want your program to live to a ripe old age, don’t trust COTS…or is that cats, I can never be sure.