Chapter 9. Transforms

Once the SchemaTree for a model has been updated as required, the next stage in the process is for transforms to be applied. These convert the Schema Tree into different formats suitable for subsequent deployment. Transforms are all implemented in XSLT.

All transforms are done using a tool called schematic that is included with a Theon installation. This tool builds a single XML-Schema document for a SchemaTree and can optionally apply a transform to that file. It can also optionally validate the SchemaTree.

Since all the results of the transforms are re-generatable (by re-applying the transforms to the same version of the SchemaTree) they do not need to be retained beyond testing/usage. The results included for a packaged model in a Theon installation are primarily there for convenience (performance) as they can all be re-generated on demand from the actual SchemaTree (also included in the installation). However, as the SchemaTree for any model is altered then so must all these transformations be re-applied for consistency across all applications that use the results.

9.1. DDL

The DDL transforms build a set of SQL DDL commands (much like the basic output of pg_dump -s) to create a physical database that will be the realisation of a model held in an SchemaTree. Ultimately, this is how a database managed in Theon is installed. The result of these transforms, a directory of sql files, is written into the current working directory by default and the primary file (ddl.sql) is directly applied to psql to create the database. The generated files are included in a packaged ModelLibrary and used in the installation and bootstrap process.

A feature of schematic that is only used in the DDL transforms is differencing. Here schematic is given two versions of a SchemaTree. Normally this would be two different revisions in the version control repository, but it could also just be a tree at two different file locations. Then the generated DDL SQL will comprise of only those necessary statements to alter the structure of a physical database from that of one SchemaTree to the other. This robustly automates the upgrade of a database structure to a new version without a data unload, destroy and data reload step (which in itself is not robust depending on the nature of the changes).

Implementation of differencing for all possible schema changes in PostgreSQL would be extremely complex (particularly when moving beyond purely additive changes) and so Theon at the moment only provides the framework. Transitive rules need to be manually written to cater for each specific change to support.

9.2. DDLDoc

The DDLDoc transforms build a tree of web pages that act as the documentation for the physical database aspects (Dictionary cluster) of a model held in a SchemaTree. Assuming all the relevant elements in the model have been fully annotated then the output of this transform can act as the sole documentation for a physical database schema. These transforms also automatically build SVG diagrams of entities, their attributes and their relationships. The result of these transforms - a directory of HTML web pages, index pages and SVG files - is written into the current working directory by default, and a local browser can simply be pointed at the root to access the content. The generated documentation tree is included in a packaged ModelLibrary.

A variation of these transforms is also used internally to build Docbook formatted output also included in a packaged ModelLibrary.

9.3. UI

The UI transforms build datasets of JSON and SHTML files that are used by the TheonUI as metadata to present and manipulate table content in the physical database described by the model, as well as to control the function of desktops and panels designed for business workflows associated with the model. The result of these transforms - a directory of structured metadata files - is written into the current working directory by default, and the TheonUI server is configured to point at this directory so that its functions will be based around that particular dataset. The generated dataset directory is included in a packaged ModelLibrary and TheonUI is configured to operate out of this directory by default.

9.4. UIDoc

The UIDoc transforms build a tree of web pages that act as the documentation for the TheonUI aspects (Reflection cluster) of a model held in an XML-Schema tree. Assuming all the relevant elements in the model have been fully annotated then the output of this transform can act as the sole documentation for TheonUI desktops and panels. These transforms also automatically build SVG diagrams of desktops and panels structure (entities, links and connectors). The result of these transforms - a directory of HTML web pages, index pages and SVG files - is written into the current working directory by default, and a local browser can simply be pointed at the root to access the content. The generated documentation tree is included in a packaged ModelLibrary.

A variation of these transforms is also used internally to build Docbook formatted output also included in a packaged ModelLibrary.

9.5. Coupler

The Coupler transforms build scripts used as the executor for processing data in streams. These scripts effectively format the external data in a stream and load it into the physical database described by the model. They then trigger the processing of individual Couples defined against the stream. The result of these transforms, a directory of Python scripts, is written into the current working directory by default. The TheonCoupler server would be configured to point at this directory so that its functions would be based around that particular set of scripts. The generated coupler configuration is included in a packaged ModelLibrary and TheonCoupler is configured to operate out of this directory by default.

9.6. CouplerDoc

The CouplerDoc transforms build a tree of web pages that act as the documentation for the TheonCoupler aspects (Streams and Couples) of a model held in an XML-Schema tree. Assuming all the relevant elements in the model have been fully annotated then the output of this transform can act as the sole documentation for TheonCoupler. These transforms also automatically build SVG diagrams of the Coupler structures (source/target entities and their "sync" relationships). The result of these transforms - a directory of HTML web pages, index pages and SVG files - is written into the current working directory by default, and a local browser can simply be pointed at the root to access the content. The generated documentation tree is included in a packaged ModelLibrary.

A variation of these transforms is also used internally to build Docbook formatted output also included in a packaged ModelLibrary.

9.7. Model

The Model transforms are used internally to Theon, and build PostgreSQL COPY datasets for each element in the XSDDB. These transforms are what facilitate the content of XSDDB being loaded from a SchemaTree. The result of these transforms, an SQL file per-XSDDB element containing a COPY statement and a set of records, is written into the current working directory by default, and other commands then use it to update XSDDB. The generated element load files are included in a packaged ModelLibrary and used by Theon for initial bootstrap of XSDDB where the user has not specified an alternative.