Minerva Persistency

Overview

The Persistency Architecture of Minerva is the same as the existing Athena persistency Architecture, except for its C++-specific limitations. Persistency to Root files will be implemented via JDO API build on top of Java RootIO. This allows a possibility to switch persistency technologies easily. Persistifiable objects may be required to implement PersistenceCapable (or similar) Interface; unlike in C++, this serves just as a label and user is not required to implement any persistency-related functionality. The OID format and Catalogging/Bookkeeping (based on a relational DB) should be defined in a language-independent way insuring interoperability and exchangeability of data between C++ and Java world.

All Converters are based on standard technologies like Java Streaming API or JDO. They are generic, i.e. they are the same for whatever objects they manage (it is, however, possible to customise transient-persistent mapping, usually using XML files). Each Converter Service is just a kind of an Algorithm. Converters can be used not only for the Persistency Service but also to change representation of Data or to communicate Data between Processes.

Q:

Problems with the Athena Persistency

  1. Root is needed for interpretation of Root persistency, StoreGate is needed for interpretation of DataObjects. TObjects and DataObjects can't exist outside Root and StoreGate.
  2. Both RootSerialisation and RootCnv have approximately one converter per Object (often created semi-automaticaly, but heavily expanding the complexity of the system). It is highly un-trivial to find a proper converter in an polymorphic environment without realistic Reflection facility.
  3. Only Objects manageable by both Root and StoreGate (DataObjects) can be processed. Restrictions are defined not by the policy, but by the implementation deficiencies.
  4. There are many intermediate components (converters, Root dictionary, ...) complicating the processing. In theory, everything is automatic; in practice, such system is very hard to maintain and debug.
  5. No other IO implementation is available for the storage - for example XML or Relational DB. This implies that we are locked in a proprietary solution.
  6. Root Objects are different from DataObjects. We may not like the idea of accessing directly Roor Objects, but it will be what people will do in reality as they are much richer than DataObjects.
  7. Problems with DataLinks and Buckets are artificial: it doesn't exist on the level of Roor Objects, it is created by RootCnv and then resolved by StoreGate.
  8. All interfaces are defined in a language specific way.
  9. Converters can only be used to perform simple IO. It is not possible to chain converters and converters can't be used for communication between processes (which is generalized IO operation).
  10. StoreGate doesn't have clearly defined mission. Its tasks are (among others):
  11. To implement the functionality, StoreGate often violates agreed coding rules (use of macros,...).

Minerva Solution

Most of the mentioned problems can be easily solved by choosing the right technology. The proposed solution maintains the file compatibility with existing solution (and with RootIO file format itself) and it implements the same Architecture.

The key component of the proposed solution is the replacement of the C++ Conversion Service with the Java Streaming Service. Other services, not well performed or not performed at all in the current Design can be satisfied using out-of-box packages, like InfoBus and Java Transaction Service (JTA+JTS).

Java streams use similar concept as C++ iostreams. User just feeds her objects (data) into a stream or reads them from a stream. There are, however, three important differences between Java streams and C++ iostreams:

  1. Java streams do not use special syntax, while C++ iostreams are mostly accessed using << and >> operators. Java streams are represented by usual objects, so the syntax of Java streaming looks similar to the syntax of C-IO (the semantics is, however, more C++-like). The advantage of the standard syntax is the possibility to manipulate Java stream as any other object.
  2. Java streams can be nested (chained, piped). So, for example when a user wants to just write to a file, she does: 
    
FileOutputStream fos = new FileOutputStream("myfile");
fos.write("blabla");
    For compressing that file on the fly, she has to add: 
    
FileOutputStream fos = new FileOutputStream("myfile");
GZIPOutputStream gos = new GZIPOutputStream(fos);
fos.write("blabla");
    And to write out Objects (instead of just text), it's enough to do (using standard Java serialization file format): 
    
FileOutputStream fos = new FileOutputStream("myfile");
GZIPOutputStream gos = new GZIPOutputStream(fos);
ObjectOutputStream oos = new ObjectOutputStream(gos);
oos.write(myObject);
    >
    One can easily imagine similar stream using RootIO file format: 
    
FileOutputStream fos = new FileOutputStream("myfile");
GZIPOutputStream gos = new GZIPOutputStream(fos);
RootOutputStream ros = new RootOutputStream(gos);
ros.write(myObject);
  3. Java streams can be used not only to read/write files, but to send data over the network, to make connection between processes, to talk to hardware.

The implementation of the proposal requires to build two (or more, if needed) Java Root streams to access RootIO files from Java in the standard Java way. There are two missing pieces for implementation of Java Root streams:

  1. Writing part: The library for reading (all) RootIO files already exists, it has been written by Tony Johnson. Tony is currently working on making its performance better, he doesn't have time, in a near future, to implement the writing part. It should be, however, quite straightforward, as the structure of the files is already known, Tony is willing to help with his advice and writing is generally easier that reading (as we know what we want to write and is not necessary to support old versions).
  2. Conformance to the Java streaming IO: Existing Java RootIO reader doesn't implement standard Java streaming API.

This solution will further evolve into full-featured OODB solution using existing Java Data Objects (JDO) implementations. This will be possible thanks to already existing implementation of the JDO (Java Data Objects) based on Java Streaming API. While there is still a bit of work to be done (like definition of OIDs), this seems to be the most direct way how to implement the true OODB based on RootIO files.

The advantages of the proposed solutions are obvious, but it's nevertheless useful to summarize at least the most important ones:

  1. All converters are generic, there is only one converter per technology so all the machinery with "automatic" creation of convertors disappears.
  2. There are other standard streams, which can be immediately used (Java files, XML files, proprietary files,...).
  3. There are other JDOs, which can be immediately used.
  4. It satisfies all Atlas+DB architectural principles; the design is different as the C++ design is dictated by C++ faults. In particular, Transient-Persistent separation is automatically available without any need for complex Atlas-specific code (StoreGate, DataLinks, ...).
  5. Highly desirable interoperability between Java and C++ via RootIO is easily possible.

All exiting functionality is available, namely:

  1. Symbolic Links: Irrelevant thanks to Reflection.
  2. Const-access: Much richer access policy is available using InfoBusPolicyHelper.
  3. Data Buckets: see Analysis of Reachability Requirements.
  4. Data Links: see Analysis of Reachability Requirements.
  5. Keys: Much richer interface is available by InfoBus Names and Flavors.
  6. History Objects: Should be independent on persistency.

There are certainly other alternatives for implementing Java RootIO:

  1. Using directly C++ RootIO + JNI is possible (maybe even easier). It would be, however, absurd to carry the whole Root where it is not needed. We would have to create Root dictionaries and parse C++ with Cint to be able to use RootIO in Java !
  2. We could implement JDO directly on top of RootIO (in Java or even in C++). This may have some performance advantages (not so many layers), but we would loose a possibility of using Root files in a simple way as streaming objects. Also, it violates Atlas Event DB Architecture.


Analysis of Reachability Requirements

We have to satisfy following main services Data Mining is provided on top):

  1. Reading/Writing Objects from/to files.
  2. Local and Global Navigability and Addressability:

For the classification purposes, it is useful to define several kinds of objects:

  1. Second-class Object with Non-data-store identity doesn't keep its identity in persistent state. It is generally the responsibility of some First-class Object to restore the identity of the it.
  2. First-class Object with Data-store identity keeps its identity in persistent state within its Storage Unit. It's identity is restored by the DB/Streaming mechanism.
  3. First-class Object with Application (primary key) identity keeps its identity in persistent state across Storage Units. It's global identity is managed by the external service.

Local Navigability Global Navigability Local Addressability Global Addressability
C++ Streaming (TObjects)

TES (DataObjects)

Java Streaming (any)

JDO (any)

Releted Objects, which don't belong to the same EDO (and are for that reason not read in at the same time) are represented as Proxies. Proxies are created automaticaly, when reference to their original Object has to be written. When this reference is requested (for the first time), it initiases location and reading of the original Object. The whole EDO is read at the same time, so a Proxy is called only once per EDO (or following references are normal references).

Q:
  1. Proxies should share OIDs.
  2. How to update all proxies of the same EDO, once first Object is read in ?
ToDo:
  1. x
J.Hrivnac, Mar'03