Test Results

During iteration #2 the following OpenTox services and implementations have been developed and thoroughly tested:

1. Compound (IDEA, IST)
2. Feature (IDEA)
3. Dataset (IDEA, IST)
4. Algorithm (IDEA, IST, NTUA, TUM)
5. Model (IDEA, IST, NTUA, TUM)
6. Validation (not tested)
7. Task (IDEA)
8. Feature Ontology (not tested)
9. Algorithm Ontology (not tested)
10. Endpoint Ontology (not tested)

The major findings are:

• a number of issues of various severity in practically all services and implementations, which have been identified, reported to the respective developers and partially or fully resolved ;
• interoperability between different implementations has been successfully demonstrated in only one single (carefully crafted) case, involving IDEA's Dataset and NTUA's Model service implementations (please see the matrix below);
• better interoperability between different implementations might be already (very close to) achievable, but time pressure and/or lack of proper guidance/documentation in some cases have prevented any further interoperability tests to succeed;
• the model building procedures seem to be quite fragile, subject to frequent failures, lack of interoperability (even in the framework of a single implementation, e.g. IST and NTUA), and/or were not fully API v1.1-compliant (e.g. TUM) at the time when the tests were performed;
• IDEA's implementation covers the largest API subset so far, has relatively good performance (measured by continuous SmokePing application level latency and availability measurements, publicly available online) and exhibits reasonable interoperability between Compound, Feature, Dataset, Algorithm, Model and Task services in the framework of a single implementation;
• the Feature and Task services are present only in IDEA's implementation so far;
• the implementation of Validation service is ongoing at ALU-FR and more or less in sync with other developments AFAIK, however it wasn't tested during iteration #2, mainly due to the lack of a Validation service public test instance deployment;
• some of the OpenTox services (e.g. IST's implementation of Compound service) partially or even fully rely on 3rd party services (e.g. CSLS) and in some cases are subject to outages and/or high latency/timeouts, especially during periods of high demand; this could be observed in more details in SmokePing's continuous measurements, which are publicly available online;
• IBMC's OpenTox service implementation has been left out of the SmokePing measurements, because it is not API v1.1 compliant;
• SL's server network connectivity suffers from high latency and substantial packet loss (up to 60%); network connectivity has to be either radically improved by some means or (if this proves to be difficult or even impossible), then any software, developed by SL (when eventually it becomes available) should be deployed in a more network-friendly location, e.g. somewhere in Europe.

Further analysis of the test results at the end of iteration #2 and comparison against the desirable level of interoperability between independent service implementations yields the following graphical summary:

A major goal for the next (third) development iteration would be to provide solid grounds for a much greener interoperability matrix, according to the stated priorities, which might be of course adjusted after further discussion. For example, each Dataset service implementation should be fully compatible with all independent OpenTox implementations of Feature, Algorithm, Model, Validation and Task services and this should be a high priority goal for all involved parties. The same holds true for every other tuple of independent OpenTox service implementations, which have a 'high priority' intersection (marked with 1) in the matrix. It might be wise to come up with a definitive list of service types/implementations which are expected to be fully compatible by the end of iteration #3. Furthermore, detailed guidance on how to make use of this interoperability on a case by case basis should be readily available and successfully passing all tests.

Another important observation is that there are missing bits in the Feature API, related to the "Get a list of available features" request. The specific query needs to be specified and a link to the ontologies has to be established. Further work is required for algorithm parameters standardisation and parameter description standardisation which should be taken into account somehow in opentox.owl. The same holds true for Model parameters as well. Another important issue is the introduction of PMML support and the provision of model migration functionality (e.g. service A should be able to run a model that has been created at service B). Moreover, the Feature Ontology, Algorithm Ontology and Endpoint Ontology APIs need to be designed, specified and documented. In particular, they should provide means for querying whether the corresponding algorithm, model or feature is owl:sameAs some user specified value from the respective ontology.

The implementation of the ToxPredict and ToxCreate prototypes would be highly dependent on the existence of some subset of interoperable services. At the time of this writing such interoperability has been found to exist mainly between the services, which are part of IDEA's implementation. It would be desirable to achieve a reasonable level of interoperability at least between a subset of services from different implementations. ToxCreatel might require substantially more efforts and higher level of commitment, because:

• the second iteration tests have revealed quite a lot of weaknesses in several of the model creation routines in different implementations (e.g. IST, NTUA);
• we haven't tested at all any of the validation routines, neither TUM's model creation routines so far;
• IDEA's implementation currently has very rudimentary model creation support and no Validation support at all.

Last but not least, it might be wise to consider making use of "in house" data and resources whenever they are readily available, instead of relying solely on 3rd party services like CSLS, PubChem, etc..., which might be not always available (24h/7d), could implement severe rate-limiting policies and might refer to dubious data, which quite often (e.g. in about 5-10% of all ECHA pre-registered substances related queries) needs manual inspection and curation. In this particular case by "in house" I mean within the community of interoperable OpenTox webservices. Of course, whenever the data cannot be found "in house", a subsequent (failover) query could be send to those 3rd party services, without risking to the same extent to hit their rate-limiting policies and without experiencing the performance drop, which is usually associated to application latency issues of these services especially during periods of high demand. BTW, it is somehow ironic and even sad that so far we seem to have achieved better interoperability with 3rd party services through their respective APIs, than between OpenTox services themselves.