Validation and Reporting Overview and Data Flow(Training-Test) Validation Workflow
The following chart illustrates the possible working process of validating an algorithm, with a user-defined training and test dataset:
(Internal) Curl commands for validating an algorithm
1. Init validation curl -X POST -d algorithm_uri="<algorithm_service>/algorithm/<algorithm_id>" \ -d training_dataset_uri="<dataset_service>/dataset/<train_dataset_id>" \ -d test_dataset_uri="<dataset_service>/dataset/<test_dataset_id>" \ -d prediction_feature="<prediction_feature>" \ -d algorithm_params="<alg_param_key_1>=<alg_param_val1>;<alg_param_key_2>=<alg_param_val2>" [OPTIONAL]\ <validation_service>/training_test_validation Subsequent (internal) curl calls: 2. Build model (the alg-parms could be a feature generation service/dataset, this has to be discussed): curl -X POST -d dataset_uri="<dataset_service>/dataset/<train_dataset_id>" \ -d prediction_feature="<prediction_feature>" \ -d <alg_param_key1>="<alg_param_val1>" \ -d <alg_param_key2>="<alg_param_val2>" \ <algorithm_service>/algorithm/<algorithm_id> -> <model_service>/model/<model_id> 5. Make predictions: curl -X POST -d dataset_uri="<dataset_service>/dataset/<test_dataset_id>" \ <model_service>/model/<model_id> -> <dataset_service>/dataset/<prediction_dataset_id> Get test-dataset (arrow is missing in chart): curl -X GET <dataset_service>/dataset/<test_dataset_id> 8. Get predictions: curl -X GET <dataset_service>/dataset/<prediction_dataset_id> At this stage one has to determine weather this is a classification or regression task. Use either the meta information of the model: curl -X GET <model_service>/model/<model_id> Alternatively use the feature type which should be included in the dataset (Numerical -> Regression else Classification) Return validation uri: -> <validation_service>/<validation_id> Reporting WorkflowReporting Workflow
General Remarks:
This is how a crossvalidation could work:
Reporting Workflow
The reports described here are reports for model validation (for the model developer) and prediction reports. Proposition for the report creation worflow:
Remarks:
The Fasttox ReportEven though the fasttox use case does not belong to validation, I would suggest to include the fasttox report (and other prediction reports) into the validation reporting facility
Formating the reports
Chart for formatting reports:
Remarks:
Special Report Formats
QMRFabout QMRF
QPRF
about QPRF, assembling of the data was discussed in a knowledge café discussion in rome proposition:
Document Actions |