Model Validation

This tutorial shows how to estimate quality of a given model with regard to error metrics, using a reference sample.

Note

This tutorial requires the approximation model trained in the Sample-Based Approximation tutorial.

Before You Begin

This tutorial requires an existing Tutorials project. If you have not created this project yet, see the Tutorial Project first. You will also need the workflow created in the Model Evaluation tutorial.

  • Open the Tutorials project.

  • Open the ApproxEvaluation workflow (ApproxEvaluation.p7wf).

  • Select File ‣ Save workflow as… to create a copy of the workflow. Save the workflow as ApproxValidation.

  • Switch to Workspace and verify that the workflow file (ApproxValidation.p7wf) is added to the project. You can see it in the Project pane.

  • While in Workspace, verify that the following files exist in the project:

    • model.gtapprox — an approximation model trained in the Sample-Based Approximation tutorial, and
    • approx_reference.csv in the samples subdirectory — a reference sample that is used for model validation.

  • Verify that you are editing ApproxValidation and continue with the tutorial.

Task

This tutorial is aimed at evaluating the prediction quality of the model trained in the Sample-Based Approximation tutorial. This quality is estimated using a standard set of error metrics, with the R2 and RMS being most frequently used. Errors are calculated on the sample that stores the values of model variables and responses. This can be the same data sample that was used to train the model. But in this case, the error values are expected to be small and model accuracy tends to be overestimated. Using a reference data sample is recommended as such validation method is more reliable and informative. The reference sample to evaluate the quality of our model is a file approx_reference.csv located in the samples subdirectory of the Tutorials project.

Solution

There are two ways of estimating model quality in pSeven. You can solve this task in a workflow with an Approximation model block. Another way is to use Model validator in Analyze.

Solving the tutorial task in a workflow requires the following general steps:

  • Configure the Approximation model block to validate a model.
  • Remove the Test Sample block that generated a data sample in the Model Evaluation tutorial.
  • Add a CSVParser block and configure it to read reference data from the approx_reference.csv file correctly.
  • Prepare the sample for the Approximation model block using a Submatrix block. The block has separate ports for the input and response parts of the reference sample, so it will split a matrix received from CSVParser to get two separate samples.
  • Send the values of variables and reference data to Approximation model. When you run the workflow, the block will calculate model responses for given inputs and also error metrics that you selected. Then you can configure the report in Analyze and compare reference sample outputs with model predictions using a 2D plot.

The steps to evaluate model quality with a Model validator are as follows:

  • Import an approximation model to an existing report, so that it appears in the Models pane.
  • Import reference data from file to the report database.
  • Add a new Model validator and configure it to validate the given model using the imported reference data.

Error Metrics

Begin with configuring the Approximation model block to validate a model.

  • Double-click the Model block to open its configuration. On the Configuration tab, switch to Advanced.
  • Use the Validation errors selector to specify the error metrics to be calculated. For example, select R2 and RMS.
../_images/page_tutorials_approx_valid_error_metrics.png

When you select the errors, the block automatically adds new ports that output calculated errors.

  • Switch to the Ports tab and verify that f.Error.R² and f.Error.RMS outputs 1 are added. Note that the block also creates f.Reference, Responses.Error.RMS, and Responses.Error.R² ports, but they will not be used in the scope of the tutorial. If you select more error types, this set of ports is created for each requested error.
../_images/page_tutorials_approx_valid_ports.png

To compare model predictions with reference outputs, you will need to capture the activity of the input port that accepts the reference sample and the output port that receives model predictions.

  • Remove all previous monitoring settings.
  • Enable monitoring for the Model.Responses.Reference 2 and Model.Responses 3 ports.
  • Verify your settings and click OK to save changes and close the configuration dialog.

Loading the Sample

The next step is to reconfigure the existing workflow as the Test Sample block used to generate a random test sample in Model Evaluation tutorial is no longer required.

  • Remove the Test Sample block. The link connecting this block to Model will also disappear.

Now you can add a block that will load and parse the reference sample, approx_reference.csv.

  • Add a CSVParser block. Name it Sample.

Open the Sample block configuration. When you load a file, the block displays a results preview (parsed sample) in the Preview pane. Use it to check whether the file is parsed correctly and adjust settings when needed.

../_images/page_tutorials_approx_valid_parser.png
  • In the Input file pane, click b_browse and browse to the sample file in your project (approx_reference.csv). Note that the block automatically saves a relative path to the file, so it will work even if you move the project to another location.
  • Verify that the file is parsed correctly using the Preview pane. Since approx_reference.csv uses the delimiter and comment characters which are default in CSVParser configuration, you do not need to change any settings here.
  • Note the Parse first row as header setting. It should be enabled to read names from the first non-commented line and use them in the workflow. When the names are parsed correctly, you can see them in the Preview pane as column titles.
  • Leave other settings default and click OK.

When the workflow starts, Sample will read the CSV file, convert the data to a RealMatrix and output this matrix to the matrix port.

Preparing the Sample

The matrix output by the Sample block cannot be sent directly to Model because it contains both input (variable) and response values. Model accepts the input and response reference data to different ports, so the reference sample should be split into two parts.

To separate the input and response values you can use a Submatrix block.

  • Add a Submatrix block. Name it Splitter.

Open the Splitter block configuration. This block allows you to define submatrices that include specified rows and columns from the input matrix. To add a submatrix, click b_blconf_add in the toolbar (default configuration is empty).

Select the input part of the reference sample.

../_images/page_tutorials_approx_valid_submatrix_x.png
  • Specify the name: x_sample.
  • Leave rows default (: selects all rows).
  • Specify columns: 0-1 (the values of variables are first two columns, indexed 0 and 1).

Similarly, select the response part.

../_images/page_tutorials_approx_valid_submatrix_f.png
  • Name: f_sample.
  • Rows: : (default, all rows).
  • Columns: 2 (the third column in the file, indexed 2, contains the response values).

Verify the Splitter block configuration.

../_images/page_tutorials_approx_valid_splitter.png

Adding a submatrix automatically adds a corresponding output port to the block, so Splitter now has two outputs: x_sample and f_sample, and one default input: matrix which you can see on the Ports tab.

  • Finally, link Sample.matrix to Splitter.matrix.

This completes the sample preparations. Splitter will receive a matrix from Sample, split it according to your submatrix settings, and output the input and response parts to the Splitter.x_sample and Splitter.f_sample ports. Next, these ports have to be connected to Model inputs.

Workflow

To complete your workflow configuration, set up the following links:

  • Connect Splitter outputs to Model inputs:
    • Splitter.x_sample to Model.Variables.
    • Splitter.f_sample to Model.Responses.Reference.
  • Uplink the Model.f.Error.R² and Model.f.Error.RMS outputs so that you can see error values in Run after the workflow finishes.

For convenience, you can assign shorter aliases to the f.Error.R² and f.Error.RMS ports. To edit the names:

  • Open workflow configuration b_runconf and specify new aliases for Model ports:
    • R² for f.Error.R².
    • RMS for f.Error.RMS.
../_images/page_tutorials_approx_valid_alias.png
  • Verify your settings and click OK to save changes.

The final workflow is very simple. The Sample and Splitter blocks are used to prepare the reference data for Model. Model calculates model responses (predictions) for reference inputs and validates these predictions against reference outputs, calculating the error metrics that you specified.

../_images/page_tutorials_approx_valid_workflow.png
  • Save the workflow and run it to see the results.

Results

After the workflow finishes, you can see calculated error metrics in Run.

../_images/page_tutorials_approx_valid_outputs.png

Use these values to decide if the model is performing well. For example, R2 close to 1 suggests good predictive ability of the model, while lower values suggest otherwise. RMS value is close to 0, which also reflects good ability of the model to accurately predict the data.

You can also use a 2D plot to compare reference sample outputs and model predictions.

  • In Analyze, create a new report and open the Data series pane.
  • Select the Model.Responses and Model.Responses.Reference records in the project database. Drag them to the Data series pane.

Names of data series will be used in plot labels, so it is recommended to rename them for convenience.

../_images/page_tutorials_approx_valid_report.png
  • Rename the data series as follows:
    • “Model.Responses” to “Model predictions”.
    • “Responses.Reference” to “Reference outputs”.
  • Add a 2D point plot. The dataset configuration is Model predictions as D0 and Reference outputs as D1.
../_images/page_tutorials_approx_valid_2d_plot.png

See section Plotting in the Results and Reports tutorial for a guide on how to add and configure plots.

  • Save the report as ValidationReport and continue.

Model Validator

You can also estimate model quality using the Model validator tool in Analyze. It allows you to test a model against reference data and make a decision on its accuracy using error plots and statistics.

To validate a model, you should first import it to the report saved in the previous step of the tutorial.

  • Use the b_analyze_import_command button on the Models pane’s toolbar or the Import from file… command from the b_context menu to import the approximation model (model.gtapprox) so that it appears in the Models pane.

The next step is to import reference data to the report database.

  • Click b_analyze_import_command button on the Data series pane toolbar and select the Import data from CSV… command. This command opens the import dialog.
  • Select approx_reference.csv in the import dialog and adjust the CSV parser settings. See section Import Data from CSV for details.
../_images/page_tutorials_approx_valid_import_csv.png
  • Leave all settings default and click Import to add data to your report. As a result, three data series “x1”, “x2”, and “f” 1 will appear in the Data series pane.
  • Select the model in the Models pane 2 and select the recently added data 1. The data will be used as a test sample for validation.
  • Click the Model validator button 3 on the report toolbar.
  • In Model validator, switch the Sample selector 4 to “test”, as the reference data comes from the test sample.
../_images/page_tutorials_approx_valid_quantile_plot.png

Model validator shows the quantile plot by default. Generally, steeper curves on a quantile plot are better: a steep curve means that higher fraction of points has low prediction error. If you switch the plot type from quantile to scatter in the Plot selector, you can see the same plot as was obtained after the workflow run. The table at the bottom 5 shows error metrics.

The tool can be also used to validate many models at once. You can add several models from a report and compare them in terms of quality using plots and error metrics. The best metric values are highlighted.

Model validator has a special mode that calculates errors between reference and prediction data samples after comparing them. In this case, the model is not required. You can use the input data from the reference sample to calculate predicted values of model outputs — for example, using the Make predictions… command and then compare these outputs with the ones of the reference sample. Configuring a Model validator to work in this mode is out of the scope of this tutorial.

Conclusion

This tutorial explains the essentials of model validation in pSeven. You can estimate model quality in a workflow with the Approximation model block. Another option is to use a dedicated Model validator tool in Analyze.

It is recommended to validate a model on the reference sample. The reference is data which is not included in the training dataset, but was sampled following the same probability distribution. In most cases, such validation approach tells us more of model quality than validation on a training set and cross-validation. Therefore, it is better to split existing data into train and test subsets in proportion 80% to 20%. 80% of the dataset will be used for model training and the remaining part will become a test sample. To split the data, you can use the Split data… command in Analyze. See section Split Data for more details.

In practice, splitting the dataset into train and test subsets is more reasonable than using all available data for model training. Using the independent sample as reference data for validation helps reduce the risk of model overtraining. This is also a preferred approach when training a model in the SmartSelection mode.

Validation can also be used when making decisions on model’s “relevance”. First, you train the model and then analyze its predictions in terms of their relevance to new data which was obtained only after training or was unavailable before or not used when training. In this case, new data is used as a reference sample to estimate model’s quality. Based on validation results you may further decide to retrain or update a model. These advanced methods are out of the scope of the tutorial. See section Predictive Modeling Tools in Results Analysis for details.