March 1, 2018

Workflow Hierarchy, Import and Export

Task decomposition and structuring are very important in engineering. It allows to solve problems step by step and helps to understand dependencies and links between problem parts. pSeven inherently supports hierarchy in workflows thanks to the Composite block. It is a special block which functions as a wrapper for a group of blocks or another workflow. Grouping blocks into Composite give the following advantages:

• Visually simplifies and logically structures a workflow.
• Provides the ability to re-use certain workflows or their parts in new workflows.
• Enables collaborative work on complicated workflows.
• Simplifies the approach to parallel computation.
• Allows re-using previously evaluated designs stored in the Composite data cache.

Decomposing a Workflow Into Subtasks

Consider a dimension reduction (DR) workflow (a similar one is found in the pSeven examples):

The workflow does the following:

1. Train a DR model using the training data sample and pass it to the compressor and decompressor.
2. Read data for computation and compress it using the trained model.
3. Process data and make some calculations.
4. Restore processed data into the original design space, using the trained model, and output this data as the result.

Let us see how can we improve this workflow. We can group the Train Sample and Train blocks into the single Prepare Model block, and also group Huge Data and Compressor into the Prepare Data block:

As a result, we have the following improvements:

• Better visual representation. Workflow is easier to understand, the Prepare Model block clearly starts the workflow.
• Better logical structure. The Prepare Data block now depends on Prepare Model. We do not need to read huge data if DR model fails to build for some reason.

Note that pSeven tries to run blocks in parallel if possible, so in the initial workflow both CSVParser blocks launch simultaneously. In the changed workflow Prepare Data will be started only after Prepare Model finishes, since a Composite block behaves like a separate workflow, and in order to start it needs data to all connected inputs.

Re-Using of Workflows

An optimization problem with an external solver is a typical task for pSeven, but working with external solvers usually requires some preparations. We can make a workflow to run the solver (prepare data in a text file, execute, read values from the resulting text file and send them to outputs) and test it without Optimizer. This will save us a lot of time since we can test our solver on different input data without configuring and running other blocks.

After the Solver workflow is ready, we can import it as a Composite block into an optimization workflow and start working on an optimization problem. Similarly, the same Solver workflow can be used to train an approximation model.

Importing a workflow copies it inside a Composite block and workflow inputs and outputs become ports of this block. New port names are copied from the aliases set for workflow inputs and outputs in workflow configuration. If the aliases were not set, then new names are copied from the original port names. This way the new names are always the same as the names seen in Run for the imported workflow. Note that if you import into an existing block, the workflow replaces all block contents. It is possible to reimport workflow into resulting Composite if ports and types of Composite are compatible with imported workflow ports, links to Composite will be preserved. Also, a path to workflow is stored inside Composite during import process and used during reimport (you can always change it in Import Workflow dialog). Advanced settings of Import Workflow dialog allows you customize resulting Composite block:

• Copy workflow input values as defaults of Composite input ports.
• Merge imported workflow configuration parameters into resulting workflow configuration.
• Keep monitoring settings from the imported workflow.

Along with import, it is possible to export any Composite block into a separate workflow and re-use it the same way as Solver above. After export, block input and output ports become workflow inputs and outputs seen in Run. Export keeps parameter and monitoring settings related to the exported block, but not parameter presets. This means that monitored ports of nested blocks will also be monitored in the exported workflow, and parameters will be added to workflow configuration, but their values are not kept (the workflow will contain only an empty default preset).

Workflow hierarchy is a fundamental pSeven feature. It allows to use the top-down approach to workflow construction and enables collaboration. Composite blocks, workflow import and export facilitate working with complex workflows developed together. Other features, such as parallel execution and caching make Composite essential for every pSeven user.