User blocks guide¶

In pSeven Enterprise, you can extend the block library with your own blocks. To do this, you need to develop a block prototype and load it into the block library. Then you can add a block from the library to the workflow.

Block prototype example¶

The Common library section of the block library provides an example of a custom block that you can use as a basis for developing your own block. To get started on your block, download the example block from the library:

Create a new workflow or open an existing one to access the block library.
In the Common library section of the block library, drag Example user block from the Special subsection onto the Explorer pane.
In the Explorer pane, use the Download files command to download the Example user block folder to a local folder on your computer.

Important

To drag a block from the library onto the Explorer pane, you must have the app_auth | user | Can develop blocks permission; otherwise, a message appears stating that you do not have permission to copy blocks from the block library. By default, every pSeven Enterprise user has this permission, but the administrator can remove it for individual users. Such users will not be able to develop blocks.

As a result of these steps, the local folder on your computer contains the Example user block.zip archive. Unzip it to extract the Example user block.p7protoblock folder. This folder contains sample code from which you can start developing your own block.

Important

The name of the folder containing the block code must have the .p7protoblock extension; otherwise, the block library will not recognize it as a block prototype.

Block implementation overview¶

Review the block example code you downloaded for implementation details. Pay attention to the comments in the code files. The block implements an HTTP server with two run modes:

The edit mode is used to work with the block during workflow setup.
The run mode is used when the block is executed during a workflow run.

In edit mode, the block server must handle the following requests:

Open block - GET request to load the block UI.
UI commands - POST requests issued by the UI when configuring the block.

In run mode, the block server must handle POST requests passing commands from the workflow runtime.

The main elements of a custom block implementation are:

Manifest - a YAML file that identifies the block, specifies its execution environment, startup commands, and other block properties. This file must be named manifest.yaml and be located in the root of the .p7protoblock folder containing the block code.
Frontend - web UI for configuring the block. Frontend requests are passed to the block server.
Backend - the HTTP server that handles requests both from the frontend (in edit mode) and from the workflow runtime (in run mode).

For implementation details, see the code comments in the custom block example, starting with the comments in the manifest.yaml file.

Developing a custom block¶

To create a new custom block, start with the code provided in the Example user block.p7protoblock folder, and follow the comments in the sample code files.

When starting to develop a new block, you should change the block ID - the id parameter in the manifest.yaml file. Otherwise, your block might conflict with other blocks in the library. A block ID is a UUID in the hexadecimal string form, a 32-character hexadecimal string without dashes. The ID for the new block in the required format you can obtain, for example, using Python:

import uuid
print(uuid.uuid4().hex)

When developing a new version of a block, do not change the block ID. To distinguish between different versions of a block, use the version parameter in the manifest.yaml file (see Block version numbering).

There is no prescribed folder structure for your code. Having loaded the block, its manifest is used to locate UI files, startup scripts, and so on, so you are free to choose the folders for your code files, provided you have made the necessary changes to the manifest. However, it is more advisable to stick to the folder structure in accordance with the default manifest settings, placing the appropriate parts of the code in the userblock, scripts and static folders.

The top-level folder name extension must be .p7protoblock, such as folder-name.p7protoblock. You can set the folder name at your discretion. Consider following a consistent folder naming scheme that specifies the name, ID, and version of your block. This will help you keep track of your custom blocks and their code versions, and will also avoid possible name conflicts when updating your block in the pSeven Enterprise block library. Recommended folder name format is <block-name>-<ID>-<version-number>.p7protoblock.

Block version numbering¶

The pSeven Enterprise block library allows you to load different versions of a custom block with the same ID. To distinguish between block versions, the version parameter in the manifest.yaml file is used.

When starting to develop a new block (with a new block ID), set version to 1.
When starting to develop a new version of an existing block (keeping the block ID), increment the value of the version parameter.
When testing and debugging a block during development, do not change its version number.

Tagging the block under development¶

When developing a new version of a custom block, it is highly advisable to give it the DEV tag, indicating that the block is under development. This tag is added through the tags parameter in the manifest.yaml file:

tags: [{"": "DEV"}]

The example user block that ships with pSeven Enterprise (see Block prototype example) has the DEV tag added by default.

The main purpose of the DEV tag:

Mark the block in the Block library pane. The DEV tag shows up next to the name of the block, indicating that this block is under development and should not be published to users.
Disable the code cache for blocks intended to run on Windows extension nodes. When pSeven Enterprise first runs a block on an extension node, it saves a copy of the block code in the node's local cache to avoid copying it to the node every time that block runs. Your modified block of the same version without the DEV tag would conflict with its earlier state stored in the cache. Disabling the cache allows the modified block to run with the same version number.

Disabling the cache may cause significant time delays when configuring the block or running it within a workflow. When you are done testing and debugging the new version of the block, remove the DEV tag before you proceed to publish the updated block. For example, if you publish the block without any custom tags, the list of tags in the block's manifest should be empty:

tags: []

If you publish the block with custom tags, the list should include only your tags, without the DEV tag.

Adding Python modules¶

If your block uses any Python modules not provided by pSeven Enterprise, these modules need to be added to the block prototype. The module distribution and all its dependencies must contain only Python code. Such distributions are commonly referred to as pure Python module distributions.

To add modules from a distribution to your block prototype, use the following command: pip install --target <directory> <distribution> where <directory> is the path to your block prototype folder (.p7protoblock), and <distribution> is the name of the distribution. Example:

pip install --target CustomBlock.p7protoblock extra_lib

This example adds the modules from the extra_lib distribution to the block prototype held in the CustomBlock.p7protoblock folder that is located in the current working directory.

Installing a module also installs all its dependencies to the block prototype. Once you have installed a module, you can import it in your block code as usual:

import extra_lib

Block assets¶

If your block needs to read data from a file, which should not be placed into the block prototype, you can add that file to block assets in the shared data storage. Typical examples of such files are:

A database or other file with reference data that updates more often than you update the block.
A simulation model, which the block runs.
A static file that does not change even if you update the block, or rarely changes.

Using block assets to store large files (models, databases) can save much disk space, but you will have to keep in mind that pSeven Enterprise does not handle any dependencies between your block and assets it may use.

When a user first adds your block to a workflow, that block is copied to the workflow library. This creates an independent copy of the protoblock folder in the workflow. All block instances in the workflow work with that copy of the protoblock folder and do not synchronize it with the protoblock in the shared data storage. Synchronization happens only if the user accepts a block update in workflow library. Keeping a copy of the protoblock in the workflow helps pSeven Enterprise to avoid many issues related to block version upgrades.

Block assets are not handled by pSeven Enterprise in a similar way. Assets are never copied automatically, and you have to track dependencies between block versions and assets yourself. If your block depends on a file from assets, all block instances in all user workflows read that same file. Moreover, in a given pSeven Enterprise deployment all block assets are common to all user blocks in that deployment - so other block developers may be using the same asset your block uses.

Before you start using block assets, consider the following:

Assets are read-only - your block cannot write to a file in assets, because at the same time that file may be in use by another block. In general, no block has write access to asset files.
Block developers and administrators of a pSeven Enterprise deployment, where block assets are used in development, have to be extra careful when publishing, updating and otherwise maintaining blocks because all dependencies between blocks and assets have to be tracked manually. Note that dependency problems are easily avoided by adding all required files to the block prototype, which makes the block self-contained.
If different versions of your block need different versions of an asset file, you will have to keep each version of that file in assets. Removing an outdated version of an asset file is risky even if you have already removed the block version that had used it.
When you remove, rename, or otherwise modify a file in assets, you risk creating issues in user workflows. As noted above, workflows store independent and non-synchronized copies of your protoblock, and some users may decide to keep an outdated version of your block in their workflows. Those workflows will likely run into an error if you remove an asset file required by the user's version of the block. Note that even if you completely unpublish an old version of a block, user workflows can still keep a copy of that version in the workflow library and continue using it. There is no way to force a block update or removal in user workflows.

An asset is a folder, which may contain many asset files or subfolders with files. To make those files available to user blocks, the asset has to be published by uploading the asset folder to the pSeven Enterprise shared data storage. Publishing an asset requires full access to the shared data storage, so it is done by a deployment administrator (see Publishing and updating assets). After an asset has been published, any user block can declare that it requires files from that asset (see Reading files from assets). Note that there are no restrictions for using the same asset in various blocks (with different prototypes).

Publishing and updating assets¶

Block assets can be published and updated only by a pSeven Enterprise deployment administrator who has full access to the shared data storage. To publish an asset, the administrator needs to copy the asset folder to the assets folder in the shared data storage and set permissions so pSeven Enterprise has read and write access but user blocks have read-only access to files in the copied folder. The steps to update a previously published asset are the same - new versions of asset files are simply copied over existing ones - but keep in mind that pSeven Enterprise does not track dependencies between user blocks and assets. Before you update an asset, verify that the update does not create issues in any of the blocks requiring files from that asset.

How to locate the shared data storage

The shared data storage location is specified by the storage.shareddata.* parameters in the deployment configuration file values.yaml (see the pSeven Enterprise deployment guide).

Follow the steps below to publish or update a block asset:

Connect to the shared data storage server as a user with root permissions.
Upload the asset: copy the asset folder (for example, common_data) to the <shareddata>/assets folder on the server, where <shareddata> stands for the location of the shared data storage. In this example, the path to the uploaded asset folder would be <shareddata>/assets/common_data.

The name of the asset folder should contain only lowercase English letters, digits, and underscores (the [a-z0-9_] character set); other characters are forbidden in asset names. pSeven Enterprise does not check the folder name when you publish an asset, but forbidden characters in the name will cause an error in any user block that declares a dependency on that asset.
You can use symlinks if you want to avoid renaming assets

If the asset folder name contains forbidden characters but you want to avoid changing it, you can create a symbolic link to the uploaded asset folder, giving the link a proper name. For example, the folder may be named Common-Data, which is an invalid name for an asset (uppercase letters and - are forbidden). Instead of renaming the folder, you can upload it as is and then create a symbolic link named common_data:
```
# ln -s <shareddata>/assets/Common-Data <shareddata>/assets/common_data
```
After this, user blocks can specify the asset name common_data to access files in the <shareddata>/assets/Common-Data folder.
pSeven Enterprise connects to the storage as a system user with UID 11111. That system user must be the owner of asset files and have read-write permissions in assets. A block connects to the storage as the user who configures or runs that block, so all users except system must have read-only access. For example, if you are publishing an asset named common_data, you can set the required permissions as follows:
- Set the pSeven Enterprise system user as the owner:
```
# chown -R 11111:11111 <shareddata>/assets/common_data/
```
- All asset subdirectories must have drwxr-xr-x permissions:
```
# find <shareddata>/assets/common_data/ -type d -exec chmod 755 {} \;
```
- Asset files must be read-only to everyone except the owner and must keep the executable flag (x):
```
# find <shareddata>/assets/common_data/ -type f -exec chmod "u+rw,go+r,go-w" {} \;
```

Reading files from assets¶

After an asset has been published to pSeven Enterprise, your block can read asset files. To get access to files, the block must declare its asset requirements, which are specified by the following parameters in the block's manifest (manifest.yaml):

edit.assets - a list containing names of assets your block requires in the edit mode.
run.assets - a list containing names of assets your block requires in the run mode.

For example:

edit:
    assets: [common_data, reference_docs]

run:
    assets: [common_data, database]

The above example assumes there are published assets named common_data, database, and reference_docs in your pSeven Enterprise deployment. When your block starts, pSeven Enterprise reads its manifest and, for each name <asset_name> in edit.assets and run.assets, populates the block environment with a variable named DA__P7__BLOCK__ASSET__<ASSET_NAME>, where <ASSET_NAME> is the uppercased name of an asset. Environment variables DA__P7__BLOCK__ASSET__<ASSET_NAME> contain paths to their corresponding asset folders - so the block receives asset paths through its environment variables.

Continuing the example above, the following environment variables would be added:

DA__P7__BLOCK__ASSET__COMMON_DATA containing the path to the common_data asset folder,
DA__P7__BLOCK__ASSET__DATABASE containing the path to the database asset folder, and
DA__P7__BLOCK__ASSET__REFERENCE_DOCS containing the path to the reference_docs asset folder.

In the block code, you can use values of those environment variables to obtain paths to asset files. For example, to get the path to the material.db file from the database asset:

import os

# path to the asset folder containing database files
db_folder_path = os.environ['DA__P7__BLOCK__ASSET__DATABASE']
# path to the material.db file in that asset
db_file_path = os.path.join(db_folder_path, 'material.db')

Using assets to add Python modules¶

As noted in section Adding Python modules, the preferred way to add custom Python modules to your block is to install the module distribution to the block prototype using pip install. By installing modules to the block prototype, you avoid creating external code dependencies, which helps maintaining and updating the block. However if you need a module that cannot be installed with pip - for example, a module from a privately developed distribution - you can publish an asset containing that distribution and import modules from there.

For example, assume that your block requires modules from several private packages named private_sdk, private_lib, and so on. You can make them available to the block as follows:

Prepare an asset folder containing all required distributions. The folder name should contain only lowercase English letters, digits, and underscores - for example, private_python. In the example, the folder structure would be like the following:
- private_python/ - the asset folder
  - private_sdk/ - a package folder
  - private_lib/ - another package folder
  - ...other packages
Publish the private_python asset as described in Publishing and updating assets. The block environment variable corresponding to your private_python asset will be named DA__P7__BLOCK__ASSET__PRIVATE_PYTHON. Its value will be the path to the published asset folder containing your private Python distributions.
In your block startup scripts, add the path stored in the DA__P7__BLOCK__ASSET__PRIVATE_PYTHON environment variable to the Python module search path stored in the PYTHONPATH block environment variable:
1. Locate the block startup scripts. The paths to those scripts are specified by the edit.exec and run.exec parameters in the block's manifest (manifest.yaml). In the user block example provided by pSeven Enterprise, there are two such scripts: scripts/start.sh (for Linux) and scripts/start.bat (for Windows).
2. In startup scripts, find the commands that set the PYTHONPATH variable and append the value of DA__P7__BLOCK__ASSET__PRIVATE_PYTHON to PYTHONPATH. For example:
  start.shstart.bat
  # Set PYTHONPATH so that Python finds the block code. export PYTHONPATH="$DA__P7__BLOCK__CODE_DIR:$DA__P7__BLOCK__ASSET__PRIVATE_PYTHON"
  :: Set PYTHONPATH so that Python finds the block code. set PYTHONPATH="%DA__P7__BLOCK__CODE_DIR%;%DA__P7__BLOCK__ASSET__PRIVATE_PYTHON%"
You can now import modules from the private_python asset as usual:
```
import private_lib
from private_sdk import utils
```
The block will search for modules and packages in the directories listed by the PYTHONPATH variable and find the private_lib and private_sdk packages in the published private_python asset.

Testing and debugging a custom block¶

Typically, some debug workflow is used to test a block during its development. Open the debug workflow and follow these steps to load your block into the workflow library:

Upload the block prototype to your pSeven Enterprise home directory: drag the .p7protoblock folder containing the block code from your local folder to the Explorer pane in pSeven Enterprise. Wait for the upload to complete. Please note that you should upload the folder itself, not a ZIP archive containing that folder.
Load the block prototype into the library: in the Explorer pane, select the prototype folder you have uploaded, and drag it onto the Workflow library pane.
Optionally, delete the block prototype folder from your pSeven Enterprise home directory. That copy is no longer needed as the block prototype is now stored in the workflow library, and the workflow will load it from the library as required.

As a result of these steps, your block will appear in the library, ready to be added to the workflow.

Important

To load a block prototype into the library, you must have the app_auth | user | Can develop blocks permission; otherwise, you cannot drag the prototype folder from your home directory onto the Workflow library pane. By default, each pSeven Enterprise user has this permission, but the administrator can remove it for individual users. Such users will not be able to develop blocks.

You can load a new block prototype without changing the block ID and version in the manifest. This will replace the block code in the library and in the workflow with the block code from the prototype folder. This feature allows you to quickly test changes in the block code.

Important

When testing and debugging a block on a Windows extension node, you must add the DEV tag to the block's manifest; otherwise, your changes in the block code might conflict with the earlier state of the block stored in the node's local cache. For details on the DEV tag, refer to section Tagging the block under development.

If you load a new block prototype with the same ID and an increased version number, the new block appears in the workflow library along with the old one. This new block version does not automatically replace the old one in the workflow. To replace the block in the workflow, you need to click the update notification next to the old block in the Workflow library pane. This allows you to test the steps that the users of the block will need to follow after a new block version is published to the common library.

Observe the following basic rules when testing and debugging a custom block:

Change the block ID only if you intend to design a new block rather than a new version of an existing block. For a new version of a block, leave the existing block ID unchanged.
While testing a new version of a block, keep the same version number in the block's manifest: do not change the version parameter in the manifest.yaml file before you load the block into the workflow library. When loaded with the same ID and version number, the updated block automatically replaces the existing one in the library and in the workflow.
Use the DEV tag when testing the new version of the block on a Windows extension node (see Tagging the block under development).
Do not manually delete the old block from the workflow library, as this will remove the block from the workflow. After you upgrade the block in the library, it will be automatically upgraded in the workflow.

Publishing a custom block¶

A pSeven Enterprise deployment administrator who has access to its file storages can publish a custom block to the common library. To do this, the administrator needs to copy the block prototype folder (.p7protoblock) to the protoblocks folder in the shared data storage and set permissions so pSeven Enterprise can read files in the copied folder.

How to locate the shared data storage

The shared data storage location is specified by the storage.shareddata.* parameters in the deployment configuration file values.yaml (see the pSeven Enterprise deployment guide).

Connect to the shared data storage server.
Copy the block prototype folder (.p7protoblock) to the <shareddata>/protoblocks folder on the server, where <shareddata> stands for the location of the shared data storage.
pSeven Enterprise connects to the storage as a system user with UID 11111. That system user needs read access to all files in the protoblocks folder. There are several ways to set the required permissions, commonly one of the following:
- If you connect to the storage as a user with root permissions, you can give ownership of the protoblocks folder to UID 11111 and set read-only access permissions, for example:
```
# chown -R 11111:11111 protoblocks/ && chmod -R 500 protoblocks/
```
  This way, only pSeven Enterprise will have access to block files, and this access is read-only.
- If you connect to the storage as a regular user, you can keep ownership of the protoblocks folder but allow read-only access for everyone, for example:
```
$ chmod -R 755 protoblocks/
```
  This way any user, including pSeven Enterprise, can read block files.

After you have copied the block to the protoblocks folder and set read permissions, the block should appear in the Common library pane in pSeven Enterprise, from where users can add it to their workflows.

Publishing block updates¶

When a new version of the block is ready to publish, a pSeven Enterprise deployment administrator can upgrade the block by copying the .p7protoblock folder of the new version to the protoblocks folder in the shared data storage. The block prototype of the new version should be prepared and packaged as follows:

Verify the version parameter value in the block's manifest. When you publish a new version of your block, increment the version parameter value from the previous version.
Test the new version of the block for backward compatibility to ensure that it works correctly with the configurations saved by any earlier version of that block.
Do not change the block ID. Otherwise, the new version will be considered a new block, different from the one you intend to update.
Remove the DEV tag from the tags parameter in the block's manifest (see Tagging the block under development for details).
Pack the new block version into a folder with a name that contains the new version number, for example <block-name>-<ID>-<new-version-number>.p7protoblock.

The administrator can now copy the folder with the new block version to the protoblocks folder in the shared data storage. It is not necessary to delete the .p7protoblock folder with the old version of the block. The Common library pane displays the block with the highest version number, even if the protoblocks folder holds other versions of that block.

In case of a version conflict, when two folders with the same ID and version number in the block manifest are copied to the protoblocks folder, the Common library pane does not display this block, which signals a block update failure. After publishing a block, it is advisable to make sure that the block is displayed correctly in the Common library pane.

Upgrading the block in the workflow¶

After the publication of a new block version, users can upgrade the block in their workflows. The Workflow library pane displays the new block version along with the old one, and proposes to upgrade the block in the workflow. To upgrade the block, click the update notification next to the old version of the block in the Workflow library pane.

The workflow uses the old block version until you upgrade it. The upgrade replaces the block with the new version in the workflow, and removes the old block version from the workflow library. If needed, you can undo these changes, having the workflow revert to the old block version.

You should not delete the old block version from the workflow library before the upgrade as you cannot upgrade the block if it is deleted. However, if a block was deleted by accident or by mistake, you can undo the deletion. Note that the old block version is automatically removed from the workflow library after you upgrade the block.