Extension node deployment

This guide covers the initial deployment of an extension node - additional pSeven Enterprise task host intended to run Windows software. Extension nodes can only be added after the pSeven Enterprise deployment has been completed.

Related guides

• pSeven Enterprise deployment: initial pSeven Enterprise deployment.
• Upgrading extension nodes: installing updates with new pSeven Enterprise versions.
• Administration: maintenance and additional configuration.

About extension nodes

pSeven Enterprise is designed for deployment on a Kubernetes cluster, but it can also execute tasks using extension nodes - additional Windows hosts, which are not Kubernetes nodes. Each extension node runs certain services to connect to pSeven Enterprise, and provides a task runtime environment. With extension nodes, pSeven Enterprise can run workflows that employ platform-dependent or host-locked software.

The extension node setup package prepares a Windows node and connects it to pSeven Enterprise. In particular, it installs:

• The OpenVPN client, needed to connect to pSeven Enterprise services running on the Kubernetes cluster.
• The HTCondor service, bringing the node under the control of the pSeven Enterprise task manager.
• Local runtime environment (Python).

The installer was tested on Windows 10 only. If you need support for another Windows version, please contact Technical support.

Before installation

pSeven Enterprise must be deployed before you add extension nodes. The extension node setup package is generated during deployment and stores certain settings specific to this deployment.

Extension nodes establish VPN connections to the Kubernetes cluster using the port specified by the entrypoint.gateOpenvpn parameter in the pSeven Enterprise deployment configuration file (values.yaml). By default, this is port 31194 (see Network parameters). Verify that your network configuration allows inbound and outbound cluster connections over TCP and UDP using that port.

Windows Defender has to be disabled on the node. The reason is that pSeven Enterprise workflow tasks running on the node frequently access files located on network shares (mounted user home directories), and Windows Defender's network security scan policies interfere with that access, causing workflow errors.

A recent version of Windows 10 (64-bit edition) is required. pSeven Enterprise was tested with extension nodes running Windows 10 Pro 64-bit, version 1909. Other Windows 10 builds should be compatible but were not actively tested.

Hardware requirements are determined by tasks and software intended to run on the node. Commonly: 8 CPU, 32 GB RAM, 1 TB disk space. At least 10 GB free disk space must be available to pSeven Enterprise system services.

Installing on Windows Server

It is possible to install the extension node on a computer running Windows Server 2019 or a later version of Windows Server, provided that the computer is a standalone server, that is, not an Active Directory domain controller. Please note that extension nodes have not been tested on computers running Windows Server and may not work properly on such computers.

Setup package

To get the extension node setup package:

• Sign in to pSeven Enterprise as administrator.

• Download:

  {sign-in URL}/pseven/@files/Users/.p7/pSevenExtensionNode.zip
  

  The {sign-in URL} is the pSeven Enterprise sign-in URL, which you get during its deployment.

The package you download is generated by pSeven Enterprise deployment and contains settings specific to this deployment. Do not use a setup package obtained elsewhere.

Install

Copy the downloaded setup package to the extension node host and follow the guidelines below.

Installation steps

• Unpack pSevenExtensionNode.zip to a temporary directory. The path to this directory must not contain spaces, for example: C:\Temp\pSevenExtensionSetup.
• Disable Windows Defender real-time protection. This is required so the installer can automatically disable Windows Defender during setup. Open Settings and select: Update & Security > Windows Security > Virus & threat protection > Manage settings. Switch Real-time protection to "Off".
• Open a command prompt as administrator and change to the directory with the unpacked installer.

• Run bootstrap.bat {release dir} {temporary dir} {pseven address}. The arguments are:

  • {release dir} - the directory where to install services and pSeven Enterprise runtime environment.
  • {temporary dir} - the directory where pSeven Enterprise will store task data and temporary files.

  • {pseven address} - the IP address or hostname from the pSeven Enterprise sign-in URL, which you get during its deployment.

    Note if there is a load balancer in your deployment

    If there is a load balancer in your deployment, specify the IP address or hostname of any worker node as {pseven address}, rather than the IP address or hostname from the sign-in URL. You get the list of worker node addresses during pSeven Enterprise deployment.

    Example:

    bootstrap.bat C:\pSevenExtension C:\pSevenExtensionTemp worker.node
    

  Note that the release and temporary directory paths must not contain spaces. The release directory must be accessible to all users (read and execute). The temporary directory must allow full access for all users. The release directory must not be in the user profile folder %UserProfile% (typically C:\Users\{user_name}), otherwise the installation will fail.

• After bootstrap.bat completes, verify that there are no error messages in its output. If you find any error messages there, save the output to a file for later contacting Technical support. If bootstrap.bat completed without errors, check the node as described in Node check.

  Save the node's master password

  Make sure you save the master password, output by bootstrap.bat - you will not be able to restore or reset it. That password is required for user block development and debugging purposes.

Setup installs several software packages and creates a number of local Windows user accounts, which takes some time. Local accounts match pSeven Enterprise user accounts and are used for isolation of user tasks on the extension node. For more details on system changes done by the installer, see section System changes.

After verifying the installation, you can delete the downloaded package and its unpacked contents (C:\Temp\pSevenExtensionSetup in examples above): they are no longer required.

Workflow blocks running on the extension node may need to run third-party engineering software (CAD/CAE packages) that requires certain environment variables. Add those variables to the block runtime as described in Engineering software on extension nodes. Otherwise, workflow blocks will not be able to run the software packages they need.

Node check

Checks in this section are mandatory after installing the node.

1. Verify the OpenVPN installation. At a command prompt, run ipconfig and check its output: the output list should contain a network adapter with the DNS suffix of svc.cluster.local. This adapter is used by OpenVPN connections, and normally has a generic name, for example:

   Unknown adapter Local area connection:
   
      Connection-specific DNS Suffix  . : svc.cluster.local
      Link-local IPv6 Address . . . . . : fe80::9524:65bc:3853:a3fc%2
      IPv4 Address. . . . . . . . . . . : 192.168.253.10
      Subnet Mask . . . . . . . . . . . : 255.255.255.252
      Default Gateway . . . . . . . . . : 192.168.253.9
   

2. Verify that the following services are up and running:

   • LanmanWorkstation - native Windows network (SMB) client, required.
   • Condor - HTCondor task management service, required.
   • OpenVPNService - OpenVPN connection service, required.
   • OpenVPNServiceInteractive - OpenVPN configuration GUI, optional.

   You can run sc query {service name} at a command prompt to check the service's state.

3. Verify the Python installation. At a command prompt, run the following commands one by one, and verify that there are no errors in their output:

   call %DA__P7__SIDECAR__ACTIVATE%
   %DA__P7__APP__SIDECAR_PYTHON_EXECUTABLE% -c "import paramiko"
   call %DA__P7__SIDECAR__DEACTIVATE%
   call %DA__P7__RUNENV__PYTHON3__ACTIVATE%
   %DA__P7__APP__SIDECAR_PYTHON_EXECUTABLE% -c "import paramiko"
   call conda deactivate
   

4. Verify that the node has registered itself with pSeven Enterprise, so that users can specify it as the host to run a task (workflow block). The registration is automatic, and normally it should complete while you perform other checks.

   • Sign in to pSeven Enterprise as a user.
   • Create a new workflow.
   • Add any block to the workflow.
   • Select this block so the right pane switches to Block properties.
   • At the top of the Block properties pane, next to the block's name, expand the "Run on:" drop-down list and verify that it contains the node's hostname.

If any checks fail, reboot the node and conduct the checks again. If any checks fail after rebooting, contact Technical support.

System changes

This section provides a summary of changes, which the node setup scripts (bootstrap.bat and other scripts it calls) apply to the system:

• Install Microsoft Visual C++ 2015-2019 Redistributable Package v14.25.28508 with default settings.
• Install Python environment: Miniconda 2, Miniconda 3, and separate Python 3 to the {release dir} you have specified; the path of this directory is stored to the DA__P7__RELEASE_DIR environment variable.
• Install OpenVPN and HTCondor services to the {release dir} and configure them to start automatically.
• Disable Windows Defender permanently to stop it interfering with workflow execution. Unfortunately, there is no other known workaround for this issue at the moment.
• Disable caching features of the SMB client, which are also known to interfere with pSeven Enterprise workflows (see the samba-config.ps1 script for details).
• Increase the timeout for the service control manager to 120 seconds (default is 30). This timeout is stored in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control registry key (the ServicesPipeTimeout value). The increased timeout is required to allow OpenVPN and HTCondor services to start properly.
• Create a number of local user accounts, which are used to run user tasks. For each pSeven Enterprise user ID, the matching local user account is created with the same ID. The number of accounts is equal to the number of users specified during pSeven Enterprise deployment (the total number of supported users).
• Add a number of environment variables. Names of pSeven Enterprise environment variables begin with DA__P7__. See Appendix A for a full list of those variables.

Notes and tips

Main installation script is bootstrap.bat. It prepares variables and sequentially starts the following scripts:

• bootstrap-system.bat - installs C++ runtimes, configures SMB client, disables Windows Defender.
• bootstrap-python.bat - installs the Python environment.
• bootstrap-openvpn.bat - installs and configures OpenVPN.
• bootstrap-condor.bat - installs and configures HTCondor.

All bootstrap scripts are designed to run independently if the DA__P7__RELEASE_DIR and DA__P7__TMP_DIR variables are set. You can use this feature to run them manually. The bootstrap-openvpn.bat also uses the DA__P7__PSEVEN_ADDRESS variable if it is set.

When you run bootstrap.bat with arguments, it sets the above variables from the arguments, and will also read these variables if you re-run it.

If you did not specify {pseven address} to bootstrap.bat, you can open %DA__P7__RELEASE_DIR%\OpenVPN\config\winnode.ovpn with a text editor and replace the DA__P7__PSEVEN_HOST placeholder there with the address from the pSeven Enterprise sign-in URL.

Known issues

There is a known issue where a workflow run stops with the error:

OSError [Errno 22] invalid argument

If you encounter this error, recheck that Windows Defender is disabled on the node. The error is caused by virus protection which locks files created by pSeven Enterprise. Note that the installer normally disables Windows Defender via node's local settings, and these changes can be reversed, for example, by Windows or a domain controller.

One more known issue is that some software running on an extension node may require an interactive session. For example, Excel cannot work with documents containing macros, if it runs in a non-interactive session. See Appendix B for guidelines on how to provide an interactive session. More details on Excel setup see in Appendix C.

There are no more known issues at the moment. If you encounter any other issue with extension nodes, contact Technical support.

Appendix A: Environment variables

The extension node setup creates a number of environment variables, listed below. These variables store pSeven Enterprise settings and are required for it to work. Their names begin with DA__P7__.

• DA__P7__RELEASE_DIR: Path to the release (installation) directory.
• DA__P7__TMP_DIR: Path to the pSeven Enterprise temporary files directory.
• DA__P7__PSEVEN_ADDRESS: pSeven Enterprise address.
• DA__P7__BLOCKFILESGATE_HOST: Hostname of the pSeven Enterprise service, which is used to mount user directories.
• DA__P7__BLOCKSSHGATE_HOST: Hostname of the pSeven Enterprise service, which handles the extension node connection to pSeven Enterprise.
• DA__P7__BLOCKSSHGATE_PORT: The extension node connection service port.
• DA__P7__BLOCKSSHGATE_USER: The system username. Extension nodes connect to pSeven Enterprise as this user.
• DA__P7__BLOCKSSHGATE_CLIENT_KEY_FILE: Path to the extension node public SSH key certificate file.
• DA__P7__HTCONDORMANAGER_HOST: Hostname of the HTCondor service to which the node connects.
• DA__P7__HTCONDORMANAGER_PORT: HTCondor service connection port.
• DA__P7__APP__SIDECAR_PYTHON_EXECUTABLE: Path to the Python interpreter. pSeven modules running on the node use this version of Python.
• DA__P7__RUNENV__PYTHON3__ACTIVATE: Path to the Python 3 runtime environment activation script.
• DA__P7__RUNENV__PYTHON3__VERSION: Python 3 runtime environment version.
• DA__P7__CONDOR_RELEASE_DIR: Path to the HTCondor installation directory.
• DA__P7__CONDOR_LOCAL_DIR: Path to the HTCondor temporary files directory.
• DA__P7__HTCONDOR_MEMORY_QUANTUM: The HTCondor memory quantum.
• DA__P7__CLUSTER__TASK_HOMES_ROOT: Path to the storage directory that contains pSeven Enterprise user home directories.
• DA__P7__CLUSTER__TASK_HOMES_DRIVE: Drive where pSeven Enterprise mounts user home directories when running user tasks.
• DA__P7__REDIS_HOST: Hostname of the system Redis database service.
• DA__P7__REDIS_PORT: Redis database service connection port.
• DA__P7__RESOURCE_MANAGEMENT: A flag indicating whether resource management features are enabled in your pSeven Enterprise deployment.

Appendix B: Interactive session setup

Sometimes the software running on an extension node requires an interactive Windows session - a user signed in to the desktop. In particular, an interactive session is required by Microsoft Excel to enable pSeven Enterprise blocks running on the node to connect to Excel.

In such cases, you must ensure that the node provides an interactive session. Set up a user account that will be used solely to create an interactive session. If the node is rebooted, that account must be signed in after the reboot. If normal sign-in is not an option, the node can be configured to sign in that account automatically, thereby creating an interactive session each time the node is started. For configuration instructions, see Microsoft's article Turn on automatic logon in Windows.

Appendix C: Enabling Excel macros

Microsoft Office tools, and Excel in particular, are not designed to run on the server side. The reasons are described in this Microsoft article: Considerations for server-side Automation of Office.

Due to this, additional setup is required to enable Excel macros on a pSeven Enterprise extension node.

1. Enable macros in Excel and programmatic access to macros:
   • In Excel, open Options > Trust Center > Trust Center Settings...
   • In Macro Settings, select "Enable all macros without notification".
   • Also enable the checkbox labeled "Trust access to the VBA project object model".
   • Apply settings, close Excel.
2. Configure DCOM:
   • Open the Component Services console as administrator:
     • For 64-bit Office, run mmc comexp.msc.
     • For 32-bit Office, run mmc comexp.msc /32.
   • Open Component Services > Computers > My Computer > DCOM Config.
   • Right-click "Microsoft Excel Application" and open its properties.
   • In the properties dialog, switch to the Identity tab and select "The interactive user".
   • In the properties dialog, switch back to the General tab and set the authentication level to None.
   • Click OK in the dialog, close the Component Services console.