Extension node deployment

This guide describes the initial deployment of an extension node - additional pSeven Enterprise task host, which runs Windows software. Extension nodes can be set up only after pSeven Enterprise has been deployed.

Related guides

• pSeven Enterprise deployment: initial pSeven Enterprise deployment.
• Upgrading extension nodes: installing updates with new pSeven Enterprise versions.
• Administration: maintenance and additional configuration.

About extension nodes

pSeven Enterprise is designed for deployment on a Kubernetes cluster, but it can also execute tasks using extension nodes - additional Windows hosts, which are not Kubernetes nodes. An extension node runs several pSeven connectivity services and provides the task runtime environment. Thus pSeven workflows can run platform-dependent or node-locked software tools.

The extension node setup package prepares a Windows node and connects it to pSeven. In particular, it installs:

• The OpenVPN client, needed to connect to pSeven services running on the Kubernetes cluster.
• The HTCondor service, bringing the node under control of the pSeven task manager.
• Local runtime environment (Python).

The installer was tested on Windows 10 only. If you need support for another Windows version, please contact the DATADVANCE support team at support@pseven.io.

Before installation

pSeven Enterprise must be deployed before you add extension nodes. The extension node setup package is generated during deployment and stores certain settings specific to this deployment.

Extension nodes establish VPN connections to the Kubernetes cluster using the port specified by the entrypoint.gateOpenvpn parameter in the pSeven Enterprise deployment configuration file (values.yaml). By default, this is port 31194 (see Network parameters). Verify that your network configuration allows inbound and outbound cluster connections over TCP and UDP using that port.

Windows Defender has to be disabled on the node. The reason is that pSeven tasks running on the node frequently access files located on network shares (mounted user home directories), and Windows Defender's network security scan policies interfere with this access, leading to workflow errors.

A recent version of Windows 10 (64-bit edition) is required. pSeven was tested with extension nodes running Windows 10 Pro, version 1909. Other Windows 10 builds should be compatible but were not actively tested.

Hardware requirements are determined by tasks and software tools you intend to run on the node. In addition to them, pSeven requires at least 10 GB free disk space. 8 GB or more RAM is also recommended.

Installing on Windows Server

It is possible to install the extension node on a computer running Windows Server 2019 or a later version of Windows Server, provided that the computer is a standalone server, that is, not an Active Directory domain controller. Please note that extension nodes have not been tested on computers running Windows Server and may not work properly on such computers.

Setup package

To get the extension node setup package:

• Sign in to pSeven Enterprise as administrator.

• Download:

  {sign-in URL}/pseven/@files/Users/.p7/pSevenExtensionNode.zip
  

  The {sign-in URL} is the pSeven Enterprise sign-in URL, which you get during its deployment.

The package you download is generated by pSeven Enterprise deployment and contains settings specific to this deployment. Do not use a setup package obtained elsewhere.

Install

Copy the downloaded setup package to the extension node host and follow the guide below.

Installation steps

• Unpack pSevenExtensionNode.zip to a temporary directory. The path to this directory must not contain spaces, for example: C:\Temp\pSevenExtensionSetup.
• Disable Windows Defender real-time protection. This is required so the installer can automatically disable Windows Defender during setup. Open Settings and select: Update & Security > Windows Security > Virus & threat protection > Manage settings. Switch Real-time protection to "Off".
• Open a command prompt as administrator and change to the directory with the unpacked installer.

• Run bootstrap.bat {release dir} {temporary dir} {pseven address}. The arguments are:

  • {release dir} - the directory where to install services and pSeven runtime environment.
  • {temporary dir} - the directory where pSeven will store task data and temporary files.

  • {pseven address} - the IP address or hostname from the pSeven Enterprise sign-in URL, which you get during its deployment.

    Note if there is a load balancer in your deployment

    If there is a load balancer in your deployment, specify the IP address or hostname of any Kubernetes node as {pseven address}, instead of the address or hostname from the sign-in URL.

    Example:

    bootstrap.bat C:\pSevenExtension C:\pSevenExtensionTemp pseven.node
    

  Note that the release and temporary directory paths must not contain spaces. The release directory must be accessible to all users (read and execute). The temporary directory must allow full access for all users. The release directory must not be in the user profile folder %UserProfile% (typically C:\Users\{user_name}), otherwise the installation will fail.

• When bootstrap.bat finishes, check its output for errors. If there are none, reboot. If you see errors, save the script's output from the command prompt and contact the DATADVANCE support team (see Technical support).

• After the reboot, check the node as described in section Node check.

Setup installs several software packages and creates a number of local Windows user accounts, which takes some time. Local accounts correspond to pSeven user accounts and are used to isolate user tasks running on the node. For more details on system changes done by the installer, see section System changes.

After verifying the installation, you can delete the downloaded package and its unpacked contents (C:\Temp\pSevenExtensionSetup in examples above): they are no longer required.

Workflow blocks running on the extension node may need to run third-party engineering software (CAD/CAE packages) that requires certain environment variables. Add those variables to the block runtime as described in Engineering software on extension nodes. Otherwise, workflow blocks will not be able to launch the software packages they need.

Node check

Checks in this section are mandatory after installation. Also note that a reboot is required before you proceed to check the node.

1. Verify the OpenVPN installation. Run ipconfig in a command prompt and check its output: it should list a network adapter with the DNS suffix svc.cluster.local. This adapter is added by OpenVPN and usually has a generic name, for example:

   Unknown adapter Local area connection:
   
      Connection-specific DNS Suffix  . : svc.cluster.local
      Link-local IPv6 Address . . . . . : fe80::9524:65bc:3853:a3fc%2
      IPv4 Address. . . . . . . . . . . : 192.168.253.10
      Subnet Mask . . . . . . . . . . . : 255.255.255.252
      Default Gateway . . . . . . . . . : 192.168.253.9
   

2. Verify that the following services exist and are running after the previous reboot:

   • LanmanWorkstation - native Windows network (SMB) client, required.
   • Condor - HTCondor task management service, required.
   • OpenVPNService - OpenVPN connection service, required.
   • OpenVPNServiceInteractive - OpenVPN configuration GUI, optional.

   You can run sc query {service name} in a command prompt to check the service's state.

3. Verify the Python installation. In a command prompt, execute the following commands one by one and make sure there are no errors in their output:

   call %DA__P7__SIDECAR__ACTIVATE%
   %DA__P7__APP__SIDECAR_PYTHON_EXECUTABLE% -c "import paramiko"
   call %DA__P7__SIDECAR__DEACTIVATE%
   call %DA__P7__RUNENV__PYTHON3__ACTIVATE%
   %DA__P7__APP__SIDECAR_PYTHON_EXECUTABLE% -c "import paramiko"
   call conda deactivate
   

4. Verify that the node has registered itself with pSeven, so that users can specify it as the host to run a selected task (a workflow block). The registration is automatic and usually takes 1-2 minutes after the node reboot, so it should finish while you perform other checks.

   • Sign in to pSeven as a user.
   • Create a new workflow.
   • Add any block to the workflow.
   • Select this block so the right pane switches to Block properties.
   • On top of the Block properties pane, next to the block's name, there is a drop-down list labeled "Run on:". Open it and verify that the node's hostname is found in the list.

If any of the checks 1-4 fails, contact the DATADVANCE support team (see Technical support).

System changes

This section provides a summary of changes, which the node setup scripts (bootstrap.bat and other scripts it calls) apply to the system:

• Install Microsoft Visual C++ 2015-2019 Redistributable Package v14.25.28508 with default settings.
• Install Python environment: Miniconda 2, Miniconda 3, and separate Python 3 to the {release dir} you have specified; the path of this directory is stored to the DA__P7__RELEASE_DIR environment variable.
• Install OpenVPN and HTCondor services to the {release dir} and configure them to start automatically.
• Disable Windows Defender permanently to stop it interfering with workflow execution. Unfortunately there is no other known workaround for this issue at the moment.
• Disable caching features of the SMB client, which are also known to interfere with pSeven workflows (see the samba-config.ps1 script for details).
• Increase the timeout for the service control manager to 120 seconds (default is 30). This timeout is stored in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control registry key (the ServicesPipeTimeout value). The increased timeout is required to allow OpenVPN and HTCondor services to start properly.
• Create a number of local user accounts, which are used to run tasks from pSeven users in isolated environments. For each pSeven user id, a corresponding local Windows user account is created with the same id. The number of accounts equals the number of users specified during pSeven Enterprise deployment (the total number of supported users).
• Add several environment variables. Names of all pSeven environment variables begin with DA__P7__. See Appendix A for the full list.

Notes and tips

Main installation script is bootstrap.bat. It prepares variables and sequentially starts the following scripts:

• bootstrap-system.bat - installs C++ runtimes, configures SMB client, disables Windows Defender.
• bootstrap-python.bat - installs the Python environment.
• bootstrap-openvpn.bat - installs and configures OpenVPN.
• bootstrap-condor.bat - installs and configures HTCondor.

All bootstrap scripts are designed to run independently if the DA__P7__RELEASE_DIR and DA__P7__TMP_DIR variables are set. You can use this feature to run them manually. The bootstrap-openvpn.bat also uses the DA__P7__PSEVEN_ADDRESS variable if it is set.

When you run bootstrap.bat with arguments, it sets the above variables from the arguments, and will also read these variables if you re-run it.

If you did not specify {pseven address} to bootstrap.bat, you can open %DA__P7__RELEASE_DIR%\OpenVPN\config\winnode.ovpn with a text editor and replace the DA__P7__PSEVEN_HOST placeholder there with the address from the pSeven Enterprise sign-in URL.

Known issues

There is a known issue where a workflow run stops with the error:

OSError [Errno 22] invalid argument

If you encounter this error, recheck that Windows Defender is disabled on the node. The error is caused by virus protection which locks files created by pSeven. Note that the installer tries to disable Windows Defender via node's local settings, and these changes may get reverted by Windows or a domain controller, for example.

Another known issue is that some software running on an extension node may require an interactive user. For example, Excel cannot work with documents containing macros, if it runs in a non-interactive session. If this is the case, see Appendix B for a guide on adding an interactive user. For more details on Excel setup specifically see Appendix C.

There are no more known issues at the moment. If you encounter any other error, contact the DATADVANCE support team (see Technical support).

Appendix A: pSeven environment variables

Extension node setup creates a number of environment variables, listed below. These variables store pSeven settings and are required to work. Names of all pSeven environment variables begin with DA__P7__.

• DA__P7__RELEASE_DIR: Path to the release (installation) directory.
• DA__P7__TMP_DIR: Path to the pSeven Enterprise temporary files directory.
• DA__P7__PSEVEN_ADDRESS: pSeven Enterprise address.
• DA__P7__BLOCKFILESGATE_HOST: Hostname of the pSeven Enterprise service, which is used to mount user directories.
• DA__P7__BLOCKSSHGATE_HOST: Hostname of the pSeven Enterprise service, which handles the extension node connection to pSeven Enterprise.
• DA__P7__BLOCKSSHGATE_PORT: The extension node connection service port.
• DA__P7__BLOCKSSHGATE_USER: The system username. Extension nodes connect to pSeven Enterprise as this user.
• DA__P7__BLOCKSSHGATE_CLIENT_KEY_FILE: Path to the extension node public SSH key certificate file.
• DA__P7__HTCONDORMANAGER_HOST: Hostname of the HTCondor service to which the node connects.
• DA__P7__HTCONDORMANAGER_PORT: HTCondor service connection port.
• DA__P7__APP__SIDECAR_PYTHON_EXECUTABLE: Path to the Python interpreter. pSeven modules running on the node use this version of Python.
• DA__P7__RUNENV__PYTHON3__ACTIVATE: Path to the Python 3 runtime environment activation script.
• DA__P7__RUNENV__PYTHON3__VERSION: Python 3 runtime environment version.
• DA__P7__CONDOR_RELEASE_DIR: Path to the HTCondor installation directory.
• DA__P7__CONDOR_LOCAL_DIR: Path to the HTCondor temporary files directory.
• DA__P7__HTCONDOR_MEMORY_QUANTUM: The HTCondor memory quantum.
• DA__P7__APP__USER_HOME_MOUNT_DIR: Path to the directory where pSeven mounts user home directories when running user tasks.
• DA__P7__APP__APPDATA_MOUNT_DIR: Path to the directory where pSeven mounts its data storage.
• DA__P7__CLUSTER__TASK_HOMES_ROOT: Path to the storage directory that contains pSeven user home directories.
• DA__P7__CLUSTER__TASK_HOMES_DRIVE: Drive where pSeven mounts user home directories when running user tasks.
• DA__P7__REDIS_HOST: Hostname of the system Redis database service.
• DA__P7__REDIS_PORT: Redis database service connection port.
• DA__P7__RESOURCE_MANAGEMENT: A flag indicating whether resource management features are enabled in your pSeven Enterprise deployment.

Appendix B: Setup interactive user

Sometimes the software running on an extension node requires an interactive Windows session - a user logged on to the desktop. In particular, an interactive session is required by Microsoft Excel to enable pSeven Enterprise blocks running on the node to connect to Excel.

In such cases, ensure that the node provides a persistent interactive session, according to that node usage conditions and the IT policies of your organization. It is advisable to create a separate (service) user account that shall provide the interactive session, and reserve it solely for that purpose.

In certain Windows editions and versions you can also set up autologon for that account, following the example below.

Add a new user:

1. Open the Local Users and Groups console (lusrmgr.msc).
2. Create a new user with any name. When creating, uncheck "User must change password at next logon" and check "Password never expires" for this user.
3. Add the user to the "Remote Desktop Users" group.

Set up autologon:

1. Open the Run dialog (Win+R) and run netplwiz to open the User Accounts dialog.

2. On the Users tab in that dialog, select the user you have just created and clear the checkbox labeled "Users must enter a user name and password to use this computer." above the list of users.

   If there is no such checkbox in the User Accounts dialog, then the Windows version you are running normally does not support autologon.

3. Click Apply in the User Accounts dialog. In the Automatically sign in dialog that appears, verify that the User name field specifies the new user you have recently created. Input the password and click OK.

4. Click OK in the User Accounts dialog to close it.

To verify that setup, reboot the extension node. Windows should boot straight to the new user's desktop without showing the logon screen.

Appendix C: Enable Excel macros

Microsoft Office tools, and Excel in particular, are not designed to run on the server side. The reasons are described in this Microsoft article: Considerations for server-side Automation of Office.

Due to this, additional setup is required to enable Excel macros on a pSeven extension node.

1. Enable macros in Excel and programmatic access to macros:
   • In Excel, open Options > Trust Center > Trust Center Settings...
   • In Macro Settings, select "Enable all macros without notification".
   • Also enable the checkbox labeled "Trust access to the VBA project object model".
   • Apply settings, close Excel.
2. Configure DCOM:
   • Open the Component Services console as administrator:
     • For 64-bit Office, run mmc comexp.msc.
     • For 32-bit Office, run mmc comexp.msc /32.
   • Open Component Services > Computers > My Computer > DCOM Config.
   • Right-click "Microsoft Excel Application" and open its properties.
   • In the properties dialog, switch to the Identity tab and select "The interactive user".
   • In the properties dialog, switch back to the General tab and set the authentication level to None.
   • Click OK in the dialog, close the Component Services console.