This article covers the requirements to setup a remote server for censhare.

Note: The content of this article is internal!

Setting up a remote server takes quite some time but is important in order to carry out latency and performance testing. Moreover, this is the common setup that most of our customers are using.

Prerequisites

To set up a remote server you need the following:

  • an already running server (e.g. on your localhost) with an already setup (local) database - this will be the Master server;
  • an individual development CentOS remote server (linux), setup by IT but without any pre-installed censhare Server - see this ecosphere article for more information - the DNS name of the remote server will be something like csdev-remote-abc - where abc is your short name;
  • optional: If you plan to do latency testing, request a WANEM server from IT - see this ecosphere article for more information about WANEM.

Basic setup of the remote server machine

First, login to your remote server with root access and prepare the system. It is a good practice not to work as root, but as a local user with restricted rights:

Create a new user account and add it to the user-group wheel to get access to sudo. Preferably, use your company short name as username (e.g. abc) to make it easier transferring data from your local to the remote machine.

The following command creates the user account abc in group wheel with the home directory /home/abc. You will be asked for a new password and logged in with the newly created user:

LOCAL-DEV $ ssh root@csdev-remote-abc
Last login: Tue Nov 21 06:53:47 2017
[root@csdev-remote-abc ~]#
[root@csdev-remote-abc ~]# useradd -b/home -m -Gwheel abc
[root@csdev-remote-abc ~]# passwd abc
[root@csdev-remote-abc ~]# su - abc
[abc@csdev-remote-abc ~]$

It might be helpful to align the directory structure with the common setup by creating symlinks:

[abc@csdev-remote-abc ~]$ ln -s /home/abc/censahre-product/censhare-Server /home/abc/css
[abc@csdev-remote-abc ~]$ ln -s /home/abc/censahre-product/censhare-Custom/censhare-Server /home/abc/cscs
[abc@csdev-remote-abc ~]$ ln -s /home/abc/censahre-product/css/work /home/abc/work

To be able to access git, you need to use your SSH key on the remote server (there is no anonymous access available yet). The most simple and secure way to do that is to use ssh-agent, especially when using OS X in which ssh-agent is automatically available since Snow Leopard-(2009). Its usage still has to be enabled in the ~/.ssh/config file:

Host csdev*
    ForwardAgent yes

Note: It is possible to enable forwarding for all servers (by using just Host *), but this is not a recommended practice.

The keys themselves are not automatically added by default; to add a default key, run the following command (on your local machine):

ssh-add

To check that the agent key forwarding works correctly, run the command below on the remote machine and it should display your push ssh key:

ssh-add -L

Log in to the remote machine, install all required software packages and set up the JAVA_HOME environment variable:

  • Git
    • Use package manager "yum".
    • Install via sudo yum -y install git
    • Validation of the installation: git --version
  • Node.js
    • Make the nodejs specific setup available to the yum repo
    • Use package manger yum to install nodejs
    • See further details at the nodejs web page Note: Part of the installation setup as shown below is executing a downloaded shell script as root. This is a security risk. As an alternative you might install a binary download from nodejs.org.
  • Yarn dependency manager
    • Use package manager yum to install yarn
    • See further details at the yarn web page
  • Java
    • Download Linux-64-bit RPM for JDK SE 11 from Oracle web page and copy it to your remote server (e.g. via scp)
    • Install via sudo rpm -i jdk-11.<VERSION>_linux-x64_bin.rpm - (make sure to check for the latest recommended version)
  • Optional Tools
    • sudo yum -y install lsof nmap - for easily checking network topics
    • exiftool from http://www.sno.phy.queensu.ca/~phil/exiftool/
      • ln -s /home/abc/exiftool/exiftool /opt/corpus/tools/exiftool/exiftool - to avoid a warning on server startup while checking image service ICC profiles
[abc@csdev-remote-abc ~]$ sudo yum -y install git
[abc@csdev-remote-abc ~]$ sudo yum -y install wget
[abc@csdev-remote-abc ~]$ sudo curl --silent --location https://rpm.nodesource.com/setup_<VERSION>.x | sudo bash -
[abc@csdev-remote-abc ~]$ sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo
[abc@csdev-remote-abc ~]$ sudo yum -y install nodejs
[abc@csdev-remote-abc ~]$ sudo yum -y install yarn
[abc@csdev-remote-abc ~]$ sudo rpm -i jdk-11.<VERSION>_linux-x64_bin.rpm
[abc@csdev-remote-abc ~]$ sudo echo "export JAVA_HOME=/usr/java/default" >> /etc/profile
[abc@csdev-remote-abc ~]$ export JAVA_HOME=/usr/java/default

Note: It is recommended to turn case sensitivity off for the git repository - this is the repository standard in censhare-product. Run git config --list to check for set configuration and git config --global core.ignorecase true to turn case sensitivity off:

[abc@csdev-remote-abc ~]$ git config --list
[abc@csdev-remote-abc ~]$ git config --global core.ignorecase true

To be able to access the remote censhare Server via HTTPs and via RMI (censhare Master server, Java client, Renderer client, etc.) you need to enable the specific ports on the firewall. Besides that it makes sense to open the remote-debugger port for Java:

  • Port 8001 : Java remote debugging
  • Port 9443 : censhare 5 web-server port
  • Port 1099 : Setup RMI
  • Port 30543 : corpus.RMIServer
  • Port 30546 : corpus.RMIServerSSL
  • Ports 10000 - 10010: YourKit profiler (optional)

Use the following command for the configuration:

[abc@csdev-remote-abc ~]$ sudo firewall-cmd --permanent --add-port=8001/tcp
[abc@csdev-remote-abc ~]$ sudo firewall-cmd --permanent --add-port=9443/tcp
[abc@csdev-remote-abc ~]$ sudo firewall-cmd --permanent --add-port=1099/tcp
[abc@csdev-remote-abc ~]$ sudo firewall-cmd --permanent --add-port=30543/tcp
[abc@csdev-remote-abc ~]$ sudo firewall-cmd --permanent --add-port=30546/tcp
[abc@csdev-remote-abc ~]$ sudo service firewalld restart
[abc@csdev-remote-abc ~]$ sudo firewall-cmd --list-all

To make the configuration for the remote and master servers easier and avoid possible conflicts in DNS resolutions, it is strongly recommended to create a hostname-entry within your local hosts file.

To do this you need the IP address of your master-server. If this is a dynamic IP address you either have to update your host-entry each time the IP changes or ask your IT for a static IP address:

[root@csdev-remote-abc ~]$ echo "<ip-of-the-master-srv> csdev-master-abc" >> /etc/hosts

Replace <ip-of-the-master-srv> with the IP address of your master server and csdev-master-abc with the hostname you want to use for your master server. This name is used within the configuration of the remote and master server.

Note: It might be helpful to use ifconfig <your-network-interface> to easily detect your local IP address.

Now you are able to clone censhare-product from develop to your remote machine and build your censhare-Server:

[abc@csdev-remote-abc ~]$ git clone https://git.censhare.com/censhare/censhare-product
[abc@csdev-remote-abc ~]$ cd censhare-product
[abc@csdev-remote-abc censhare-product]$ ./censhare-Server/bin/build.sh clean init jar

The initial setup is finished if you are able to compile the censhare Server. In the next step, we need to configure the master server as well as the remote server.

For comparison of the log files, both servers should have the same timezone and clock.

[abc@csdev-remote-abc ~]$ sudo yum -y install ntpdate
[abc@csdev-remote-abc ~]$ sudo ntpdate ch.pool.ntp.org
[abc@csdev-remote-abc ~]$ export TZ="UTC-2"
[abc@csdev-remote-abc ~]$ date
[abc@csdev-remote-abc ~]$ # if ok install into global profile
[abc@csdev-remote-abc ~]$ sudo cat 'TZ="UTC-2"'  >> /etc/profile

Configure remote and master server

Tip: The following section is based on the ecosphere article "censhare Remote Server - Cluster Multiple censhare Server".

To configure the remote as well as the master server, start your local censhare Server and connect to it via the censhare Admin Client. For example, if you want to compile the censhare Server start it and follow the logging constantly, e.g. you can type:

LOCAL-DEV $ ./censhare-Server/bin/build.sh clean init jar && censhare-Server/bin/StartServer.sh <YourMasterServerID> && less -iS +F censhare-Server/work/logs/server-0.0.log

After your local server is running open the censhare Admin Client and configure the server as follows:

Master data/Application servers

  1. Make sure the ID of the entry for Master server is identical to the one you have used to start your server via the StartServer.sh [YourMasterServerID] script. If it is different, configure it here accordingly.
  2. Create a new entry with the ID for your Remote Server (e.g. remote) and a meaningful name (e.g. Remote Server).

Configuration/Server

  • /General: Create a new configuration for your remote server
    • Server name: The name you've set up in Application servers (e.g. Remote Server)
    • Master server name: The name of your master (e.g. Master server)
  • /General: Adapt configuration for your master server
    • Server name: The name you've set up in Application servers (e.g. Master Server)
    • JVM properties:
      • java.rmi.server.hostname: The hostname you've set up in /etc/hosts on your remote machine (e.g. csdev-master-abc). Additionally activate the checkbox behind this entry to enable the hostname resolution.
  • /Launcher: Create a new configuration for your remote server
    • Server name: The name you've set up in Application servers (e.g. Remote Server)
    • JVM properties:
      • censhare.automatic.database.update: disable checkbox

Configuration/Services/Database/

  • Configuration: Create a new configuration for your master server
    • Server name: The name of your master (e.g. Master server)
    • Service enabled: YES
    • Connections, Maximal: Increase value to 210 (+60 for each remote server)
  • Configuration: Create a new disabled configuration for your remote server
    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: NO

Configuration/Services/Database Query/

  • Configuration: Create a new disabled configuration for your remote server
    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: NO

Configuration/Services/DatabaObject/

  • Configuration: Create a new configuration for your master server
    • Server name: The name of your master (e.g. Master server)
    • Service enabled: YES
    • Invocations: Increase value to 50 (+10 for each remote server)
  • Configuration: Create a new disabled configuration for your remote server
    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: NO

Configuration/Services/Filesystem/

  • Configuration: Create a new configuration for your remote server
    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: YES
    • Minimal File system: Just setup Temp as shown in the screenshot below
    • Note: If you plan to work with assets which persist on the remote server filesystems, a further elaborate setup would be needed later

Configuration/Services/Module service/

  • Configuration: Create a new configuration for all servers or modify the existing one
    • Server name: keep empty
    • Service enabled: YES
    • Path: ${runtime}/web:../censhare-Client5/web:../censhare-Server/web-commands
    • Mode: YES, enable Development
    • TypeScript: YES, enable TypeScript compilation
    • Node.js command: node

Configuration/Services/Remote-Server/

  • Configuration: Create a new configuration for your master server
    • Server name: The name of your master (e.g. Master Server)
    • Service enabled: YES
  • Configuration: Create a new configuration for your remote server
    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: YES
    • Add a new section to Remote Server
      • Remote system > Server name: The name of your master (e.g. Master Server)
      • Remote system > URL: Adapt to the hostname you've set up in /etc/hosts on your remote machine (e.g. rmis://csdev-master-abc:30546/corpus.RMIServerSSL)
      • Local system > URL: Adapt to your remote-server hostname (e.g. rmis://csdev-remote-abc:30546/corpus.RMIServerSSL)

It is required to setup authentication for the RMI connection between your master and remote server. You can either:

  • use SSL key pair authentication as described in this ecosphere article (recommended);
  • use passwords for the system user.

Note: The default password with the MD5 hash value 54b530... won't work. You can get the real password from the password-digest entry of your app/config/server.xml file. Make sure to additionally disable the checkbox Key Authentication.

Configuration/Modules/Administration/Maintenance/

  • Check database locks (automatic): Create a new configuration for all servers

    • Server name: keep empty
    • Service enabled: YES
    • Run only on master server: YES

  • Rebuild Index (automatic): Create a new configuration for all servers

    • Server name: keep empty
    • Service enabled: YES
    • Run only on master server: YES

  • Update Database Statistics (automatic): Create a new configuration for all servers

    • Server name: keep empty
    • Service enabled: YES
    • Run only on master server: YES

Configuration/Modules/Administration/Embedded database/

  • Embedded database sync using CCNs (automatic): Create a new configuration for all servers
    • Server name: keep empty
    • Service enabled: YES
    • Run only on master server: NO

Configuration/Modules/Asset Deletion/

  • Delete Assets (automatic): Create a new configuration for all servers

    • Server name: keep empty
    • Service enabled: YES
    • Run only on master server: YES

  • Mark Asset Version for Deletion (automatic): Create a new configuration for all servers

    • Server name: keep empty
    • Service enabled: YES
    • Run only on master server: YES

Configuration/Modules/Previews/

Either disable preview creation on remote server or install the image tools on the remote.

  • Create previews (automatic): Create a new configuration for remote server

    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: NO

  • Create text previews (automatic): Create a new configuration for remote server

    • Server name: The name of your remote (e.g. Remote Server)
    • Service enabled: NO

Apply configuration and start remote server

The configuration is now complete and can be written to your remote machine. To do that, execute the standard Update server configuration command from censhare Admin Client.

Then open a terminal and change to your censhare-product directory. Transfer the directory censhare-Custom to your remote machine:

LOCAL-DEV $ cd censhare-product/
LOCAL-DEV $ rsync -avz censhare-Custom csdev-remote-abc:censhare-product

Note: The rsync options from above will not remove files already uploaded to your server. If there are leftovers from previous syncs you will need to delete them manually.

You can now log in to your remote machine and start the remote server. After initialization the remote server should be connected to your master server:

LOCAL-DEV $ ssh csdev-remote-abc
Last login: Fri Nov 24 03:59:42 2017 from 10.157.0.97
[abc@csdev-remote-abc ~]$ cd censhare-product/
[abc@csdev-remote-abc censhare-product]$ ./censhare-Server/bin/StartServer.sh [YourRemoteServerID] && less -iS +F censhare-Server/work/logs/server-0.0.log

Replace [YourRemoteServerID] with the ID you've set up for your remote server.

Watch for RemoteServerService messages in the server log file to see the connection progress.

Update the configuration of the remote server

If both servers are running and you want to update the configuration of your remote server, you can do that from the censhare Admin Client (connected to your master). After you have configured and updated the master server, execute the action Synchronize remote servers to deploy your changes onto the remote server.

Result

If the synchronization was successful, you can connect to the remote server via https://csdev-remote-abc:9443/censhare5/client/.

Trouble spots

  • Hostname resolving

You may use nmap -PN -v -p 1099 csdev-master-abc to check if your master server is accessible from the remote server. But even with positive network connectivity the remote server may still be unable to reach the master server. A typical error message in server log:

2018.05.14-06:53:37.058 WARNING: RemoteServicesWatchdog: RemoteServerService: \
remote.RemoteServicesWatchdog: Failed to update remote server info: master
java.rmi.NoSuchObjectException: no such object in table

The error is not directly pointing to the problem. The "master" remote server is correct. It is just the name taken from application server master data. The solution is to properly set the java.rmi.server.hostname values in app/config/server.*.xml

  • Authentication with key pairs

We do not recommend changing the server names or filenames. Even if it looks strange (using twice "system@remote-server" as alias) it will prevent you from wasting time to sort out SSL certificate issues. Furthermore, the fallback with password authentication can be used as a workaround.