4. Cargador Service Installation

Attention

The “Cargador Service Installation” chapter contains information which you might need only if you are installing Cerebro along with local Cargador server, i.e., in hybrid cloud or local configuration (see “Deployment Options” chapter).

4.1. Installing Cargador onto a local server

4.1.1. Cargador Catalogue

Cargador catalogue is a directory that contains all files that go through Cerebro in one way or another. Files are stored in this catalogue after they have been attached to a forum message and downloaded/uploaded by users afterwards.

You must take into account total size of files that will be stored in Cerebro and select a file storage with sufficient capacity for the catalogue.

Note

In Unix-based systems you need to create a user account named cargador and make this user the owner of the catalogue folder.

The users in your LAN will access Cerebro files directly, using smb(cifs), nfs or similar protocols. So, you have to share the catalogue with them. In order to prevent accidental deletion or modification of the files in the catalogue it is strongly recommended to make the catalogue read-only.

You can add and move files inside the catalogue manually – Cargador will eventually track it (though it may take some time). Also, if you remove a file from the catalogue, this won’t cause service malfunction. Although, if a copy of the deleted file isn’t available elsewhere in other connected file storages, the Cerebro users won’t be able to access/download it.

Note

Cargador stores several service files in the catalogue. They are located in the .system subfolder. In most cases, if a subfilder name starts with a point - “.”, Cargador skips the subfolder while indexing users’ files.

4.1.2. Cargador Launch Directory

Next you will need to create a directory for Cargador executable and configuration files on your server (this must be a separate directory, different from the Cargador catalogue). For example, in Linux-based systems the folder can be /cerebro/cargador.

Unpack the following objects from the service-tools package:

  • executable file cargador.<linuxNN|macNN|win32.exe> according to your OS and its type (x86 or x64);
  • configuration file sample for your OS cargador.<linux|mac|win>.conf.example;
  • configuration file samples main.conf.example and cron_conf.py.example;
  • folders cron, py_cerebro, py_cerebro2, py-site-packages;
  • default image file absent.png.

Note

If the server uses Windows OS it is necessary to add CRT libraries (msvcm90.dll, msvcp90.dll, msvcr90.dll) to Cargador launch Directory, as well as Microsoft.VC90.CRT.manifest – they all can be found in the service-tools.zip package.

4.1.3. Basic parameters of a file storage

You can set basic parameters of the service by editing three settings in the main.conf file, the sample file is available in the service-tools.zip package. The default file looks like following:

<?xml version='1.0' encoding='UTF-8'?>
<!-- basic main.conf for cargador SERVICE/DAEMON -->
<config>
	<log>
		<debugLevel Value="2"/>
		<path Value="*** WHERE STORE LOG FILES ***"/>
	</log>

	<default>
		<cargador>
			<root Value="*** local path to catalog ***" />
			<net_root Value="*** network path to catalog ***" />
		</cargador>
	</default>
</config>

As you see, the information is saved in XML format, so you need to set three path values as follows.

  • log/path – path to the folder to store log files. In practice, log files can be very helpful sometimes, although you can switch the logging off deleting this key.

    You can adjust the debugging level by changing the debugLevel parameter (0 - inothing will be displayed, 10 - everything will be displayed).

    Note

    You have to specify an existing (best if dedicated) folder for Cargador log files. Cargador must have write permissions in this folder, so we recommend to make the cargador user the owner of the folder.

    Warning

    Under Linux, if you initially launch Cargador with root permissions, it will create a log file which it won’t be able to overwrite later if launched with restricted permissions.

  • cargador/root – a local path to the catalogue (see “Cargador Catalogue”).

    Warning

    Under Linux, if you initially launch Cargador with root permissions, then Cargador will create several service files which it won’t be able to overwrite later if launched with restricted permissions.

  • cargador/net_root – a network path for clients to access files in the catalogue.

    If you use client applications for different operating systems, these paths will be different for them, so yuo should specify only one of the paths here (Windows path, for example). Corresponding paths for other operating systems can be set up later, with directory mapping (see chapter “Directory Mapping”).

4.1.4. File Storage Sample Configuration

Let’s assume that you have a Windows-based server, named SERVER.

You created a Cargador catalogue with local path: x:\data\cargador.catalog. You shared this folder in the read-only mode with network path \SERVER\cargador.catalog.

You should write the following into the main.conf file (located in the same folder that contains Cargador executables):

<cargador>
    <root Value="x:\data\cargador.catalog" />
    <net_root Value="//SERVER/cargador.catalog" />
</cargador>

So if some file is stored in the catalogue with local path x:\data\cargador.catalog\someProject\file.foo, it means that its location for network users will be resolved by Cargador as //SERVER/cargador.catalog/someProject/file.foo (it doesn’t matter for Windows if there are forward slashes or backslashes in the path).

If you have workstations running under Linux or MacOS, this path will not work for them. To make the files accessible for Linux or MacOS users, you should do the procedure called directory mapping (see chapter “Directory Mapping”).

Note

In this example you may skip the directory mapping for Linux-based workstations. Instead of mapping, create local folders /SERVER/cargador.catalog on the workstations and mount the catalogue network path there. In this case Linux will locate the network files by the direct path //SERVER/cargador.catalog/someProject/file.foo.

Warning

Pay attention to the letter case when dealing with case sensitive operating systems.

4.1.5. File Structure Setup

The default file catalog structure that Cargador creates on your file storage is following:

{Project name}/{upload date}/{filename}

You can change the structure so that it represents a task tree of a corresponding Cerebro project. Cargador service supports up to 10 levels of folders and subfolders.

Use cargador.<linux|mac|win>.conf configuration file to set the format of the tree:

<catalog>
    <income_url_format>$(url[0])/$(date)</income_url_format>
</catalog>

The format of income_url_format is set by variables, which are folder names separated by a slash ‘/’.

The variables of income_url_format can have the following states:

  • $(url[0]), $(url[1]) … $(url[9]) – nesting level. Maximum level - 10.

    $(url[0]) corresponds to a project name in Cerebro, $(url[1]) corresponds to a task name of the project, $(url[2]) – to its subtask, etc.

    An example of setting a 3-level catalog tree:

    <catalog>
         <income_url_format>$(url[0])/$(url[1])/$(url[2])</income_url_format>
    </catalog>
    
  • $(date) – upload date folder.

  • $(date_utc) – upload date folder (in UTC format).

4.1.6. Deleting Files From Storage

Cerebro users can delete files that were previously uploaded to the storage. On deletion, first, the database entry is being erased, and after that, the file should be removed from the storage.

The trick is that one file can be referred to from several messages (possibly from different users as well), and in this case deleting one link should not affect file accessibility. On the other side, Cerebro may be configured to work with several storages, and that file may not exist on every storage.

Taking this issue into consideration, Cerebro uses the following file deletion procedure:

  • a Python script for file deletion is launched on every file storage periodically (on schedule);
  • a database query made from under a special account gets a list of all files marked for deletion by Cerebro users;
  • if a file from the list is found in a Cargador catalogue, it is physically deleted.

Note

You need to have Python (http://www.python.org) version 3.х or higher installed in your system to launch this script. Make sure that Python directory has been added to system path. This information, as well as the current Python version number, can be obtained by running the following command in the command line:

python3 --version

The file deletion script parameters can be configured by cron_conf.py script, its sample is located in service-tools.zip package. The set of parameters for file deletion procedure is described below:

DB_HOST=""
DB_PORT=45432
DB_SCHEMA="memoria"

CARGADOR_HOST=""
CARGADOR_RPC_PORT=4040

CARGADOR_DEL_PASSWORD='' # password for sweeping deleted files.
#    Must match with cargador.conf/config/catalog/del_password

Parameter values:

  • DB_HOST – database application server IP address (or DNS name) in the LAN or Internet (depends on Cerebro installation type, see “Deployment Options”);
  • DB_PORT – network port used by PostgreSQL on the database application server (default port is 5432);
  • DB_SCHEMA – database name (default name is memoria);
  • CARGADOR_HOST – Cargador server IP address (or DNS name) in the LAN (e.g., localhost);
  • CARGADOR_RPC_PORT – port used to connect to Cargador via XML-RPC protocol, used by the file deletion script (4040 by default);
  • CARGADOR_DEL_PASSWORD – password for file deletion from a storage (see the explanation below).

The CARGADOR_DEL_PASSWORD parameter must match with password in the cargador.<linux|mac|win>.conf configuration file, the sample of which you unpacked from the service-tools.zip archive into the Cargador launch directory.

During standard Cargador installation procedure, the only value you must set in this file is the password - the del_password parameter in the config/catalog section. You may specify any value here, the only requirement is that it must be equal to CARGADOR_DEL_PASSWORD value in cron_conf.py.

4.1.7. Cargador Additional Parameters

Besides file storage management, server-based Cargador can handle some additional functions. The possible options for interaction between Cergador and other modules are:

  • file exchange and storage management (default ports: 45430, 45431) – it is used by client applications to upload/download files via LAN and/or Internet;
  • HTTP protocol (default port: 4080) – it is used to download files from a storage using HTTP protocol (following a link from an email message or in the Cerebro web interface);
  • XML-RPC protocol (default port: 4040) – it is used to call Cargador procedures remotely, for example, by Cerebro API Python scripts (for more information, please refer to Cerebro Python API manuals).

The cargador.<linux|mac|win>.conf file is used to configure Cargador access via HTTP and XML-RPC. It is necessary to specify ports to address XML-RPC and HTTP queries in the file:

<tcpServer>
    <httpGate>
        <port>4080</port>
    </httpGate>

    <rpcGate>
        <port>4040</port>
    </rpcGate>
</tcpServer>

After saving changes in configuration file, restart Cargador service for changes to take effect.

4.2. Service Start and Setup

We strongly recommend you NOT to install/deploy Cargador as a service/daemon until you make sure that the system is set up properly. The best option for debugging is to run Cargador iteratively from the console (in Windows use the cmd.exe application for that).

For Linux or MacOS it is recommended to run the terminal under the user account which is intended to be used for Cargador later on (use the su command). This recommendation is not critical for Windows-based servers.

For the first launch go to the Cagador executable folder in the console and run the file. The program must display a text like this:

.cargador.info : Running at c:/cerebro.files, net map: <//server/cargador>
.cargador.Warning : {code:(Object not found), 'Не удается найти указанный файл. c:/cerebro.files/.system/download.queue'} - load queue file failed
.cargador.Warning : {code:(Object not found), 'Не удается найти указанный файл. c:/cerebro.files/.system/upload.queue'} - load queue file failed

Pay attention to the first line. It should display the paths specified in the main.conf configuration file. Two other warnings can be ignored on the first launch. They inform you that service files can’t be opened, because the catalogue is empty for now.

If you need to restart Cargador, press Сtrl+C to stop the process.

Warning

Make sure that your firewall allows incoming connections to ports 45430, 45431, 4080, especially if you’re installing Cargador under Windows. Use the telnet command to check the connections (see chapter “Checking TCP connection with telnet utility”).

4.3. Setting Up Cargador Service/Daemon

Cargador is a single executable file. The ways to run it automatically on server start are different on different platforms.

Note

You can get helpful information about command line keys by running Cargador executable with a -h or --help key.

4.3.1. Installing Under Windows

In Windows Cargador can be installed as a system service. To do that, you should run the following command as an administrator (in Windows Vista/7 you should use the Run As Administrator menu item):

cargador.win32.exe --service_install

Besides, you can set a username and a password for the account Cargador should work with using command line keys --service_as_user и --service_as_passwd.

You can remove service from the system with the following command:

cargador.win32.exe --service_remove

Warning

Make sure that launched process has a write permission in folders containing the catalogue and the log files.

4.3.2. Installing Under Linux

In Linux daemons are started using rc scripts. A sample of such a script is located in the service-tools.zip archive, in the linux-init.d folder – a file named cargador.<general|debian> - depending on your Linux OS type.

In the launch script you should specify location of executable file on your server and probably change username and group. By default, script contains following values:

CARGA_SERVE=/usr/local/bin/cargador
CARGA_USER=cargador
CARGA_GROUP=cargador

After being edited, the file should be placed into the corresponding folder – usually it is /etc/init.d.

Then you have to create links to the script in the structure of rc folders. Locations and formats of these links vary from one Linux distribution to another. For example, for a Debian-like Linux (e.g., Ubuntu) use the following command:

update-rc.d cargador.debian defaults

4.3.3. Installing Under Mac OS

In Mac OS X services are launched by the launchd daemon. It searches for *.plist files in the /Library/LaunchDaemons folder. Then it reads its contents and launches corresponding daemons.

A sample of such a file is located in the service-tools.zip archive, in the macos-launchd folder.

The com.cerebrohq.cargador.plist file has a simple text format. You should configure the location of Cargador daemon, as well as user names and group names on behalf of which the service should be started. Pay attention to the following parameter:

<key>RunAtLoad</key>
   <true/>

You can prevent Cargador daemon from launching on your server by changing its value to false.

4.4. External Access to the File Storage

In order to provide external access for uploading / downloading files from your local storage, you need to make Cargador accessible from the Internet via TCP ports 45431 and 4080.

As a rule, local networks are located within a private IP address range and contact the Internet via a router with NAT running on it.

If you have this type of network structure you have to set up port forwarding on your router from 45431 and 4080 external ports to your Cargador server within your LAN.

In hardware routers this function is usually called VirtualServer. If linux-box is used as a router, you can get it using these commands (they must be added to rc-scripts):

iptables -A PREROUTING -t 45431 --dport 45431 -j DNAT --to-destination <SERVER IP>:1234

or:

redir --lport=45431 --cport=45431 --caddr=<SERVER IP>

Warning

Make sure that the external connection to Cargador is not blocked by a firewall on your server or on a router.