Synthesepy

The synthese.py script

This manual describes the synthese.py script, which is used to manage Synthese (building, installation, …) and Synthese projects. The various configuration settings are detailed.

Invoking the script

The synthese.py script is a Python script. It can be run by running the synthese.py script in the sources (synthese3/tools/synthese.py) or from an installed location (/opt/synthese/bin/synthese.py by default) using a Python 2.6 or 2.7 interpreter (Python 3 not supported). For instance:

python ~/dev/synthese3/tools/synthese.py ARGS...

Script storage path

The script will use a directory (called the {thirdparty_dir} below) where it stores and reads a number of files:

a Python “environment” directory ({thirdparty_dir}/env)
script configuration files ({thirdparty_dir}/config)
build dependencies ({thirdparty_dir}/…)

The default location of that directory is in ~/.synthese/ on Linux (or USERPROFILE\.synthese on Windows). That directory can be changed by setting a SYNTHESE_THIRDPARTY_DIR environment variable (NOTE: that variable then needs to be set every time you run the synthese.py script, it might be good to add it to your permanent environment).

The first time the script is launched (or each time the dependencies version is updated), it will create a Python environment in a directory. The script can take a while to run when the environment is created and it requires an internet connection for fetching dependencies.

The script alias

To save on typing, it is recommended to define an alias in your shell.

The sy alias will be used in the following steps.

Windows

Create a file named sy.bat in a directory present in the PATH environment variable, with the following content :

@ECHO OFF
c:\Python27\python.exe c:\synthese3\tools\synthese.py %1 %2 %3 %4 %5 %6 %7 %8 %9

Note : The paths must match your configuration.

Linux

alias sy='python ~/rcs/synthese3/tools/synthese.py'

Windows with Cygwin

As an alternative to the .bat script above, you can use the following alias in your Cygwin shell.

alias sy='/cygdrive/c/Python27/python $(cygpath -aw ~/dev/synthese3/tools/synthese.py)'

Script modes

The synthese.py script operates in two modes:
1) general mode (no -project/-p argument given)
2) project mode (-project/-p argument given)

Configuration

The Synthese script can take parameters from the command line and from configuration files. Using parameters in configuration files helps to reduce the amount of typing required and to organize build and projects.

By default, the configuration files are located in $HOME/.synthese/config directory on Unix or %USERPROFILE%/.synthese/config on Windows. See the “Script storage path” section above for details. Alternatively, that path can be specified with the --config-path command line parameter.

The configuration is read from a number of files, interpreted in order (so that a later file can override a parameter in an already parsed file):

config.py
config_local.py
config_local_.py (where is the computer’s host name).

Example configuration

Below is a sample configuration file that can be used as a starting point. You may want to adjust paths and settings according to your environment. See above for the path where it should be stored.

import os
from os.path import join
import sys

common = {
    # Set here 
    'test_conn_strings': [
        'sqlite://',
        'mysql://debug=1,db=synthese_test,host=mysql-hostname,user=synthese,passwd=synthese'
    ],

    
    # Larger session time to avoid frequent logins.
    # You can also add a auto_login_user=root parameter for being automatically
    # logged as a specific user (for development only, huge security hole!).
    'extra_params': 'session_max_duration=999999',
    # Uncomment to kill the daemon on Windows when building.
    #'kill_daemons_when_building': True,
}

# Path to spatialite Windows tools:
SPATIALITE_BASE = r'C:\dev\spatialite'

if sys.platform == 'win32':
    common['spatialite_gui_path'] = SPATIALITE_BASE + \
        r'\spatialite-gui-win-x86-1.2.1\bin\spatialite-gui.exe'
    common['spatialite_path'] = SPATIALITE_BASE + \
        r'\spatialite-tools-win-x86-2.3.1\bin\spatialite.exe'
    common['editor_path'] = r'C:\Program Files (x86)\Notepad++\notepad++.exe'

dbg = {
}

rel = {
    'mode': 'release',
}

rd = {
    'mode': 'relwithdebinfo',
}


# Change here to the path containing all the projects.
if sys.platform == 'win32':
    PROJECTS_PATH = 'c:\\dev\\rcs\\projects'
else:
    PROJECTS_PATH = os.path.expanduser('~/dev/rcs/projects')

# Sample project specific configuration
demo = {
    'example_config_parameter': 'some_value',
}
myproject = {
    'project_path': r'c:\tmp\myproject',
}

PROJECTS = (
    # (Project name, alias1, alias2, ...)
    ('demo', 'd'),
    ('the_project', 't'),
)


for project_parts in PROJECTS:
    project_name = project_parts[0]
    aliases = project_parts[1:]
    if project_name in locals():
        proj_config = locals()[project_name]
    else:
        proj_config = {}
        locals()[project_name] = proj_config

    if not 'project_path' in proj_config:
        proj_config['project_path'] = join(PROJECTS_PATH, project_name)

    for alias in aliases:
        locals()[alias] = proj_config

Using the configuration

The Synthese script can take a -c command line parameter to specify the name of the variable (a Python dict) in the configuration files that contains the options to use. The common variable (a Python dict) can contain options to use in any case. This mechanism makes it possible to create “profiles” easily recallable from the command line.

With the configuration above, you can build in debug mode with this command:

sy -cdbg build

Or in release with

sy -crel build

It is possible to specify several configuration variables. They will be loaded in order where the latter ones override the former ones. For instance you can have a “dbg” or “rel” configuration entry to specify debug or release mode, and then provide a “myproject” entry with project specific settings. Then you can run your project in debug or release mode with:
Run project with Synthese Debug

sy -cdbg,myproject run

Run project with Synthese Release

sy -crel,myproject run

Additionally, a parameter has been specified with the database connection information for running the MySQL tests.

Configuration options

By default, the configuration file options have the same name as the command line options, with “-” replaced by “_”. For instance, --boost-dir=/opt/boost command line option is equivalent to this line in the configuration file:

boost_dir = "/opt/boost"

Some options are only available from the configuration file. See each command below for the list of options that can be used from the command line or in the configuration file.

Global options

Some options are global: it means that they can be used with any commands. You can find below the list of all the global options, with a description.

-h, —help show this help message and exit —config-path CONFIG_PATH Directory containing the configuration files -c CONFIG_NAMES, —config CONFIG_NAMES Configuration entries to use

See the “Using the configuration” chapter.

-p PROJECT_PATH, —project-path PROJECT_PATH Path to the synthese project to use

See the projects chapter

-v, —verbose Be verbose -u, —dummy Dummy mode, doesn’t execute commands or have side effects WARNING: NOT FULLY IMPLEMENTED YET -t {cmake,installed}, —env-type {cmake,installed} -b ENV_PATH, —env-path ENV_PATH Env path

This parameter can be used to specify the environment path. It corresponds to the path where synthese is built or installed.
It can be used if you want to build Synthese in a non default location (other than build_cmake/ )

-m {debug,release,relwithdebinfo}, —mode {debug,release,relwithdebinfo}

See the build command.

—beep Emit a beep on completion —port PORT —no-proxy —site-id SITE_ID —gdb Run daemon under gdb —restart-if-crashed Automatically restart the daemon if it crashes -l LOG_LEVEL, —log-level LOG_LEVEL Daemon log level -s, —stdout Log daemon output to stdout —static-dir STATIC_DIR Directory containing static files served by the HTTP proxy -e EXTRA_PARAMS, —extra-params EXTRA_PARAMS Daemon extra parameters, using format "param0=value0 param1=value1"

See the “rundaemon” project command for details.

-d CONN_STRING, —dbconn CONN_STRING

See the conn_string parameter in the projects chapter.

—no-root-check Disable root user checks -n ENV_CONFIG_NAMES, —env-config-names ENV_CONFIG_NAMES Environment configs to use (comma separated)

General Mode Commands

The commands in general mode are for managing aspects not related to projects, such as building, installing, running the tests, creating projects, …

build command

That command is used for compiling Synthese and its related tools (tests, utilities, …)

The build script will download all the required dependencies. Make sure you have network connectivity the first time it is launched. See the “script storage location” section above for details on where the dependencies are stored.

Build related general options

NOTE: general options must be specified before the command (e.g. sy -m release build)

-m MODE (config:mode)

the build mode. It can be debug, release or relwithdebinfo (a release build, which has symbols unstripped for debugging).

build specific options

NOTE: specific options must be specified after the command (e.g. sy build -g)

-g, —generate-only Generate only the build files, but don’t start the build.

This is useful for development on Windows, where you may prefer to only generate the Visual Studio project first (by using -g), and then use the “ide” command (see later) to open Visual Studio and run the build.

—mysql-params MYSQL_PARAMS MySQL connection string used for the unit tests. For instance: “host=localhost,user=synthese,passwd=synthese” —without-mysql Disable MySQL database support —mysql-dir MYSQL_DIR Path to MySQL installation (Not needed on Linux if using standard MySQL installation) —boost-dir BOOST_DIR Path to Boost installation (Not needed on Linux if using standard Boost installation) —parallel-build PARALLEL_BUILD Number of build threads to use —kill-daemons-when-building Kill all running daemons before building -o BUILD_ONLY, —build-only BUILD_ONLY Build only the specified project

This option is very useful if you want to build only the Synthese daemon without all the tests. Use “-o s3-server” in this case.

-d, —clear-cache (config: clear_cmake_cache) Clear the CMake cache before building.

Other config file only options

x64 (default: True)

Builds in 64bit mode (32bit mode is untested).

Example usage

sy -m debug build

Build output

The build will be produced in the directory {synthese_path}/build_cmake/MODE (for instance, with a debug build on Linux: /home/user/synthese3/build_cmake/debug).

clean command

The clean command deletes the build directory.

install command

Install Synthese to the prefix directory specified when running the build command.

ide command

Launch the IDE with a configured environment for running tests.

This is only used on Windows for now. It will set some environment variables before launching Visual Studio, so that the tests are able the load the required dlls.

See also the Automated-tests page for more details about the tests.

testshell command

Launch a new shell with a configured environment for running tests. This is similar to the “ide” command, except that it runs a shell with the new environment instead of opening Visual Studio.

doxygen command

Generate Doxygen documentation.

runtests command

Run unit tests. For more information about the unit tests, see Automated-tests

runtests specific options

—test-conn-strings TEST_CONN_STRINGS [TEST_CONN_STRINGS …] Database connection strings used for the tests —no-init Don’t start/stop the daemon or initialize the db. Can be used to reuse an already running daemon (see —test-daemon-only) —test-daemon-only When a HTTPTestCase is run, initialize the project and keep the daemon running. Meant to be used in parallel with —no-init

sqlite_to_mysql command

Convert a SQLite database to MySQL

sqlite_to_mysql specific options

—sourceconn SOURCECONN SQLite database input connection string (i.e. sqlite://path=/myproject/db/config.db3) —targetconn TARGETCONN MySQL database output connection string (i.e. mysql:// host=localhost,user=synthese,passwd=synthese,db=synthese)

create_project command

Create a Synthese project. For more details about Synthese projects, see the “Projects” chapter below.

create_project specific options

—path PATH Path of the project to create —system-packages [SYSTEM_PACKAGES [SYSTEM_PACKAGES …]] System packages to include in the created project —conn-string CONN_STRING Database connection string of the created project —overwrite Erase destination if it already exists

Example

sy create_project —path c:\\tmp\\myproject —system-packages demo

This creates a new project on the given path which includes the demo system package.

continuous_integration (ci) command

Run the continuous integration commands.

This command is mainly useful when using a continuous integration tool, such as Jenkins.

It can be used to build Synthese, run the unit tests and create a binary distribution. Refer to the following commands for more details (they are executed in order).

build
package
runtests

continuous_integration specific commands

—no-clean-if-build-fails Don’t clean the build directory and build again in case of build failure —no-package-overwrite Don’t overwrite existing package. —force-create-package Force package creation. —no-tests Don’t run unit tests.

Other config file only options:

suites (default: all)

Which test suites to run

should_build_package’: lambda env: False

A Python function that is called to decide if a package should be built or not. This is not used if —force-create-package is true.

system_install command

This command will install all the Synthese dependencies (it only works on Debian Linux at the moment).

This command is automatically executed when you install Synthese using the package (see the package command below).

package command

Create a Synthese binary distribution package.

A Synthese binary package is simply a .tar.gz file that contains all of the required binaries and files for Synthese to run on a system.

Constraints:
– A Synthese package is not portable across platforms or distributions, because it depends on system libraries.
– The environment should be the same between the build machine and the target machine.

RCSmobility offers Synthese packages for Debian 6 64bit from the latest Synthese trunk versions at http://ci.rcsmobility.com/~build/synthese/lin/release/trunk/

Package install script

When a package is built, a script is also generated alongside the binary package for installing the package.

This script requires you to set up the packages_save_path and packages_access_url options. The package command will generate a package into the path:

PACKAGE_SAVE_PATH/PLATFORM/MODE/BRANCH/REVISION

Where:

PLATFORM: platform name (lin for Linux, win for Windows)
MODE: build mode (debug, release, relwithdebinfo)
BRANCH: branch name (trunk, …)
REVISION: revision name (rNNNN).

The package command will also create a symbolic link “latest” which points to the latest revision built. This allows you to use a fixed URL for installing the most recent available package.

To allow the install script to be used for installation through HTTP, you should make the resulting .tgz packages accessible from your HTTP server.

To install the package using the generated script, you can use the following command:

curl -s http://SERVER/path_to_package/lin/release/trunk/latest/install_synthese.py | python

See Debian-Installation-TGZ for more detailed instructions.

package specific options

—no-package-overwrite Don’t overwrite existing package.

Other config file only options:

packages_save_path (default: None)

Path where to save the binary packages (see above for where the package is saved).
Example: os.path.expanduser('~/public_html/synthese/')

packages_access_url (default: None)

URL used to access the generated package directory.
Example: 'http://your_server/~%s/synthese/' % getpass.getuser()

checker command

Synthese checker interface for finding regressions.

More information about the checker tool can be found on Synthese-Checker-Tool.

dashboard (d) command

Run the Synthese Dashboard Web server for managing projects.

Projects

Introduction

Synthese uses a project mechanism to run. The project contains:

Project configuration
Database
Static files
Initial data to load in the database

The synthese.py script is used to manage projects, such as starting the daemon, loading initial data and so forth.

Structure of a project

Here’s a sample structure of a project

myproject
  config.py: Project configuration file
  db: Database (for SQLite)
  impex: old location for storing import scripts and data.
  manager
    imports_config.py: optional imports configuration
    main.py: optional script for custom project commands
  logs: log files
  sites: Directory with all sites
    admin: The admin site (for example)
      config.py: site specific configuration
      admin: admin package
    www: Another site (example)
      config.py: site specific configuration
      main: package for the site specific files and configuration.
      mypackage: some package

  packages: optional directory to store packages.
  
  # Generated directories
  imports: storage for imports logs and data (automatically generated)
  conf
    generated: generated config files
  deploy: storage for deployment files.

Creating a new Synthese project

See the “create_project” command documented above.

project/*config.py

Project configuration is split into several files. Some configuration files are optional and only relate to some specific feature (imports, custom scripts, …).

project main configuration

The main configuration file for project is the config*.py files at the root of the project. The settings can be put in several files, which are read in order (this is used to have host specific configuration):

config.py
config_local.py
config_local_.py (where is the computer’s host name).

Below is a list of configuration option that you can define in the config*.py files.
On the right of the “:” is the default value.

‘port’: 8080,

Port that the daemon should listen on.

‘wsgi_proxy_port’: None,

Port used by the Development Web server (defaults to port + 1).

‘conn_string’: ‘sqlite://’,

Database connection string. Can be omitted if using sqlite with the default database location (in project/db/config.db3).
For another location, you can use:

sqlite://path=/myproject/db/config.db3

If you want to use MySQL, you can use:

mysql://db=synthese_test,host=localhost,user=synthese,passwd=synthese ‘sites_config’: {},

This can be used to specify configuration in the config.py file that would normally go in the site configuration (useful for overriding a site configuration from a host specific file). For instance:

sites_config['www'] = {
    'apache_vhost_port': '*',
    'apache_server_name': 'mywebsite',
}

  'mail_admins': (),
  'mail_sender': None,
  'mail_host': 'localhost',
  'mail_port': 25,
  'mail_user': None,
  'mail_password': None,
  'mail_tls': False,

Mail configuration. The mail_admins value is used for sending crash or hung emails (see send_mail_on_restart).

‘use_udf_proxy’: False, ‘udf_proxy_port’: 9080, ‘udf_proxy_options’: ’’, ‘udf_proxy_dispatch_ports’: None,

Configuration for using the UDF proxy. It is now recommended to use the newer s3 proxy instead.

‘use_s3_proxy’: False, ‘s3_proxy_port’: 9080, ‘s3_proxy_password’: ‘changeme’,

S3 proxy configuration See S3-Proxy for more details.

‘send_mail_on_restart’: False,

If true, send a mail to the mail_admins when a restart happens.

  # web frontend configuration
  # Put in the project config.
  # Generate with: python -c "import os; print repr(os.urandom(24))"
  'web_secret_key': 'override me!',

You should set a secret key in the project as specified.

‘web_debug’: False,

Set to True when debugging the manager interface.

‘web_admins’: set((‘root’,)),

List of Synthese users that are considered administrator for the manager interface.

  # remote project management and deployment
  'remote_server': None,
  'remote_project_path': '/srv/synthese/{project_name}',
  # If dealing with an old-style project, use this instead:
  # '/srv/data/s3-server/config.db3',
  'remote_db_path': None,
  'remote_dump_prefix': 'remote_',

Configuration for remote project management. See the following project commands “db_remote_dump” and “deploy” for more details.

‘ssh_global_opts’: ’’, ‘ssh_opts’: ’’, ‘rsync_opts’: ’’,

ssh and rsync options that can be set during deploy or database transfer.

  # ineo realtime
  'ineo_conn_string': None,
  'ineo_stop_code_prefix': '',
  'ineo_ver': 1,
  'ineo_planned_data_source': '16607027920896001',
  'ineo_realtime_data_source': '16607027920896002',

Configuration for ineo.

  # bgstart/bgstop
  # One of: supervisor, initd, python, dummy
  'bg_process_manager': 'python',

Configures how the daemon should be started:

python: this is the default mode: synthese.py will launch the Synthese daemon directly, and run a development Web server for serving static files. This mode is best suited for development.
supervisor: Syntheses is launched through supervisor. This is the mode to be used in production
initd: option for backward compatibility with the old init.d script
dummy: dummy launcher that does nothing. For debugging.

‘supervisorctl_path’: ‘/usr/bin/supervisorctl’,

Path to the supervisorctl command, if the executable isn’t at the default location.

project/manager/imports_config.py

See Imports-configuration.

project/manager/main.py

See the project_command (pc) chapter for more information.

Sites

Introduction

A project can contain one or more sites. Sites maps to the site concept of Synthese (similar to a Apache virtual host).

Each site can contain one or more packages. See the Packages chapter below for more information.

Example site structure:

myproject sites www config.py: site specific configuration main: package for the site specific files and configuration. mypackage: some package

The convention for site names is the following:

www: main website
admin: the site for the administration interface
mobile: mobile website

Configuration

Site configuration is located in the config.py file inside the site.

id

Synthese identifier of the site.

root_page_id

(optional) id of the Synthese page that should be used when accessing the root (“/”) url of the site.

site_id

The built-in Web server for serving static files doesn’t support virtual hosts. If you have several sites, it is required that you specify which site it should serve static files for.

default_site

The default site is used in the Apache configuration to specify which Synthese site is served if no virtual host match any site. It should be the name of one of the available sites.

The following options are related to the generated Apache configuration file. For more detail how this is managed, see the system_install, system_uninstall, system_install_prepare commands below.

apache_server_name

Name of the Apache server

apache_server_admin

Email of the server administrator

apache_aliases

List of Apache virtual hosts for this site.
Example:

apache_aliases = (
‘alias1.example.com’, ‘example.org’,
)

apache_custom_config

Additional Apache configuration to insert in the generated file:

Example:

apache_custom_config = """ ErrorLog /var/log/apache2/synthese-error_log CustomLog /var/log/apache2/synthese-access_log combined

don’t loose time with IP address lookups
HostnameLookups Off
needed for named virtual hosts
UseCanonicalName Off
configures the footer on server-generated documents
ServerSignature On
"""

rewrite_rules

List of rewrite rules that will be added to the built-in ones.
Example:

rewrite_rules = [ [‘^screen/(.*)$’, ‘synthese?SERVICE=tdg&m=$1’], [‘^monitor/(.*)$’, ‘synthese?SERVICE=tdg&m=monitor_$1’], ]

synthese_user
synthese_group

Used for the WSGI Apache configuration: system user and group that will be used to run the WSGI processes.

Packages

Introduction

A package contains static files and initial database content.

Note: the “main” package is special. It is the one used as a site DocumentRoot in Apache.

Content

Here’s an example of a package content:

core files core pages pages.py pages_local pages.py sample.html apache.conf: optional apache configuration to include config.py: package configuration install.sql: optional sql file to load

Content detail

files

This directory contains the static files that will be reachable using the url http://SERVER/PACKAGE_NAME/FILE_NAME

For instance, if you store a file into core/files/core/hello.html it will be accessible with http://server/core/hello.html

By convention, and to prevent conflicts, the static files should be put in a directory that has the same name as the package.

An exception is the “main” package, where you can put files directly under “files”, which can then be accessed at the root of the website.

files_local

The files_local directory can contain files that are meant to be loaded inside the project only once. They can be seen as template files that you can copy in the project and then override.

When you run the load_local_data command (see below for more details), the files in that directory will be copied to the “main” package of your site, into the “main/files/local” directory.

For instance, if you have a package “map” with a file inside “map/files_local/map/img/myImage.png”, it will be copied (only if it doesn’t already exist) into your site main package: “myproject/sites/www/main/files/local/map/img/myImage.png”. Then, you can access it with “http://yourserver/local/map/img/myImage.png”

pages and pages_local

The pages and pages_local directory can be used to store content that should be loaded into the site CMS.

There are two categories of pages that can be loaded into the CMS. The semantic is similar to the files and files_local separation discussed above:

“pages” are the pages that are not meant to be overridden in your project. They should be considered as read-only in the project (they can still be modified due to a technical limitation, but you shouldn’t do that). These pages will overwrite existing pages each time you run the load_data command (see below for more detail about the load_data command) or when you update the project from the manager interface and you have the “Run load_data and load_local_data” checkbox checked (it is checked by default).
In the CMS, these pages appear under the hierarchy of your site, under a “package/PACKAGE_NAME” folder.

The “local pages” are pages that are meant to be modified in your project. They won’t be overwritten after they are loaded (unless you use a specific option). They can be considered as templates, that are loaded when you initialize your project. If these pages changes after you project is created, you have to manually merge the change.
In the CMS, these pages are attached to a special site named “Shared Pages (read-only)”, under a “package/PACKAGE_NAME” folder.

The way to create a page in the pages or pages_local directories is similar: you need to create a config.py file with the following structure:

pages = [{
    'id': pid(0),
    'rank': 0,
    'title': 'packages',
    'content1': '',
}, {
    'id': pid(1),
    'rank': 0,
    'up_id': pid(0),
    'title': 'core_local',
    'content1': '',
}, {
    'id': pid(2),
    'rank': 0,
    'up_id': pid(1),
    'title': 'Sample page',
    'smart_url_path': '/sample.html',
    'content1': 'file:sample.html',
}]

Each item in the pages list corresponds to a page to be created. The attributes are the same as the columns of the t063_web_pages table.
You can use the pid() special function to generate identifiers for the page (it is implemented in a way that generates unique ids per package).

You can load the content of a page from a file inside the pages or pages_local directory. To do this, set the “content1” attribute with a value “file:myFile.html” and then add a file with the given name inside that directory (myFile.html in that example).

apache.conf

You can include such a file in the package. When the Apache configuration is generated (see the system_install command below), the content of that file will be included at the end of the Apache configuration file.

config.py

This file contains configuration data for the package.

dependencies = (‘core’, ‘map’)

This lists the other packages this package depends on.

base_page_index = 1

In order to generate unique identifiers for pages that are loaded in the CMS, each package needs to have a unique base_page_index. The list of base_page_indexes can be found on Packages#Packages-properties. You should add an entry there when you add a Synthese package.

.sql, .sql.gz

You can include .sql files (optionally gzipped) inside your package. These .sql files will be imported when you run the “load_data” project command.

The Manager Interface

The synthese.py script provides a Web interface for managing Synthese. You can access it with the URL: http://server/w/manager. The login uses the same users as the Synthese administration interface.

Important Note: When you are using Apache to serve the Manager interface, you need to restart Apache if you update the project configuration and want your modifications to be taken into account.

Project deployment

The synthese user

For security reasons, the synthese daemon runs with the user “synthese”. Project files should belong to that user. It is recommended to log in as the synthese user for managing the project.

Preparing the system for projects

# as root:
mkdir /srv/synthese
chown synthese. /srv/synthese

Then, either create the project with the create_project command or checkout it into /srv/synthese/PROJECT_NAME

Install the project on the system

This will register the daemon to run automatically, generate the Apache configuration and register it with the system apache.

Log in as root

# Make sure supervisor is running.
supervisorctl status
# if it indicates it is not running, stop and start it:
/etc/init.d/supervisor stop
/etc/init.d/supervisor start
# If you get an error, try killing the supervisord process, or removing the /var/run/supervisord.pid file
# TODO: investigate why it doesn't work well sometimes.

# Install the project on the system
python /opt/synthese/bin/synthese.py -p /srv/synthese/myproject/ system_install

If you want to unregister the project from the system:

python /opt/synthese/bin/synthese.py -p /srv/synthese/myproject/ system_uninstall

For more details, see the system_install, system_uninsatll and system_install_prepare commands below.

Project Mode Commands

To run a project command, you should pass the “—project” or “-p” command line argument with the path of the project to the synthese.py script or alias.

In order to reduce on typing, you can use the “-c” option. See the “Using the configuration” chapter above for more details.

rundaemon (run,start) command

This command is used to launch the Synthese daemon. It will launch the Synthese daemon on the configured port and run a built-in Web server on another port (port + 1 or wsgi_proxy_port if specified) for serving static files and proxying other requests to Synthese (un.

A few command line parameters or project options influence how Synthese is launched:

rundaemon related general options

—port PORT

Port the daemon should listen on.

—no-proxy

If specified, do not start the built-in Web server which serves static files and proxies requests to the Synthese daemon.

—site-id SITE_ID

See the “site_id” project configuration option above.

-d CONN_STRING, —dbconn CONN_STRING (config: conn_string) -l LOG_LEVEL, —log-level LOG_LEVEL Daemon log level -s, —stdout (config: log_stdout) Log daemon output to stdout —static-dir STATIC_DIR Directory containing static files served by the HTTP proxy -e EXTRA_PARAMS, —extra-params EXTRA_PARAMS Daemon extra parameters, using format "param0=value0 param1=value1" —gdb Run daemon under gdb —restart-if-crashed Automatically restart the daemon if it crashes

These options should be self explanatory.

load_data and load_local_data command

Loads project data in the project. The content loaded inside the project is detailed in the project “Content detail” chapter.

reset command

This command is used to clear the database content and load the initial data.

WARNING: this will delete all the added data.

stopdaemon (stop) command

Stops the synthese daemon.

runproxy command

This command can be used to run the development Proxy server only (to serve static files). It is best used in conjunction with the run “runproject” command with —noproxy.

runwebapp command

This command can be used to run the manager Web application only.

project_command (pc) command

This command can be used to run project specific Python scripts. These scripts are stored in the project/manager/main.py file.

When you run that command, synthese.py will call the project_command Python function declared inside main.py. You can use the “—args” command parameter to specify parameters that you want to give to the project_command function. The project_command function is called with the parameters:

project: reference to the current project object.
args: additional arguments given on the command line.

Here’s an example:

in project/manager/main.py

def project_command(project, args):
    if args[0] in ('hello',):
        print "Hello World project. Project path is %s" % project.path

You can call that script with the arguments:

sy -p /path/to/myproject pc —args hello

All the synthese.py API is available in the project scripts. For instance, you can run database queries:

def extract_main_connection_places(project):
    sql = """
        select distinct
            cp.id
        from
            t007_connection_places cp
        where
            cp.is_city_main_connection = 1
    """

    ids = {}
    for obj in project.db_backend.query(sql):
        ids[obj['id']] = 1

    output_path = join(thisdir, os.pardir, 'sites', 'mysite',
        'main', 'files', 'data', 'mainConnectionPlaces.json')

    with open(output_path, 'wb') as f:
        json.dump(ids, f, sort_keys=True, indent=2)

def project_command(project, args):
    if args[0] in ('extract_main_connection_places', 'emcp'):
        extract_main_connection_places(project)

sy -p /path/to/myproject pc —args extract_main_connection_places

Database related commands

db_view

Open the database in spatialite GUI (for sqlite only).

db_view_gis

Open the database in spatialite GIS GUI (for sqlite only)..

db_shell

Open a command interpreter on the database.

db_dump

Create a database dump into project/db

db_open_dump (db_opendump)

Open a database dump in the configured editor.

db_sync_to_files db_sync_from_files db_sync

Save the CMS web pages to disk or vice-versa.

db_restore

Restore a database dump.

db_remote_dump

Dump a database from the remove_server to project/db

db_remote_restore

Performs a remote database dump and then a restore from that dump.

ssh command

Open ssh to the configured remote server.

imports command

This command is used for managing imports from the command line. See Imports-configuration for more details. You can also manage imports from the manager Web interface.

system_install_prepare, system_install and system_uninstall commands

Introduction

These commands are used to install Synthese on the system. In particular, they register with the following applications:

supervisor: Supervisor is a tool for running commands and managing log files, startup at boot, restart in case of crash, … synthesepy will generate a supervisor config file and register it on the system
apache: Apache is a Web server. synthesepy will generate an Apache configuration file and register it on the system.
wsgi: A wsgi script is generated.

The system_install_prepare commands will create the configuration files in the conf/generated directory.

The system_install command will add symlinks pointing to the generated files.

Supervisor

The supervisor package is used to manage the daemon. The supervisorctl command is used to interact with the daemon. When the project is installed on the system, a supervisor entry called “synthese-” is created. You should use the root user to call the supervisorctl command. To start/stop/restart the daemon, use:

supervisorctl stop|start|restart synthese-<project_name>

You can see the logs live with:

supervisorctl tail -f synthese-<project_name>

Log files are available in the logs/synthese.log file inside the project.

For a list of all commands:

supervisorctl help

The generated Apache configuration

The Apache configuration file is generated from the Synthese configuration when the system_install_prepare or system_install command is run. A few variables can be set to specify a few Apache configuration settings. The main variables can be set in the site configuration file, for instance myproject/sites/www/config.py: (all variables are optional).

See the site configuration chapter above for more details about the Apache configuration settings you can use.

The administration Web interface

The generated Apache configuration (see section above) includes settings to give access to the Synthese Web administration interface. It should be accessible with the address http://your_server/w/admin/.

Technical details

If you use the procedure above with the generated configuration, you can skip this section.

The Web interface is implemented in Python (flask framework). The generated Apache configuration uses mod_wsgi to access the administration interface.

If you don’t want to use the generated Apache configuration above, you can see an example below:

  # WSGI
  WSGIDaemonProcess synthese-demo-www user=synthese group=synthese
  WSGIProcessGroup synthese-demo-www

  WSGIScriptAlias /w /path/to/demo/conf/generated/wsgi/app.wsgi
  <Directory /path/to/demo/conf/generated/wsgi>
      Order allow,deny
      Allow from all
  </Directory>

The configuration above will use the generated app.wsgi file. Here’s an example

import os
from os.path import join
import sys

bootstrap_config = {
  'env_bin_dir': '/home/user/.synthese/env/bin', 'project_path': '/home/user/synthese_projects/projects/demo',
  'env_type': 'cmake', 'mode': 'relwithdebinfo', 'tools_dir': '/home/user/synthese3/tools',
  'env_path': '/home/user/synthese3/build_cmake/relwithdebinfo'}

activate_this = join(bootstrap_config['env_bin_dir'], 'activate_this.py')
execfile(activate_this, dict(__file__=activate_this))

sys.path.append(bootstrap_config['tools_dir'])
import synthesepy.web

application = synthesepy.web.get_application(bootstrap_config)

bgstart and bgstop commands

Command used for starting/stopping the daemon using the configured bg_process_manager setting. These commands are usually used by scripts and the manager interface. How it behaves depend on the value of the “bg_process_manager” project variable (see above for detail).

root_delegate command

Used by the web application. You don’t need to call it directly.

update_synthese and update_project commands

Commands uesd to update synthese and/or the project. They are usually used by scripts and the manager interface.

deploy, deploy_remote_prepare, deploy_remote_restore commands

Commands used for running the deploy.

ineo_install_triggers commands

This commands installs the ineo triggers in the database. You must set the ineo_* parameters.

Development Guide

This chapter gives some information if you want to develop the synthese.py script.

synthese.py script startup

The synthese.py makes use of a Python environment. When you start the synthese.py script, it will automatically create a Python environment if it doesn’t exist yet. See the “Script storage path” section above for the path location.

Code structure

tools/synthese.py: main synthese.py script. This script takes care of creating the Python environment (which loads the Python dependencies) and then runs the tools/synthesepy/cli.py module.

tools/synthesepy/checker: checker tool

tools/synthesepy/checker/static: Static files for the checker tool

tools/synthesepy/checker/templates: Templates for the checker tools

tools/synthesepy/functional_test: Python functional tests

tools/synthesepy/third_party: this directory contains the third party modules used by synthese.py

tools/synthesepy/web/manager: Files for the Manager Web interface
tools/synthesepy/web/manager/static: static files for the Manager interface.
tools/synthesepy/web/manager/templates: templates for the Manager interface

tools/synthesepy/config.py: Default configuration settings
tools/synthesepy/env.py: Environment abstraction class
tools/synthesepy/cli.py: Default configuration settings
tools/synthesepy/project_manager.py: Classes for managing projects

File naming rules: in most cases, each synthese.py command maps to a Python module of the same name (build.py, deploy.py, …)

Dependencies and libraries

synthese.py uses the following Python dependencies:

argparse: module for argument parsing
BeautifulSoup: module for parsing Web Pages (used for the HTTP Api and Unit tests).
flask: Small web framework used by the Manager, checker and dashboard.
mechanize: automated page testing for the unit tests.
Paste: WSGI utilities
requests: library for fetching HTTP resources
static: module for serving static files
tempita: templating module used for config generation
unittest2: module for unit tests.

FAQ and Howto

How to change the versions of the SYNTHESE prerequisites?

Open the tools/synthesepy/build.py file.

If you need to update the MySQL version, change to MYSQL_VER variable. Then, you need to update the md5sum of the archive. See the _install_mysql method in that file.
For spatialite, update the LIBSPATIALITE_DLLS variable.

Why is my configuration change not taken into account in the Manager?

You need to restart Apache if you change your project configuration and want to use the Manager.

Sidebar

SYNTHESE Documentation

Specifications
Developer documentation
Database
API
CMS template syntax
Interoperability
Administrator manual

Developer-documentation

Synthesepy

The synthese.py script

Invoking the script

Script storage path

The script alias

Windows

Linux

Windows with Cygwin

Script modes

Configuration

Example configuration

Using the configuration

Configuration options

Global options

General Mode Commands

build command

Build related general options

build specific options

Other config file only options

Example usage

Build output

clean command

install command

ide command

testshell command

doxygen command

runtests command

runtests specific options

sqlite_to_mysql command

sqlite_to_mysql specific options

create_project command

create_project specific options

Example

continuous_integration (ci) command

continuous_integration specific commands

Other config file only options:

system_install command

package command

Package install script

package specific options

Other config file only options:

checker command

dashboard (d) command

Projects

Introduction

Structure of a project

Creating a new Synthese project

project/*config.py

project main configuration

project/manager/imports_config.py

project/manager/main.py

Sites

Introduction

Configuration

Packages

Introduction

Content

Content detail

files

files_local

pages and pages_local

apache.conf

config.py

*.sql, *.sql.gz

The Manager Interface

Project deployment

The synthese user

Preparing the system for projects

Install the project on the system

Project Mode Commands

rundaemon (run,start) command

rundaemon related general options

load_data and load_local_data command

reset command

stopdaemon (stop) command

runproxy command

runwebapp command

project_command (pc) command

Database related commands

ssh command

.sql, .sql.gz