Semmle 1.22
Skip to end of metadata
Go to start of metadata

On this page:

Related topics:

HIDDEN

This topic describes the options available to configure Semmle Core for a large scale deployment.

Basic setup

By default, Semmle Core assumes that all files used and generated by Semmle Core are stored within a single directory structure under a top-level odasa directory in Semmle Core. When you run the setup.sh (Linux and Apple OS X) or setup.bat (Windows) script, the odasa command is put on the PATH and the environment variable SEMMLE_DIST is defined as the directory that the script was run from. All configuration, license files and output data files are assumed to be stored in this location. The optional environment variables SEMMLE_HOME and SEMMLE_DATA are unset and fall back SEMMLE_DATA -> SEMMLE_HOME -> SEMMLE_DIST.

The legacy ODASA_HOME environment variable is also set to the same location as SEMMLE_DIST to provide backwards compatibility for configuration files and scripts. 

Split deployment

You can set a specific path for one or both of SEMMLE_HOME and SEMMLE_DATA to store different types of files in different locations. Set:

  • SEMMLE_HOME only to separate the configuration, license files, and output data files from the Semmle Core distribution files
  • SEMMLE_HOME and SEMMLE_DATA to define separate locations for configuration and license files (SEMMLE_HOME), output data files (SEMMLE_DATA), and Semmle Core distribution files

SEMMLE_DIST always defines the location of the distribution files. Using these optional environment variables to separate distribution files from other types of file is particularly recommended where you want to:

  • Simplify upgrades to Semmle Core
  • Separate out the configuration files to simplify backing-up or storage in a version control system
  • Store the data files separately
  • Set up a large-scale deployment across multiple servers

The locations defined by SEMMLE_DATA and SEMMLE_HOME must always be writable by Semmle Core commands. From Semmle 1.10 onward, if you set SEMMLE_HOME, then the location defined by SEMMLE_DIST contains only the Semmle Core installation files and may be read-only.

When all three environment variables are set the locations for different types of files are as follows:

Type of filesLocationContains

Distribution

SEMMLE_DIST/docsSome help text used in the query help files
SEMMLE_DIST/queriesRules and associated help files
SEMMLE_DIST/templates Project and dashboard templates
SEMMLE_DIST/toolsodasa command-line tools

Configuration

(and license)

SEMMLE_HOME/dashboardsOne sub-directory per dashboard, each containing a Configuration file and an optional variables file
SEMMLE_HOME/license Semmle license, unless a specific location is set using SEMMLE_LICENSE_DIR
SEMMLE_HOME/projects One sub-directory per project, each containing a project file and an optional variables file

Output data

 

SEMMLE_DATA/dashboardsOne sub-directory per dashboard, each containing a state and an optional deployed folder
SEMMLE_DATA/projects One sub-directory per project, each containing snapshot folders. These contain a snapshot and variables file, with external, log, output, stc and working sub-directories

Using the optional environment variables

The SEMMLE_[DIST/HOME/DATAenvironment variables can be set in the command shell in the same way as any other environment variables. See Standard set up—easily maintained in the Installing Semmle Core topic for information about storing all configuration and data files outside the odasa directory, in a single location.

Migrating from a basic to a standard deployment

You want to migrate a system where all files are stored under the odasa directory, to a standard set up where configuration and data files are stored in a separate location. This makes upgrading to a new version of Semmle Core simpler and more robust. 

To set up and use a separate location for distribution files
  1. Create an odasa directory in a new location. 
  2. Move the Semmle Core distribution files to the new directory, that is, the  docsqueriestemplates, and tools directories.
  3. Update any scripts that you use to set up the Semmle environment so that they set:
    • SEMMLE_DIST to the new odasa directory, that is, location of the distribution files.
    • SEMMLE_HOME to the old location, where the dashboards, license, and projects directories are stored. The license directory may not be present as this location can be defined using the SEMMLE_LICENSE_DIR environment variable.
  4. If your organization uses any custom queries:
    1. Move the queries from the SEMMLE_DIST/queries directory to a sub-directory of the location defined by SEMMLE_HOME
    2. Ensure that the queries for each programming language are stored under a "parent" directory that contains a queries.xml that defines the language to analyze.
    3. If any of the queries has an associated .qhelp file stored in the SEMMLE_DIST/docs/queries directory, move the file into the directory where the custom query is stored. From Semmle 1.10 onward, queries and qhelp files are stored in the same directory.
    4. Update the path for custom queries in all configuration files where they are used.

See Upgrading to a new version of Semmle Core for information about upgrading a a split distribution.

Migrating from a standard to a large-scale deployment

You can store configuration files separately from the output data files. This makes it easy to back up or apply version control to just the configuration files.

To set up and use a separate location for configuration files
  1. Create a directory in the new location for configuration files. 
  2. Update any scripts that you use to set up the Semmle environment so that they set:
    • SEMMLE_HOME to the new top-level directory created in step 1.
    • SEMMLE_DATA to the location where configuration and data files are currently stored.
  3. In the new SEMMLE_HOME directory, create sub-directories called: dashboards and projects.
  4. Move any custom queries and their associated .qhelp files to a sub-directory of SEMMLE_HOME
  5. For each dashboard in the installation:
    1. Create a sub-directory of the same name in the new dashboards directory.
    2. Move the Configuration file, variables file (where present), and dashboard-settings folder for the dashboard into the new sub-directory.
    3. Review all Configuration files and update any relative paths. Use variables to refer to data or distribution files.
    4. Check that any custom queries are correctly defined.
  6. For each project in the installation:
    1. Create a sub-directory of the same name in the new projects directory.
    2. Move the project and variables files for the project into the new sub-directory.
    3. Check whether project files contain <source-location> elements. These elements define absolute paths so must be changed if you move the configuration file.
  7. If the Semmle license file is stored in SEMMLE_DATA/license, move the directory to SEMMLE_HOME/license. (If this directory is absent, your scripts probably set the SEMMLE_LICENSE_DIR environment variable to define an alternative location for the license file.

You can then add the top-level directory for the configuration files to your standard back-up process and/or to your version control system. You may also wish to move the data directory to a new location.

Checking the environment variables

You can check the settings of your environment variables by running:

odasa selfTest

This command displays the current assignment of the SEMMLE_DISTSEMMLE_HOME and SEMMLE_DATA variables. Further information about your Semmle Core installation can be displayed by supplying the appropriate flags. For further information see selfTest.

Large-scale deployments on multiple servers

If you're setting up a large-scale deployment, we recommend that you split the files that make up Semmle Core into the three different locations discussed above. This gives you flexibility over how each type of file is stored and backed up. In addition, it simplifies the process of testing a new distribution in parallel with the installed version before deploying it to production. You can run both versions of Semmle Core using a single set of configuration files stored under SEMMLE_HOME.

For example, you could set SEMMLE_DIST to the location of the new distribution and SEMMLE_DATA to a new data directory (with sub-directories for the projects that you want to analyze). Then set SEMMLE_HOME to the standard location, run the new setup script and run analysis using the new distribution. To run the same analysis using the old distribution, simply set the SEMMLE_[DIST/HOME/DATA] variables to the normal values, run the old set up script and run the same commands.