Semmle documentation

Quick links

In this space

Page tree
Semmle 1.20
Skip to end of metadata
Go to start of metadata

This topic describes how to upgrade a deployment of Enterprise Insight to use a newer version of Semmle Core.

Overview

New versions of Semmle Core are released regularly. Each release contains improvements to the extractors used to build snapshot databases, the queries used to analyze snapshots and the queries used to load this data into an Enterprise Insight database. Many of these improvements are the direct result of customer feedback. To benefit from the improvements, you need to upgrade Enterprise Insight to use the new version of Semmle Core.

1. Preparing to upgrade

It is important that, before you upgrade Semmle Core on the machines that publish data to Enterprise Insight, you first upgrade the Insight server. This ensures that all data published to the Insight server can be processed correctly.

-

Although the Insight server will accept and process data collected by earlier versions of Semmle Core, all systems should be upgraded to the same version of Semmle Core as soon as practical to ensure uniformity of data collection methods.

-

Before upgrading your system, you should back up:

  • All Insight server configuration files
  • All custom queries and custom scripts
  • All Enterprise Insight databases

2. Upgrading Semmle Core

Upgrade Semmle Core on the Insight server by following the standard upgrade instructions (Upgrading to a new version of Semmle Core). 

3. Upgrading the databases

If you are upgrading from 1.9.7, 1.9.8 or 1.9.9 you must upgrade all Enterprise Insight databases to the latest schema. If you are upgrading from an older version, you should discuss the upgrade with Semmle support.

There have been no changes to the Enterprise Insight database schema since version 1.10, so you do not need to perform a database upgrade if you are upgrading your Semmle installation from 1.10 or later. However, you must make sure to use the most recent version of Insight server after upgrading Semmle Core. The process for doing so is described below.

SSH-based Insight servers

Perform the following steps for each Insight server that accepts connections over SSH.

SSH-based Insight server: database upgrade
  1. Stop the Insight server by running:
    odasa insightClient --stop <workspace-directory>

  2. If you are upgrading from Semmle releases 1.9.7, 1.9.8 or 1.9.9 , perform the following steps:

     Click here for details ...
    1. Open the Insight server configuration file for editing. It is stored in the workspace directory of the Insight server service and is named insight-server.xml.

    2. Between the connection and datamodel elements, insert the following elements:

      Excerpt from Insight Server configuration file
        <upgrade-directory>${semmle_dist}/queries/semmle-insight-queries/sql/enterprise/upgrades</upgrade-directory>
  <databaseconfig src="${semmle_dist}/templates/enterprise-insight/database.xml" />

      These define:

      1. upgrade-directory—the location of the upgrade directory in the Semmle distribution.
      2. databaseconfig—the location of the database schema for the latest version of Semmle Core.
    3. Save the updated file.
  3. The 1.12 release introduced changes to the datamodel element of the Insight server configuration file. Open the Insight server configuration file (in the workspace directory of the Insight server service) and check the value of the datamodel element. If it references the datamodel definition in the Semmle Core distribution, for example:
    Excerpt from Insight Server configuration file
    <datamodel src="${semmle_dist}/templates/[team|enterprise]-insight/datamodel.xml" />

    then the datamodel has already been updated when you upgraded Semmle Core in section 2 above. In this case you can continue with the next step.

    However, if the value of the datamodel element refers to a file in a different location, you will have to update that file to the new format introduced in release 1.12. If the file that is referred to is an unchanged copy of the Insight server configuration file that came with the previous Semmle Core distribution, replace that file with a copy of the new datamodel.xml file, which you can find in the ${semmle_dist}/templates/[team|enterprise]-insight directory.

    If the referenced datamodel.xml file has been customized, you must transfer the modifications to a copy of the new version of the file. In particular, each transaction element now needs a name attribute and a constraint element as its first child element. For more information, see Scheduling database transactions.

  4. Do one of the following:  

    • If you are upgrading from release 1.10 or later, use the insightServer command to restart the SSH service for the Insight server:

      odasa insightServer < workspace-directory>

    • If you are upgrading from Semmle releases 1.9.7, 1.9.8 or 1.9.9, use the insightServer command with the --upgrade flag to upgrade the database, using the information defined by the  upgrade-directory  and databaseconfig  elements:

      odasa insightServer --upgrade < workspace-directory>

      You can check the server.log  file stored in the log subdirectory of the Insight server's workspace directory for details of the changes that were made.

Semmle recommends upgrading manually, using insightServer  command with the  --upgrade  flag. However, if you would prefer upgrades to be run automatically, you can enable the autoupgrade feature described for HTTPS-based servers below.

HTTPS-based Insight servers

Databases for HTTPS-based Insight servers are upgraded using the autoupgrade functionality introduced in release 1.9.10. The following process includes steps for enabling autograde if this has not previously been done:

HTTPS-based Insight server: database upgrade
  1. Stop the Insight server.

  2. Replace the insight-server.war file in your Web Services container (for example, Tomcat) with the new version of this file in the $SEMMLE_DIST/tools directory of the new release.

  3. If you are upgrading from Semmle releases 1.9.7, 1.9.8 or 1.9.9—or if you did not enable autoupgrade in later releases:

     Click here for details ...

    Edit each configuration file for the database(s) maintained by this Insight server. These files are stored in the workspace directory and are XML files with a root element of insight-server. For each file:

    1. Between the  connection  and  datamodel  elements, insert the following elements, if these are not already included:

      Excerpt from Insight server configuration file
        <upgrade-directory>${semmle_dist}/queries/semmle-insight-queries/sql/team/default/upgrades</upgrade-directory>
  <autoupgrade>true</autoupgrade>
  <databaseconfig src="${semmle_dist}/templates/team-insight/database.xml" />

      These define:

      • upgrade-directory—the location of the upgrade directory in the Semmle distribution.
      • autoupgrade—automatic upgrades are enabled.
      • databaseconfig—the location of the database schema for the latest version of Semmle Core.
    2. Save the updated Insight server configuration files.

  4. The 1.12 release introduced changes to the datamodel element of the Insight server configuration file. Open the Insight server configuration file (in the workspace directory of the Insight server service) and check the value of the datamodel element. If it references the datamodel definition in the Semmle Core distribution, for example:
    Excerpt from Insight Server configuration file
    <datamodel src="${semmle_dist}/templates/[team|enterprise]-insight/datamodel.xml" />

    then the datamodel has already been updated when you upgraded Semmle Core in section 2 above. In this case you can continue with the next step.

    However, if the value of the datamodel element refers to a file in a different location, you will have to update that file to the new format introduced in release 1.12. If the file that is referred to is an unchanged copy of the Insight server configuration file that came with the previous Semmle Core distribution, replace that file with a copy of the new datamodel.xml file, which you can find in the ${semmle_dist}/templates/[team|enterprise]-insight directory.

    If the referenced datamodel.xml file has been customized, you must transfer the modifications to a copy of the new version of the file. In particular, each transaction element now needs a name attribute and a constraint element as its first child element. For more information, see Scheduling database transactions.

  5. Restart the Insight server. 

    Depending on your Java Web Services container, the Insight server might only be started once a client accesses that service. In this case you can use insightHttpClient to upload an empty zip file to trigger this process.

    When the Insight server restarts, the upgrade-directory is checked and any new upgrade steps are implemented during the startup process. 

4. Upgrade other servers

When you have upgraded the Insight server, you are ready to upgrade Semmle Core on the servers that publish data to the Insight server. You can upgrade Semmle Core on these servers by following the standard upgrade instructions (Upgrading to a new version of Semmle Core). 

Open the template ei-client.xml file included in the odasa/templates/enterprise-insight directory of the new version of Semmle Core. For each server that publishes data to the Insight server, check whether the ei-client.xml files used by projects analyzed on that server need to be updated. For example, the update may include a new dashboard query. If so, update the file to include the new dashboard query.

 

Copyright © 2019 Semmle Ltd