Note: This documentation is for the legacy command-line tool odasa.
The final version was released in May 2020. Support for this tool expires in May 2021.

For documentation on the new generation CodeQL CLI, see CodeQL CLI .
In particular, you may find the Notes for legacy QL CLI users useful in planning your migration.

Skip to end of metadata
Go to start of metadata

On this page:

Related topics:

HIDDEN

Purpose

The upgrade tool can be used to upgrade snapshot databases that were created with an earlier version of Semmle to enable you to analyze them using the latest version. The database schema of the snapshot databases is updated so that it is compatible with current version of Semmle analysis.

The upgrade tool makes a "best effort" attempt to upgrade the existing snapshot database to the new schema, however, the results are unlikely to be perfect because extractor and database schema changes are often designed to record additional data. While the upgrade tool can extend the database schema to include the new columns for data, it may not be possible to populate these columns.

We strongly recommend that you test the upgrade tool on a copy of a snapshot database and then compare the results of analyzing this database against the results generated by analyzing the original database. This will help you to identify any artifacts caused by the upgrade.

Usage

This tool is run from the command line as follows:

odasa upgrade [--verbose] [--verbosity <level>] 
 [--project <project_name>] [--concurrent] [--earliest]
 [--latest] [--dry-run] [--upgrade-packs <pack>]
 <snapshot_name>

Flags

The upgrade tool supports the following flags:

FlagsValueExampleNotes
--verbose--

Optional. Output more detailed information about actions. This increases the verbosity to level 4.

Default: level verbosity

--verbosity<level>2

Optional. Define the precise level of reporting required where 0 suppresses all output and 6 reports all levels of detail available. You can use the --verbose flag as shorthand for --verbosity 4 .

Default: 3

--project<path>projects/myproject

Optional, defines the location of the project configuration file to use with the --earliest or --latest flag.

Default: current directory

--concurrent--

Optional, use to allow the concurrent execution of other commands. Use with care.

Default: the snapshot files are locked to other tools until this tool has completed.

--earliest-

Optional, use to upgrade the oldest snapshot for the project configuration defined using the --project flag. Alternatively, define a snapshot to upgrade.

--latest--

Optional, use to upgrade the most recent snapshot for the project configuration defined using the --project flag. Alternatively, define a snapshot to upgrade.

--dry-run--Optional, run the upgrade process and report details to stdout without making any changes to the snapshot database. This is a useful test to run before you upgrade the snapshot.
--upgrade-packs<pack>upgrade/patch-42698

Optional, define one or more additional upgrade packs to run.

Default: the upgrade packs required to upgrade the snapshot to the current version of the command-line tools are run.

 <snapshot_name>revision-2017-March-25--23-10-15Optional, defines the name of the snapshot to upgrade.

Results

This tool runs all the default upgrade steps and those defined in any additional upgrade packs that have been specified. If the --dry-run flag is specified then the effect of running the upgrade steps is reported on stdout but the snapshot database is not changed. If the --dry-run flag is omitted then the snapshot database schema is upgraded as defined by the upgrade steps.

Compatibility

When you run the upgrade command with the --dry-run flag, each step required to perform the upgrade is reported on stdout with details of the expected compatibility that running the step will achieve. At the end the tool reports the expected outcome of running all the steps to upgrade the snapshot. The compatibility will be one of:
  • full - the step is completely safe and will preserve all functionality. The results of analyzing the upgraded snapshot will be identical to the results of analyzing the same version of the code built with the new version of the toolchain. 
  • backwards - the step is safe and preserves the meaning of the old data. There is no reason to rebuild the snapshot with the new version of the toolchain unless a new feature is desired. Adding tables or types for Java 7, where the code base is Java 6, is an example of a backwards compatible upgrade step.
  • partial - the step is safe and preserves the meaning of the old database. However, changes to the extractor mean that you would get better results if you rebuilt the snapshot with the new version of the toolchain. Corrections to the collection and storage of data for a specific element is an example of a partially compatible upgrade step.
  • breaking - the step is unsafe and will prevent certain queries from working. However, upgrading the snapshot database will allow partial usability of the snapshot - that is, it will be possible to run some queries. Examples of upgrade steps that break compatibility include completely changing the data stored by a table, or adding new columns to a table.

If any of the upgrade steps report either partial or breaking compatibility then we recommend that you upgrade a copy of a snapshot and re-analyze this snapshot with the new version of the tool chain. This will enable you to see exactly which queries are affected by the compatibility problems and make a decision about whether to upgrade or rebuild snapshots.

Expected output

When the upgrade steps are run, information about the steps required and how successfully they have been applied is output to stdout. If you want to see more detailed information then you can specify the --verbose flag.

Example output using --dry-run flag
odasa upgrade --project projects\commons-io --latest --dry-run
Unpacking database for commons-io - revision-2016-January-15--13-09-07: ...
Finished unpacking database; about to run analyses
[2017-03-22 12:01:43] Would perform upgrade: Remove `externalDefects` and `externalMetrics` relations
[2017-03-22 12:01:43]   -> Compatibility: full
[2017-03-22 12:01:44] Would perform upgrade: Generalize @typeormethod to @typeorcallable
[2017-03-22 12:01:44]   -> Compatibility: full
[2017-03-22 12:01:44] Would perform upgrade: Improved extraction of generics
[2017-03-22 12:01:44]   -> Compatibility: partial
[2017-03-22 12:01:44] Would perform upgrade: Improved extraction of lambdas and method references
[2017-03-22 12:01:44]   -> Compatibility: partial
[2017-03-22 12:01:44] This would bring the database in line with the latest version.
Deleting temporary files for commons-io - revision-2016-January-15--13-09-07: ...
Done deleting temporary files (all files deleted successfully)

In this example we are considering upgrading a Java snapshot created using Semmle 1.9.8 to be compatible with Semmle 1.12. The key information is:

  • Compatibility: full - these upgrade steps will give full compatibility with Semmle 1.12
    Would perform upgrade: Remove `externalDefects` and `externalMetrics` relations
    Would perform upgrade: Generalize @typeormethod to @typeorcallable

  • Compatibility: partial - these upgrade steps will give only partial compatibility with Semmle 1.12
    Would perform upgrade: Improved extraction of generics
    Would perform upgrade: Improved extraction of lambdas and method references
  • This would bring the database in line with the latest version. - Running this upgrade will make the snapshot database compatible with the latest version of Semmle analysis.

Since the upgrade does not re-build and extract the source code, it is not possible to provide an upgrade step to provide full compatibility for most extractor changes. This means that queries which examine generics, lambdas and method references may contain false negatives or other artifacts. Otherwise, the upgrade should make the snapshot database fully compatible with Semmle 1.12.

Example output for upgrade
Unpacking database for commons-io - revision-2016-January-15--13-09-07: ...
Finished unpacking database; about to run analyses
[2017-03-22 12:06:38] Upgrading: Remove `externalDefects` and `externalMetrics` relations
[2017-03-22 12:06:38]   -> Creating new dbscheme
[2017-03-22 12:06:38]   -> Finalising upgrade...
[2017-03-22 12:06:38]   -> Cleaning up...
[2017-03-22 12:06:38]   -> Upgrade completed.
[2017-03-22 12:06:38] Upgrading: Generalize @typeormethod to @typeorcallable
[2017-03-22 12:06:38]   -> Creating new dbscheme
[2017-03-22 12:06:38]   -> Finalising upgrade...
[2017-03-22 12:06:38]   -> Cleaning up...
[2017-03-22 12:06:38]   -> Upgrade completed.
[2017-03-22 12:06:38] Upgrading: Improved extraction of generics
[2017-03-22 12:06:38]   -> Creating new dbscheme
[2017-03-22 12:06:38]   -> Finalising upgrade...
[2017-03-22 12:06:39]   -> Cleaning up...
[2017-03-22 12:06:39]   -> Upgrade completed.
[2017-03-22 12:06:39] Upgrading: Improved extraction of lambdas and method references
[2017-03-22 12:06:39]   -> Creating new dbscheme
[2017-03-22 12:06:39]   -> Finalising upgrade...
[2017-03-22 12:06:39]   -> Cleaning up...
[2017-03-22 12:06:39]   -> Upgrade completed.
[2017-03-22 12:06:39] Updating toolchain version of snapshot commons-io - revision-2016-January-15--13-09-07 to current version (5 August 2016)
Deleting temporary files for commons-io - revision-2016-January-15--13-09-07: ...
Done deleting temporary files (all files deleted successfully)

No problems have been reported. The snapshot is ready to analyze using Semmle 1.12