Note: This documentation is for the legacy command-line tool odasa.
The final version was released in May 2020. Support for this tool expires in May 2021.

For documentation on the new generation CodeQL CLI, see CodeQL CLI .
In particular, you may find the Notes for legacy QL CLI users useful in planning your migration.

Skip to end of metadata
Go to start of metadata

This topic describes how to use the QL command-line tools to prepare snapshots offline to upload to LGTM for further analysis. 

This topic describes the QL command-line tools (odasa). For information about the new CodeQL CLI, that was released with Semmle 1.23, see https://help.semmle.com/codeql/.

Overview

To analyze a project on LGTM enterprise, you need to specify any custom build processes required to create a snapshot of your code in an lgtm.yml configuration file. However, if your project has complex build requirements that cannot easily be encapsulated in an lgtm.yml configuration file, you can build it offline using the QL command-line tools. Externally-built snapshots can then be uploaded to LGTM using the LGTM API, along with any data about previous analyses.

Prerequisites

Before you can create or export a snapshot of your code, you must create a project file that defines how to access your source files and build the codebase. There are various ways to create and edit a project file to include the necessary build processes. For further information, see Advanced project creation.

Once you have created a project, the details in the project file can be used to generate a snapshot of your code. To learn more about generating snapshots using the QL command-line tools, see Generating a snapshot of your code.

Snapshot compatibility

If the version of the QL command-line tools that you used to generate your snapshot is older than the version of LGTM that you want to upload it to, LGTM will try to upgrade and analyze the snapshot when the upload is complete.

You can include cached predicates with the snapshot, but they will be ignored as the cache is repopulated after the snapshot has been upgraded and analyzed. However, if you want to include query results your snapshot, as outlined below, you must make sure the snapshot is compatible before you start the upload session. Snapshots with results generated using older versions of the QL command-line tools won't be upgraded and cannot be uploaded to LGTM. 

To display the version number of the QL command-line tools that you are using, simply run odasa version. If your QL command-line tools are older than your version of LGTM, then download the matching version of the tools before generating any new snapshots. Historic snapshots can be upgraded using the upgrade command.

LGTM query suites

The QL command-line tools support the use of query suites, which allow you to easily reuse queries in analyses across multiple projects. The LGTM query suites are included with the QL command-line tools, so you can run exactly the same queries offline as are run in an analysis by LGTM. If your snapshot is compatible with LGTM, you can include the results of the analysis with your snapshot when you upload it to LGTM–your results will quickly become available to view on LGTM as no queries will have to be rerun. Additionally, you can include the cached predicates that were computed during the offline analysis in your upload. This makes running future queries, for example in the LGTM query console, much faster, as LGTM can reuse the cached data rather than having to compute each predicate from scratch.

The LGTM query suites can be found at the following locations:
  • C/C++ LGTM query suite: ${semmle_home}/queries/customer/default/cpp-lgtm
  • C# LGTM query suite: ${semmle_home}/queries/customer/default/csharp-lgtm
  • Go LGTM query suite: ${semmle_home}/language-packs/go/config/suites/lgtm
  • Java LGTM query suite:  ${semmle_home}/queries/customer/default/java-lgtm
  • JavaScript LGTM query suite: ${semmle_home}/queries/customer/default/javascript-lgtm
  • Python LGTM query suite: ${semmle_home}/queries/customer/default/python-lgtm

where $(semmle-home) is the top-level QL command-line tools directory containing the queriesprojects and tools directories.

Running the LGTM query suites with the QL command-line tools

To run the LGTM query suites on a snapshot created using the QL command-line tools, you need to use the analyzeSnapshot command, specifying the suite using the  --suite flag. For example, to run the query suite for JavaScript on the most recently generated snapshot in the current project directory, you would use:

odasa analyzeSnapshot --latest --suite ${semmle_home}/queries/customer/default/javascript-lgtm --output-file results.sarif --format sarifv2.1.0 

where the results format is specified as SARIF v2.1.0. This format matches the results format used by LGTM. The results file, results.sarif, is output to the current project directory. For more specific information about the SARIF format, see SARIF results file. To learn more about running analyses from the command line, see Generating query results using command-line tools.

Exporting data to use in LGTM

To export a snapshot in the format required by LGTM, use the export command. If your snapshot is already compatible with LGTM, and you want to include results from analyses already run using the QL command-line tools and cached data for previously computed predicates, you should include the --keep-results and --keep-cached flags respectively. For example, to export the most recently generated snapshot from the current project directory, including previously computed data in the exported file, you would use:

odasa export --latest --keep-results --keep-cached

The snapshot and the additional data are exported into a zip archive in the current directory, which can be uploaded to LGTM. For more detailed information on exporting data using the the QL command-line tools, see Exporting data for QL plugins and extensions.

Uploading snapshots to LGTM

After you have prepared your snapshot using the QL command-line tools, you can upload it to LGTM using the LGTM API. For further information, see About the API for LGTM in the LGTM administrator help.

Upgrading an LGTM snapshot

When LGTM Enterprise is upgraded to a new version, the snapshot database schemas change. This may have an impact on your uploaded snapshots. For further information, see the following sections.


 For LGTM Enterprise 1.23, or later

When LGTM Enterprise 1.23 is upgraded to a new version, the snapshot database schema changes. LGTM will upgrade the snapshots in upload analysis mode and and reanalyze them, so that up to date results are available to users. If you want to ensure that any newly supported language features will be included in the database, then you can also create and upload a new snapshot using the latest version of the QL command-line tools, by following the guidelines above. 

There are no database schema changes in maintenance releases. For example, you can upgrade from LGTM Enterprise 1.21.0 to 1.21.1, and snapshots won't be upgraded. However, snapshots would be upgraded when upgrading LGTM Enterprise from 1.21.x to 1.22.x.


 For LGTM Enterprise 1.22, or earlier

When LGTM Enterprise 1.22 (or earlier) is upgraded to a new version and the snapshot database schema changes, it's no longer possible for users to download or query the uploaded snapshots. There are two options:

  1. Create and upload a new snapshot using the guidelines above.
  2. Upgrade the existing snapshot to the new database schema.

Where possible, the first option gives better results because any newly supported language features will be included in the database. If snapshots are generated and uploaded on a schedule, it's often helpful to upgrade the current snapshot as a temporary measure.

To upgrade a snapshot that you've downloaded from LGTM, use the unexportSnapshot command. This command unzips and automatically upgrades the snapshot. For example:

odasa unexportSnapshot </path/to/snapshot>

After this, you'll need to reexport the snapshot (as described above) and then upload it to LGTM.