Semmle 1.22
Skip to end of metadata
Go to start of metadata

This topic describes how to use the QL command-line tools to prepare snapshots offline to upload to LGTM for further analysis. 

Overview

To analyze a project on LGTM Enterprise, you need to specify any custom build processes required to create a snapshot of your code in an lgtm.yml configuration file. However, if your project has complex build requirements that cannot easily be encapsulated in an lgtm.yml configuration file, you can build it offline using the QL command-line tools. Externally-built snapshots can then be uploaded to LGTM using the LGTM API, along with any data about previous analyses.

When you upgrade LGTM Enterprise, you may also need to replace any snapshots that you've uploaded. That is, upload either a new snapshot you created with the same version of the command-line tools, or an existing snapshot that you've upgraded to the same version. This is not needed when you upgrade to a new maintenance release, but for other releases the associated projects will be unavailable for analysis until new snapshots are uploaded.

Prerequisites

Before you can create or export a snapshot of your code, you must create a project file that defines how to access your source files and build the code base. There are various ways to create and edit a project file to include the necessary build processes. For further information, see Advanced project creation.

Once you have created a project, the details in the project file can be used to generate a snapshot of your code. To learn more about generating snapshots using the QL command-line tools, see Generating a snapshot of your code.

To ensure that the data generated using the QL command-line tools are compatible with your version of LGTM, the version numbers must match. To display the version number of the QL command-line tools that you are using, simply run odasa version. If your QL command-line tools are older than your version of LGTM, then download the matching version of the tools before generating any new snapshots. Historic snapshots can be upgraded using the upgrade command.

LGTM query suites

The QL command-line tools support the use of query suites, which allow you to easily reuse queries in analyses across multiple projects. The LGTM query suites are included with the QL command-line tools, so you can run exactly the same queries offline as are run in an analysis by LGTM. You can then include the results of the analysis with your snapshot when you upload it to LGTM–your results will quickly become available to view on LGTM as no queries will have to be rerun. Additionally, you can include the cached predicates that were computed during the offline analysis in your upload. This makes running future queries, for example in the LGTM query console, much faster, as LGTM can reuse the cached data rather than having to compute each predicate from scratch.

The LGTM query suites can be found at the following locations:
  • C/C++ LGTM query suite: ${semmle_home}/queries/customer/default/cpp-lgtm
  • C# LGTM query suite: ${semmle_home}/queries/customer/default/csharp-lgtm
  • COBOL LGTM query suite: ${semmle_home}/queries/customer/default/cobol-lgtm
  • Java LGTM query suite: ${semmle_home}/queries/customer/default/java-lgtm
  • JavaScript LGTM query suite: ${semmle_home}/queries/customer/default/javascript-lgtm
  • Python LGTM query suite: ${semmle_home}/queries/customer/default/python-lgtm

where $(semmle-home) is the top-level QL command-line tools directory containing the queriesprojects and tools directories.

Running the LGTM query suites with the QL command-line tools

To run the LGTM query suites on a snapshot created using the QL command-line tools, you need to use the analyzeSnapshot command, specifying the suite using the  --suite flag. For example, to run the query suite for JavaScript on the most recently generated snapshot in the current project directory, you would use:

odasa analyzeSnapshot --latest --suite ${semmle_home}/queries/customer/default/javascript-lgtm --output-file results.sarif --format sarifv2.1.0 

where the results format is specified as SARIF v2.1.0. This format matches the results format used by LGTM. The results file, results.sarif, is output to the current project directory. For more specific information about the SARIF format, see SARIF results file. To learn more about running analyses from the command line, see Generating query results using command-line tools.

Exporting data to use in LGTM

To export a snapshot in the format required by LGTM, use the export command. If you want to include results from analyses already run using the QL command-line tools and cached data for previously computed predicates, you should include the --keep-results and --keep-cached flags respectively. For example, to export the most recently generated snapshot from the current project directory, including previously computed data in the exported file, you would use:

odasa export --latest --keep-results --keep-cached

The snapshot and the additional data are exported into a zip archive in the current directory, which can be uploaded to LGTM. For more detailed information on exporting data using the the QL command-line tools, see Exporting data for QL plugins and extensions.

Uploading snapshots to LGTM

After you have prepared your snapshot using the QL command-line tools, you can upload it to LGTM using the LGTM API. For further information, see About the API for LGTM in the LGTM administrator help.

Upgrading an LGTM snapshot

When LGTM Enterprise is upgraded to a new version and the snapshot database schema changes, it's no longer possible for users to download or query the uploaded snapshots. There are two options:

  1. Create and upload a new snapshot using the guidelines above.
  2. Upgrade the existing snapshot to the new database schema.

Where possible, the first option gives better results because any newly supported language features will be included in the database. If snapshots are generated and uploaded on a schedule, it's often helpful to upgrade the current snapshot as a temporary measure.

To upgrade a snapshot that you've downloaded from LGTM, use the unexportSnapshot command. This command unzips and automatically upgrades the snapshot. For example:

odasa unexportSnapshot </path/to/snapshot>

After this, you'll need to reexport the snapshot (as described above) and then upload it to LGTM.

There are no database schema changes in maintenance releases. For example, you can upgrade from LGTM Enterprise 1.21.0 to 1.21.1 without needing to replace snapshots. However, you would need to replace snapshots when upgrading from 1.21.x to 1.22.x.