This topic describes how to use the QL command-line tools to prepare snapshots offline to upload to LGTM for further analysis.
odasa). For information about the new CodeQL CLI, that was released with Semmle 1.23, see https://help.semmle.com/codeql/.
To analyze a project on LGTM enterprise, you need to specify any custom build processes required to create a snapshot of your code in an lgtm.yml configuration file. However, if your project has complex build requirements that cannot easily be encapsulated in an lgtm.yml configuration file, you can build it offline using the QL command-line tools. Externally-built snapshots can then be uploaded to LGTM using the LGTM API, along with any data about previous analyses.
Before you can create or export a snapshot of your code, you must create a
project file that defines how to access your source files and build the codebase. There are various ways to create and edit a
project file to include the necessary build processes. For further information, see Advanced project creation.
Once you have created a project, the details in the
project file can be used to generate a snapshot of your code. To learn more about generating snapshots using the QL command-line tools, see Generating a snapshot of your code.
If the version of the QL command-line tools that you used to generate your snapshot is older than the version of LGTM that you want to upload it to, LGTM will try to upgrade and analyze the snapshot when the upload is complete.
You can include cached predicates with the snapshot, but they will be ignored as the cache is repopulated after the snapshot has been upgraded and analyzed. However, if you want to include query results your snapshot, as outlined below, you must make sure the snapshot is compatible before you start the upload session. Snapshots with results generated using older versions of the QL command-line tools won't be upgraded and cannot be uploaded to LGTM.
To display the version number of the QL command-line tools that you are using, simply run
odasa version. If your QL command-line tools are older than your version of LGTM, then download the matching version of the tools before generating any new snapshots. Historic snapshots can be upgraded using the upgrade command.
LGTM query suites
The QL command-line tools support the use of query suites, which allow you to easily reuse queries in analyses across multiple projects. The LGTM query suites are included with the QL command-line tools, so you can run exactly the same queries offline as are run in an analysis by LGTM. If your snapshot is compatible with LGTM, you can include the results of the analysis with your snapshot when you upload it to LGTM–your results will quickly become available to view on LGTM as no queries will have to be rerun. Additionally, you can include the cached predicates that were computed during the offline analysis in your upload. This makes running future queries, for example in the LGTM query console, much faster, as LGTM can reuse the cached data rather than having to compute each predicate from scratch.
- C/C++ LGTM query suite:
- C# LGTM query suite:
- Go LGTM query suite:
- Java LGTM query suite:
- Python LGTM query suite:
$(semmle-home) is the top-level QL command-line tools directory containing the
Running the LGTM query suites with the QL command-line tools
To run the LGTM query suites on a snapshot created using the QL command-line tools, you need to use the
analyzeSnapshot command, specifying the suite using the
where the results format is specified as SARIF v2.1.0. This format matches the results format used by LGTM. The results file,
results.sarif, is output to the current project directory. For more specific information about the SARIF format, see SARIF results file. To learn more about running analyses from the command line, see Generating query results using command-line tools.
Exporting data to use in LGTM
To export a snapshot in the format required by LGTM, use the
export command. If your snapshot is already compatible with LGTM, and you want to include results from analyses already run using the QL command-line tools and cached data for previously computed predicates, you should include the
--keep-cached flags respectively. For example, to export the most recently generated snapshot from the current project directory, including previously computed data in the exported file, you would use:
The snapshot and the additional data are exported into a zip archive in the current directory, which can be uploaded to LGTM. For more detailed information on exporting data using the the QL command-line tools, see Exporting data for QL plugins and extensions.
Uploading snapshots to LGTM
After you have prepared your snapshot using the QL command-line tools, you can upload it to LGTM using the LGTM API. For further information, see About the API for LGTM in the LGTM administrator help.
Upgrading an LGTM snapshot
When LGTM Enterprise is upgraded to a new version, the snapshot database schemas change. This may have an impact on your uploaded snapshots. For further information, see the following sections.
When LGTM Enterprise 1.23 is upgraded to a new version, the snapshot database schema changes. LGTM will upgrade the snapshots in upload analysis mode and and reanalyze them, so that up to date results are available to users. If you want to ensure that any newly supported language features will be included in the database, then you can also create and upload a new snapshot using the latest version of the QL command-line tools, by following the guidelines above.
There are no database schema changes in maintenance releases. For example, you can upgrade from LGTM Enterprise 1.21.0 to 1.21.1, and snapshots won't be upgraded. However, snapshots would be upgraded when upgrading LGTM Enterprise from 1.21.x to 1.22.x.
When LGTM Enterprise 1.22 (or earlier) is upgraded to a new version and the snapshot database schema changes, it's no longer possible for users to download or query the uploaded snapshots. There are two options:
- Create and upload a new snapshot using the guidelines above.
- Upgrade the existing snapshot to the new database schema.
Where possible, the first option gives better results because any newly supported language features will be included in the database. If snapshots are generated and uploaded on a schedule, it's often helpful to upgrade the current snapshot as a temporary measure.
To upgrade a snapshot that you've downloaded from LGTM, use the unexportSnapshot command. This command unzips and automatically upgrades the snapshot. For example:
After this, you'll need to reexport the snapshot (as described above) and then upload it to LGTM.