LGTM Enterprise 1.23.1

Adding externally-built CodeQL databases to LGTM

The API provides a method for LGTM administrators to upload externally-built CodeQL databases to LGTM. You can do this with the /snapshots endpoint, as each CodeQL database represents a snapshot of the codebase at a specific time.

Adding databases using this method is useful for projects that are easier to build offline using the CodeQL command-line interface, due to their complex build requirements. You can only add databases to existing LGTM projects that are configured for upload analysis mode, or while adding a project to LGTM using the API. For more information on adding projects to LGTM using the API, see Adding a project in upload analysis mode.

Prerequisites

Before uploading an externally-built database using the API, you must:

  • Create an access token with the snapshots:write scope. The endpoints you need to access during the upload process all require a token with this scope. For information about generating access tokens, see Managing access tokens in the user help.
  • Set the project to run in upload analysis mode. This means that LGTM will only attempt to analyze the project when a database is uploaded, and no other scheduled analyses will take place. For more information, see Adding a project in upload analysis mode, or Configuring project analysis.
  • Check that analysis of the database language is enabled on LGTM. You can see which languages are available for analysis on the Project administration page. For more information, see Reviewing project details.
  • Check that any data included with your database is compatible with the version of LGTM you are using. If your database was generated using an older version of the CodeQL command-line interface (CLI), LGTM will try to upgrade and analyze it when the upload is complete. You can include cached predicates with the database, but they will be ignored as the cache is repopulated after the database has been upgraded and analyzed. However, if you want to include query results with your database, you should make sure it is compatible before you start the upload session. Incompatible databases with results won't be upgraded and cannot be uploaded. For further information on exporting externally-built databases, see Preparing CodeQL databases to use with LGTM Enterprise in the CodeQL CLI help.

When you upgrade LGTM Enterprise, externally-built databases, created using older versions of the CodeQL CLI, will be upgraded so that they are compatible with your version of LGTM.

Starting the upload session

To upload an externally-built database to an existing project on LGTM, you must first start a database upload session by making a POST request to the /snapshots/{project-id}/{language} endpoint. In addition to your access token, your request must include:

  • The numeric identifier for the project: project-id.
  • The language that the database is written in: language. This language must already be configured for the project on LGTM.
  • An external identifier for the commit to analyze: commit.
  • Optionally, you may include the date.

For more information about finding the project-id and the enabled languages in the administrator interface, see Reviewing project details.

To start an upload session for a JavaScript database for the project with the identifier 100001, you would use the following Curl command:

curl -X POST '{lgtm-server-url}/api/v1.0/snapshots/100001/javascript?commit=123456' \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer 086ddbf968c84b9efb91f90d0b4f95a23bc9804fc1a416a4e2daca3e9570f498'

If successful, the response will include an id for the upload session and a URL that you use to upload your database to LGTM.

Alternatively, a database upload session is started automatically when you make a request to add a project in upload analysis mode using the API. For more information, see Adding a project in upload analysis mode.

If you're using the Windows command shell, you need to replace single quotes with double quotes. You should also use the caret symbol (^) as a line continuation character in place of the backslash symbol (\).

Uploading a database

After you have started an upload session, you can upload an exported CodeQL database to LGTM by making a PUT request to the upload URL. This URL, returned by your initial POST request, corresponds to the /snapshots/uploads/{session-id} endpoint.

It is important to ensure that your externally-built CodeQL databases have been successfully exported and are for the correct language before you start the upload process. Databases that have not been exported correctly, or are the wrong language may still be uploaded, but cannot be analyzed properly.

To upload database.zip to the upload URL returned in the example response above, you would use the following Curl command:

curl -X PUT '{lgtm-server-url}/api/v1.0/snapshots/uploads/1d17a5ab55baa84f9b2a0fc8105cbda9056d04f429aa95dd9e8a43631801481d' \
  --data-binary @database.zip \
  -H 'Content-Type: application/zip' \
  -H 'Authorization: Bearer 086ddbf968c84b9efb91f90d0b4f95a23bc9804fc1a416a4e2daca3e9570f498'
To avoid time out problems, you may want to reduce the size of the file in your PUT request. For example, you could use the split command to divide a large database into several smaller parts to be uploaded separately. Database parts must be > 5MB, except for the final part. You should make a series of PUT requests to upload the parts in order, and wait for each part to finish uploading before making the next upload request.

Completing the upload session

After uploading the final database part, you need to complete the upload session in order to start the analysis. To do this, send a POST request to the database upload URL. For example:

curl -X POST '{lgtm-server-url}/api/v1.0/snapshots/uploads/1d17a5ab55baa84f9b2a0fc8105cbda9056d04f429aa95dd9e8a43631801481d' 
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer 086ddbf968c84b9efb91f90d0b4f95a23bc9804fc1a416a4e2daca3e9570f498'

This request tells LGTM that all parts have been uploaded and that it is ready for analysis. A successful response includes:

  • The status of the analysis, which is initially pending.
  • The operations id, which can be used to monitor the progress of the analysis task. For further information on using the operations endpoint to track the progress of a task, see Tracking progress of the analysis in the user help.
  • The task-result field, which includes information about the analysis task, including an analysis id that can be used to view the analysis results. For more information about using the analyses endpoint to view analysis results, see Viewing the results in the user help
  • The log-url, used to view the analysis logs in your browser. If the analysis fails, the reason for failure will be reported here. For more information, see Using the log viewer.
  • If the analysis fails, the uploaded database is not retained. To reanalyze the database, you have to restart the upload process.
  • The results-url, used to view the full analysis results in your browser. The page is initially empty, but when the analysis completes successfully, it displays the files that triggered alerts in this version of the code. For more information, see Using the file browser.

Canceling a database upload

You can cancel a database upload session by sending a DELETE request to the upload URL. For example, to cancel the session started using the example POST request above, you would use the following Curl command:

curl -X DELETE'{lgtm-server-url}/api/v1.0/snapshots/uploads/1d17a5ab55baa84f9b2a0fc8105cbda9056d04f429aa95dd9e8a43631801481d' 
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer 086ddbf968c84b9efb91f90d0b4f95a23bc9804fc1a416a4e2daca3e9570f498'

You cannot restart an upload session after a successful DELETE request has been made, and any database parts that were already successfully uploaded are deleted. If you make the DELETE request while a PUT request is still in progress, the database part may not be deleted correctly. Therefore, to ensure all data is successfully deleted, you should make the DELETE request after all PUT requests have completed.

Failed responses

If there is a problem with any of your requests, you may see one of the following responses:

  • 400 error response: indicates that the project specified in your POST request to start an upload session is not in upload analysis mode, or that the specified language is not available for that project. For more information about checking analysis settings, see Reviewing project details.
  • 403 error response: indicates that you have a problem with your access token. For more information on access tokens, see Managing access tokens in the user help.
  • 500 error response: indicates that there was a problem completing the database upload, for example a database part was too small. If your database parts are the correct size, try repeating the POST request to complete the database upload session. If the error persists, cancel the current upload session and start a new one.

More information

For full details of the API, see the LGTM API specification.