LGTM Enterprise 1.23.1

Adding a project in upload analysis mode

LGTM provides an upload analysis mode for codebases that you can't build, or don't want to build, using the LGTM worker hosts.

To create a project in upload analysis mode, you must specify the project details and upload one or more CodeQL databases using the LGTM API. Each database represents a snapshot of the codebase at a specific time. For general information about this mode, see Using upload analysis.

Prerequisites

Before creating a project in upload analysis mode using the API, you must:

  • Create an access token with the administrator scopes: projects:write and snapshots:write. The endpoints you need to access require a token with these scopes. For information about generating access tokens, see Managing access tokens in the user help.
  • Generate one CodeQL database for each language you want to analyze. Check that any data included with your database is compatible with the version of LGTM you are using. If your database was generated using an older version of the CodeQL command-line interface (CLI), LGTM will try to upgrade and analyze it when the upload is complete. You can include cached predicates with the database, but they will be ignored as the cache is repopulated after the database has been upgraded and analyzed. However, if you want to include query results with your database, you should make sure it is compatible before you start the upload session. Incompatible databases with results won't be upgraded and cannot be uploaded. For further information on exporting externally-built databases, see Preparing CodeQL databases to use with LGTM Enterprise in the CodeQL CLI help.

Requesting a new project

To create a new project, you need to make a POST request to the /projects API endpoint. This request tells LGTM the repository URL and starts one or more database upload sessions. In addition to your access token, the request must include:

  • The repository URL for the codebase: repository. This must match one of the integrations, with repository addition enabled, defined for the system, otherwise the request will be rejected.
  • A commit identifier: commit. This should identify the version of the code contained in the initial CodeQL database, or databases, that you will upload to create the new project.

You can include the optional language parameter to specify the languages you want to upload. Otherwise upload sessions are created for all supported languages. For full details of the query parameters available, see LGTM API specification in the User help.

There are many ways to make a POST request with an access token. Here we've used Curl to add Maven as a Java project in upload analysis mode. The commit identifier is the SHA of the version used to build the CodeQL database:

curl -X POST '{lgtm-server-url}/api/v1.0/projects?repository=https://github.com/apache/Maven&mode=upload&language=java&commit=58de88f95ca7d6ab93264727d96a1501f7c6e9e9' \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer {access-token}'

You should replace the items in {} with the values for your system. The access token will be a long alphanumeric string, for example, 086ddbf968c84b9efb91f90d0b4f95a23bc9804fc1a416a4e2daca3e9570f498.

If you're using the Windows command shell, you need to replace single quotes with double quotes. You should also use the caret symbol (^) as a line continuation character in place of the backslash symbol (\).

Reviewing the response

The response includes five main sections:

  • id—an operations identifier that's used to monitor the progress of the analysis task. For further information on using the operations endpoint to track the progress of a task, see Tracking progress of the analysis in the user help.
  • uploads—a section with one or more programming languages, each with a database upload URL. This tells you where to upload the externally-generated databases for the project.
  • status—the overall status of the project addition. Initially, this is pending.
  • task-type—the type of task.
  • task-result—the commit identifier from the API request, the current status for each language, and the log-url for an LGTM logs page where you can access information about the project addition.

For example, the POST request above returns the following information:

{
    "id": 1000007,
    "uploads": {
        "java": {
            "id": "5076f11b8da896c06a94069f01d955db6cf0678048e90529f1007f28aa7c86cd",
            "url": "{lgtm-server-url}/api/v1.0/snapshots/uploads/5076f11b8da896c06a94069f01d955db6cf0678048e90529f1007f28aa7c86cd"
        }
    },
    "status": "pending",
    "task-type": "analysis",
    "task-result": {
        "commit-id": "58de88f95ca7d6ab93264727d96a1501f7c6e9e9",
        "languages": [
            {
                "language": "java",
                "status": "pending"
            }
        ],
        "log-url": "{lgtm-server-url}/logs/a839d507c652f229df15bd0bdab64ff639e65a50"
    }
}

Uploading databases for analysis

Upload each database using a PUT request. When the upload is complete, make a POST request to the same URL to close the upload session. For detailed information about uploading databases using these URLs, see Uploading a database.

What happens next?

LGTM processes each CodeQL database by:

  1. Checking whether the database schema for the database is compatible with the CodeQL analysis in LGTM.
  2. Checking whether the database includes results for the LGTM built-in and installation-wide queries.

Compatible databases are accepted for analysis. If results for some queries are present in the uploaded data, they are kept. Any queries where the results aren't present are run on the database.

Incompatible databases without results will be upgraded and LGTM will run all of the built-in and installation-wide queries. If any results are included, the databases will not be upgraded and they will not be added to LGTM.

Once the database for a project is added, the new project is visible to users immediately. They can explore the alerts or run queries on the project in the same way as for other newly added projects.

Checking progress

When you've finished uploading databases for the project, you can monitor the progress of analysis using the log-url included in the responses to the /projects and /snapshots endpoints.

Once the analysis is complete, you can refresh the page to access the log files:

For uploaded databases, the Checkout and Extraction tabs are always empty. The Import tab contains a little data and will report any problems with database compatibility, or if an upgrade is required. The Analysis tab contains details of any queries that were run on the database.

If one or more databases are incompatible, check your LGTM Enterprise version and install the same version of the CodeQL command-line interface. Then you can upgrade the databases and re-try the upload. If the new project was created, use the /snapshots endpoint to start a new upload. If the project creation failed because no databases were compatible, repeat your request to create a new project and use the new upload URLs generated by that request.

Failed responses

If there is a problem with the request to create a project, you may see one of the following responses:

  • 400 error response: indicates a problem with the request URL. The response will normally include a message describing the problem. For example, the repository URL may not match an existing integration, or that repository may already be analyzed by LGTM.
  • 403 error response: indicates that you have a problem with your access token. For more information on access tokens, see Managing access tokens in the user help.

Related topicsRelated topics