LGTM Enterprise 1.27

Writing custom queries to include in LGTM analysis

It is possible to customize LGTM analysis by including custom queries. These queries need to have a specific form, so that LGTM can recognize and run them.

When you write a custom query to include in LGTM analysis, we recommend that you follow these steps: 

  1. Develop a new query
  2. Convert it to a valid LGTM query
  3. Test your query
  4. Save the query file
  5. Write and save the help for the query
  6. Add the custom query and associated files to LGTM

You can also write CodeQL libraries to import into your custom queries. For information about how to write a valid CodeQL library, see CodeQL library modules.

Develop a new query

You can use any text editor to write a CodeQL query. However, the easiest way to develop a new query is to try it in the query console on LGTM. That way you can experiment with a new query, run the query on multiple projects, view the results and then make further refinements.

For resources and tutorials on writing queries, see Learning CodeQL and CodeQL queries.

Once you've written a query that returns useful results, make sure that it compiles and runs in the current LGTM environment. In particular: 

  • It contains a single select statement.
  • It is compatible with the built-in LGTM CodeQL libraries.

To check that the query works on LGTM, we recommend that you test it by running it on multiple projects in the query console. For more information on the query console user interface and how to use the query console to develop a new query, see What does the query console look like? and Writing, testing and running your own queries, respectively.

Convert it to a valid LGTM query

Add the query metadata

Before including a custom query in LGTM analysis, you need to define extra properties, known as query metadata. This metadata uniquely identifies each query and tells LGTM what to do with the results. It also gives other users information about what the query results mean.

You must add the metadata at the top of the query file, inside a QLDoc comment ( below). A QLDoc comment is a valid CodeQL comment that begins with /** and ends with */. You can find more information about metadata, QLDoc comments and requirements in the Query metadata style guide on GitHub and the Query metadata reference.

Each query that you want to include for custom analysis in LGTM must have a valid query identifier (@id). This is a unique string that LGTM uses to identify a query. You should choose this value carefully, and make sure that you don't change it accidentally!

The query identifier (@id) must be unique within a query pack.

You should also define a meaningful name (@name) and a description (@description) to explain what the query does.

In addition, each query must include the @kind property, which not only defines the query type, but also ensures that query results are interpreted and displayed correctly:

  • @kind problem denotes a query that finds problems in the code and whose results are displayed by LGTM as alerts.
  • @kind path-problem identifies a path query, that is, a query that describes the flow of information between a source and a sink in the code. Results displayed by path queries are displayed by LGTM as alerts
  • @kind metric indicates that the query computes a metric, such as the number of LGTM alert suppressions per project. The results of metric queries are included in BI exports. They're not included in LGTM itself and won't appear in any SARIF downloads.

For metric queries, the following additional properties are mandatory:

  • @metricType—defines the code element that the query acts on. This information is used by client applications.
  • @metricAggregate—defines the allowable aggregations for the metric.

We recommend that you define tags (@tags) for the query. Tags group queries together in categories.

For alert queries, we suggest you also define the severity (@problem.severity) and precision (@precision) of the query.

Severity and tags are useful for searching for queries and filtering alerts (for alert queries) in LGTM.

The screenshot below shows a built-in alert query opened in the query console:

Query in query console

QLDoc comment, containing the metadata for the query.

select statement. See Check the 'select' statement below.

Element of the select statement.

String of the select statement.

You can see more examples of query metadata by opening a built-in LGTM query in the query console. (Search for a query, display the query help page, and click Open in query console.)

You can also add custom metadata to your custom queries. You may find this useful if you want to reuse data that's specific to your organization or to a particular project in a third-party application.

In LGTM, you can export the alerts displayed on screen to a SARIF file. Query metadata, including custom metatada, will be exported during SARIF export, for alerts that match the currently selected filters. For more information, see Exporting alerts as SARIF.

Check the 'select' statement

The information contained in the results of a query is controlled by the query's select statement ( in screenshot above).

A valid alert query on LGTM selects:

  • Element ( in screenshot above)—a code element that's identified by the query. This defines the location of the alert.
  • String ( in screenshot above)—a message to display for this code element, describing why the alert was generated.

For more information about how to write an appropriate select statement for an alert query, see Defining the results of a query.

A valid path query on LGTM selects:

  • Element—a code element that's identified by the query. This defines the location of the alert.
  • Source— the location the data comes from.
  • Sink—the location where the data ends. (Data flows from the source to the sink.)
  • String—a message to display for this code element, describing why the alert was generated.

For more information about how to write an appropriate select statement for a path query, see Constructing a path query.

A valid metric query on LGTM selects:

  • Element—the object for which the metric was calculated, that is, a file or method.
  • Number—the associated numeric value.

Query checks, which occur whenever a query pack is added to LGTM, or updated, will fail if the select statement isn't consistent with the @kind property. You can investigate the reason for failure by exploring the logs in the Logs page of the Queries tab.

Test your query

To check that the query works, it's best that you test it by running it on multiple projects in the query console. If you test your query on a single project, it may not return useful results on projects that use different frameworks or coding conventions to the one that you originally tested your query on:

  1. In LGTM, open the query console by clicking Query console on the navigation bar at the top.
  2. Select a programming language. The Query tab opens.
  3. Overwrite the few lines of code on that tab (these are meant to provide you with a starting point), apart from the import statement, with the contents of your text editor. LGTM compiles the query.
  4. Click Select projects to query.
  5. Select the projects to run the query on.
  6. Click Run. After a short while, the names of the projects you are analyzing are listed on the page—each with a progress bar. Hover over the progress bar to see the percentage of the analysis that is complete. The results are displayed below the query as analysis completes.
  7. Click View results ( below) or scroll down the page. The View as: Alert control adjacent to a project ( below) indicates that results were found for that project.

Exploring results in query console

Save the query file

Save the query as a .ql file: copy the contents of the query console into a text file and save it with a .ql extension.

Similarly, you should save any CodeQL libraries you write as text files with a .qll extension.

Write and save the help for the query

We recommend you write a help file for every custom query you generate. The help file provides detailed information about the purpose and use of the query. If you create a custom query help file and add it to LGTM, the information will be available to users within your organization as part of the integrated help.

Query help files are written using a custom XML format, and stored in a file with a .qhelp extension. They must have the same base name as the query file (.ql) they relate to.

For information on how to write help for queries, see How do I write help for a custom query? For details on how to access the query help in LGTM and for an example of query help displayed in LGTM, see Query help.

Add the custom query and associated files to LGTM

Once the custom query is ready, you need to add it, as well as its associated files—help file (.qhelp) and library files (.qll)—to LGTM so that it can be run alongside the built-in queries, and the query help can be displayed in LGTM.

For details on how to do this, refer to How do I add custom queries to LGTM?

More information

As well as running your custom queries on your projects in LGTM, you can contribute to the built-in queries in our open source repository on GitHub. For more information on how to contribute CodeQL libraries, see Contributing to CodeQL.