Semmle 1.22
Skip to end of metadata
Go to start of metadata

This topic describes how to prepare custom queries, which can then be used to analyze your code.

Overview

You can use custom queries to extend the analysis of any project. For example, you may want to write queries to highlight requirements or calculate metrics that are specific to your company or project. You must save custom queries in a queries directory and then compile them before they can be used in any Semmle Core analysis. For information about writing queries, including information on query metadata, see Writing QL queries in the QL help.

Prerequisites

You should test your custom queries using the qltest command to ensure that results are generated as expected. Custom queries that fail qltest can be debugged using one of the QL plugins or extensions. For more information, see Testing custom queries

Your queries must also include the following metadata properties:

  • query identifier (@id): a sequence of words composed of lowercase letters or digits, delimited by / or -, identifying and classifying the query. This property is mandatory for queries used with analyzeSnapshot.
  • query type (@kind): identifies the query is an alert (@kind problem), a path (@kind path-problem), or a metric (@kind metric). This property is mandatory for queries used with prepareQueries and analyzeSnapshot.

Additionally, the results pattern defined in the select statement of your custom query must be consistent with the @kind property. Incorrect results patterns will cause both prepareQueries and analyzeSnapshot to fail. 

For more information on choosing the correct @kind property and results patterns for your query, see Introduction to query files in the QL help.

Query directories

You should store custom queries in a directory that is outside the main Semmle Core directory structure. This simplifies upgrading to a new version of Semmle Core. If your configuration files are stored separately from Semmle Core, in a location defined using SEMMLE_HOME, you may want to create a custom-queries directory in this location. You can then use the ${semmle_home} variable to define the location for custom queries. For further information, see Semmle variables. If your configuration files are stored under the main Semmle Core directory structure, you should create your custom-queries directory in a location outside Semmle Core and define queries with absolute paths where necessary. When using any of these directory structures, you should also create a "parent" subdirectory for each language. This directory must contain a queries.xml file that defines the language of the queries stored in the subdirectories.

When preparing custom queries, we also make the following recommendations:

  • If you need to customize a standard query, create a copy of it in your organization's dedicated query directory rather than modifying it in place.
  • In custom queries, always use library classes provided by the standard library, rather than accessing low-level database relations directly. This will make it more likely that your queries will continue to work with new versions of the standard libraries.
  • If you have customized standard queries, when you upgrade you should review any differences between the old and new versions. Then you can apply any fixes that are relevant to your version.
  • Always back up your custom query directory, or, ideally, store it in a version-control system such as Subversion or Git.

To create a directory structure for custom queries
  1. Create a new subdirectory for all custom queries in a suitable location (as discussed above), for example mycompany-queries.
  2. Create a new subdirectory for each programming language that you plan to write queries to analyze. For example, if you are interested only in Java and C# then you might create the following subdirectories:
    cd mycompany-queries
    mkdir java
    mkdir csharp
  3. In each subdirectory for a programming language, create a queries.xml file to define the programming language of queries stored in that directory. Either create the file by hand, or copy a suitable queries.xml file from the standard query subdirectory.

Each queries.xml file contains a single XML element that defines the programming language that queries in the directory are written to analyze. For example:

  • Java queries.xml contains: <queries language="java"/>
  • C# queries.xml contains: <queries language="csharp"/>
 Click to show the language values supported...


  • C or C++ — language="cpp"
  • C# — language="csharp"
  • COBOL—language="cobol"
  • Java — language="java"
  • JavaScript — language="javascript"
  • Python — language="python"

The other supported option is language="odasa-dashboard" which is used to analyze dashboard databases

 

Once you have finished creating the directory structure for custom queries, you should move all custom queries into the appropriate mycompany-queries/language subdirectory.

Compiling queries

Uncompiled or out of date queries are automatically compiled when you run analyzeSnapshotHowever, you should pre-compile your custom queries if you want to distribute them to several installations, so that the compilation only happens once. The standard queries supplied as part of Semmle Core have already been compiled, so you can use them to analyze your code without further processing. You can compile your custom queries using the prepareQueries command, and specifying their location. 

To compile any uncompiled queries

To compile all queries in your current working directory and all of its subdirectories, you would use

odasa prepareQueries .

where . represents the current folder.

To compile the queries in a specific subdirectory of your current working directory, you would use

odasa prepareQueries ./<folder-1>

For each query queryName.ql found, a compiled query is created in the central query compilation cache (queryName.qlo). For further information, see prepareQueries.

What next?

After the custom queries have been saved in the appropriate directory and compiled, they can be added to a query suite and used in analysis. See Grouping queries and Generating results using command-line tools for further information.