Semmle 1.19
Skip to end of metadata
Go to start of metadata

This topic describes how to prepare custom queries, which can then be used to analyze your code.

Overview

You can use custom queries to extend the analysis of any project. For example, you may want to write queries to highlight requirements or calculate metrics that are specific to your company or project. You must save custom queries in a queries directory and then compile them before they can be used in any Semmle Core analysis. For information about writing queries, see Learning QL and for more information about the requirements of a query file see Query file requirements

Prerequisites

Your custom queries must have been tested using the qltest command to ensure that results are generated as expected. Custom queries that fail qltest can be debugged using one of the QL plugins or extensions. For more information, see Setting up automated tests for custom queries. Before you can run analysis with custom queries, you must ensure that all query files include a header section that defines the name, description, and type of the query. Specifically, you must define @id and @kind properties for any queries used with the analyzeSnapshot command. Queries that calculate metrics must also define which code element they measure and which forms of aggregation are valid. See Query file requirements for full details of the mandatory and optional properties for query files.

Query directories

You should store custom queries in a directory that is outside the main Semmle Core directory structure. This simplifies upgrading to a new version of Semmle Core. If your configuration files are stored separately from Semmle Core, in a location defined using SEMMLE_HOME, you may want to create a custom-queries directory in this location. You can then use the ${semmle_home} variable to define the location for custom queries. For further information, see Semmle variables. If your configuration files are stored under the main Semmle Core directory structure, you should create your custom-queries directory in a location outside Semmle Core and define queries with absolute paths where necessary. When using any of these directory structures, you should also create a "parent" subdirectory for each language. This directory must contain a queries.xml file that defines the language of the queries stored in the subdirectories.

When preparing custom queries, we also make the following recommendations:

  • If you need to customize a standard query, create a copy of it in your organization's dedicated query directory rather than modifying it in place.
  • In custom queries, always use library classes provided by the standard library, rather than accessing low-level database relations directly. This will make it more likely that your queries will continue to work with new versions of the standard libraries.
  • If you have customized standard queries, when you upgrade you should review any differences between the old and new versions. Then you can apply any fixes that are relevant to your version.
  • Always back up your custom query directory, or, ideally, store it in a version-control system such as Subversion or Git.

To create a directory structure for custom queries
  1. Create a new subdirectory for all custom queries in a suitable location (as discussed above), for example mycompany-queries.
  2. Create a new subdirectory for each programming language that you plan to write queries to analyze. For example, if you are interested only in Java and C# then you might create the following subdirectories:
    cd mycompany-queries
    mkdir java
    mkdir csharp
  3. In each subdirectory for a programming language, create a queries.xml file to define the programming language of queries stored in that directory. Either create the file by hand, or copy a suitable queries.xml file from the standard query subdirectory.

Each queries.xml file contains a single XML element that defines the programming language that queries in the directory are written to analyze. For example:

  • Java queries.xml contains: <queries language="java"/>
  • C# queries.xml contains: <queries language="csharp"/>
 Click to show the language values supported...


  • C or C++ — language="cpp"
  • C# — language="csharp"
  • COBOL—language="cobol"
  • Java — language="java"
  • JavaScript — language="javascript"
  • Python — language="python"

The other supported option is language="odasa-dashboard" which is used to analyze dashboard databases

 

Once you have finished creating the directory structure for custom queries, you should move all custom queries into the appropriate mycompany-queries/language subdirectory.

Compiling queries

The standard queries supplied as part of Semmle Core have already been compiled, so you can use them to analyze your code without further processing. However, when you add your own custom queries you have to compile them using the prepareQueries command. This command can simply be run in the directory containing the new custom queries. For further information, see prepareQueries.

To compile any uncompiled queries

Run the prepareQueries tool in the directory containing the queries that you want to compile. For example:

odasa prepareQueries

For each query queryName.ql stored in the current directory and its subdirectories a compiled query is created in the central query compilation cache (queryName.qlo).

What next?

After the custom queries have been saved in the appropriate directory and compiled, they can be added to a query suite and used in analysis. See Grouping queries and Generating results using command-line tools for further information.