Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space SDmaster and version 1.24
Dont print
Panel
borderColor#39B54A
bgColor#ffffff
titleColor#39B54A
titleBGColor#ffffff

On this page:

Table of Contents
maxLevel2

Related topics:

Excerpt
hiddentrue

Semmle Core: run queries on a Semmle database from the command line. Particularly useful as part of a script to generate CSV reports or if you want to test a new query for a system before you add it to the dashboard configuration.

Purpose

The runQuery tool is used to run queries from the command line. It is particularly useful as part of a script to generate CSV or SARIF reports, or if you want to test a new query for a system before you add it to LGTM.

Usage

This tool is run from the command line using one of the following modes:

  • odasa runQuery --query <query-filename> --snapshot <snapshot> – to run a query against a snapshot database.
  • odasa runQuery --query <query-filename> --dashboard <dashboard> – to run a query against a dashboard database.
  • odasa runQuery --query <query-filename> --db <Semmle-database> – to run a query against any other type of Semmle database.

By default, the tool automatically detects an appropriate database schema and QL library to use for the analysis. For example: you can use odasa runQuery --query queries/custom/ViolationReport.ql --db dashboards/example to run the custom query  ViolationReport.ql on the dashboard database stored in the example directory - there is no need to define additional options unless you want to change the default output, use a custom QL library or otherwise customize the analysis.

Full options available for this tool:

Code Block
languagenone
odasa runQuery --query <query-filename> [--verbose] [--verbosity <level>] 
 [--snapshot <snapshot>] [--dashboard <dashboard>]
 [--db <Semmle-database>] [--dataset <dataset>]
 [--dbscheme <semmle-dbscheme>] [--library-path <QL-library>]
 [--output-file <filename>] [--compress] [--internal-ids] [--no-display-strings]
 [--fail-on-warnings] [--compilation-cache <path>] [--urls]
 [--suppress-results-without-url] [--icon-paths] [--titles]
 [--timeout <time>] [--debug-log] [--save-compiled-query]
 [--save-cache] [--concurrent] [--threads <int>] [--ram <memory>]
 [--no-include-results-without-url] [--fast-compilation] [--local-checking]

Flags

The runQuery tool supports the following flags:

Standard use

The --query flag and one of  --snapshot , --dashboard , --db or --dataset must always be specified.

FlagsValueExampleNotes
--query<query-file>queries/custom/ViolationReport.qlThe QL query file to execute.
--snapshot<snapshot>projects/example/revision-2014-March-01

Defines the location of the directory containing then snapshot database to analyze.

--dashboard<dashboard>dashboards/example

Defines the location of the directory containing the dashboard database to analyze.

--db<Semmle-database>baseline/release-3.0/working/db-java

Defines the location of the Semmle database to analyze.

Note that if you want to analyze a snapshot database using this command, then you must first unzip the db-${language}.zip archive stored in the working directory of the snapshot. If you use the --snapshot option then the tool performs this task for you automatically.

--dataset<data-set>-

Defines the location of the dataset to use. Setting this option disables auto-detection.

Customize the results

The following optional flags can be used to customize the way that the results of the analysis are output:

FlagsValueExampleNotes
--output-file<filename>results/violations.csv

Optional. The name and location of an output file for the results.

Default: results are returned on standard output.

--compress--

Optional. GZip-compress the output (to stdout or file).

Default: results are not compressed

--titles--

Optional. Include column titles in the CSV output.

Default: no title row is included.

--urls--

Optional. Include a column containing the URL for each result.

Default: URLs are not included in the output.

--no-include-results-without-url--

Optional. Enabling this option makes the --urls option mandatory. Suppress results when they do not provide a URL, rather than including a fake URL.

Default: all results are included and results without a valid URL are supplied with a fake URL.

--internal-ids--

Optional. Include a column containing the internal identifier for each result.

Default: internal identifiers are not included.

--no-display-strings--

Optional. Suppress the default column containing the display string for each result, calculated using the toString() method

Default: the string representation of each location is reported for every result.

--icon-paths--

Internal: include a column containing the icon path for results that have them. This has no effect on most queries and should be regarded as an internal flag.

Default: no icon paths are included.

--verbose--

Optional. Display brief progress messages on stdout. This information shows exactly what actions are identified by the auto-detection and can be useful if the results are different from expected.

Default: no output unless the analysis fails.

--verbosity<level>2

Include Page
_--verbosity
_--verbosity

--debug-log--

Optional. Enabling this option makes --output-file mandatory. Detailed information about the full progress of the analysis is printed to stdout. This is useful if you need to send the support team information about the analysis.

Default: no output unless the analysis fails or unless --verbose is specified.

--fast-compilation--

Optional. If query compilation is required, perform a quick compilation, omitting some optimizations.

Default: any compilation of queries includes all possible optimizations.

--local-checking--

Optional. If query compilation is required, restrict error checking to the parts of the QL program that are used. Any areas of the imported QL libraries that are not used by the query are ignored.

Default: any error checking covers the full QL program, including all areas of the QL libraries that are imported.

--fail-on-warnings--

Optional. Any warnings reported during compilation are treated as an error and the compilation fails.

Default: any warnings are ignored. The process only fails if an error is detected.

--compilation-cache<path>custom-location/cache

Optional. Use to define an alternative directory to use for the compilation cache.

Default: the directory defined by the SEMMLE_COMPILATION_CACHE environment variable is used.

Customize the analysis

Please note that some of these options override the automatic detection of the appropriate database schema and QL library to use.

FlagsValueExampleNotes
--ram<memory>2048

The amount of memory to use.

Include Page
_--ram
_--ram

Default: 1536 MB for 32-bit systems, 4096 MB for 64-bit systems.

--threads<int>2

The number of threads to use.

Default: 1.

--timeout<time>100

The timeout after which to abort query evaluation.

Include Page
_--timeout
_--timeout

Default: 300 seconds

--save-compiled-query--

Save (and use, if it exists) the query in its compiled form. The compiled query is stored in the same directory as the query file with the same filename and the extension .qlo. This is particularly useful if you create a script to run one query on multiple dashboards or snapshot.

Default: the compiled query is not saved.

--save-cache--

Save the intermediate results of running the query to the cache folder for the snapshot database. This will increase the time taken to run the current query but will speed up any subsequent queries that can re-use the intermediary results.

Default: intermediate results are not stored.

--dbscheme
<semmle-dbscheme>
-

Not normally required. Use only if you need to override the database schema determined by automatic detection. Setting this option disables auto-detection.

Default: the appropriate database schema is automatically detected.

--library-path<QL-library>dashboards/example/deployed/Dashboard-ODASA/WEB-INF/classes/config

Not normally required. Use only if you need to override the database schema determined by automatic detection. Setting this option disables auto-detection.

Default: the appropriate QL library is automatically detected.

--concurrent--

Allow concurrent executions of other commands (dangerous).

Default: no other Semmle analysis commands can be run while the runQuery tool is analyzing data.

Results

The query is run against the Semmle database database specified. Results are either reported on stdout or saved to the file defined by the --outputFile flag. The format for the results is determined by:

  • the select statement in the query
  • the results flags used (for example, --titles).

Examples

The main requirements are controlled by whether you are running a query against a snapshot database or a dashboard database.

Example: Snapshot database query

To run a custom query, FindErrorInCode.ql, stored in the SEMMLE_DIST/queries/custom/java directory on a snapshot database created for the Hadoop project (stored in projects/Hadoop), you would run the following command from the odasa directory :

Code Block
languagenone
 odasa runQuery --snapshot projects/Hadoop/revision-2016-February-24--14-50-01 --query queries/custom/java/FindErrorInCode.ql

The results of the FindErrorInCode.ql query on the snapshot from 24 February 2016 are output to the console in CSV format, without headings. If you want to output the results to a CSV file with column headings, then you would also need to specify the --titles and --outputFile flags. 

Example: Dashboard database query

To run a custom query, ReportCorrectnessViolations.ql, stored in the SEMMLE_DIST/queries/custom/dashboard directory on a dashboard database created for the Hadoop project (stored in dashboards/Hadoop). You could run the following command from the odasa directory:

Code Block
languagenone
odasa runQuery --dashboard dashboards/Hadoop --query queries/custom/dashboard/ReportCorrectnessViolations.ql --titles --outputFile CorrectnessViolations.csv

The results of the ReportCorrectnessViolations.ql query would be output to a CSV file called: CorrectnessViolations.csv.

Known limitations

Some of the flags available for use with this tool are intended for internal use only and should be avoided or used with care.