Semmle 1.20
Skip to end of metadata
Go to start of metadata

 

-

On this page:

Configuration files

FilePurpose

Configuration file

Semmle Core: defines the analysis for one Semmle dashboard, including the snapshots to analyze and the queries to evaluate for each snapshot. The file controls the behavior of the buildDashboard tool. The resulting dashboard can be viewed using Mend or Project Insight. The data can be published to Enterprise Insight.

Default path: SEMMLE_HOME/dashboards/<dashboard>/Configuration

dashboard-config.json

Project Insight: controls application-wide settings for a single dashboard, for example: default violation metric, optional header image, date format. Changes affect the dashboard web archive generated by the buildDashboard tool.

Default path: SEMMLE_HOME/dashboards/<dashboard>/dashboard-settings/dashboard-config.json

dashboard-settings directory

Project Insight: directory containing configuration files that control the interface for Project Insight. Changes to files in this directory affect the dashboard web archive generated by the buildDashboard tool.

Default path:  SEMMLE_HOME/dashboards/<dashboard>/dashboard-settings

datamodel.xml

Team Insight and Enterprise Insight: This file defines how to process data passed to the insightTableau command. Optionally, may be used by the Insight Server configuration file to define how to process data published to an Insight server service. This can be used to configure data processing for any type of Team Insight data.

ei-client.xml

Enterprise Insight: defines what data from a Project Insight dashboard to export to a specific Insight server service when you run the eiPublish command.

Default path:  SEMMLE_HOME/dashboards/<dashboard>/ei-client.xml

FailedRevisions.csv file

Team Insight: A two-column CSV file that lists failing revisions and the classification that was assigned as the reason for the failure.

Insight server configuration file

Insight server: defines how to connect to an Enterprise Insight or Team Insight database and how to process data published to the Insight server. Where the Insight server receives data over an SSH connection, or from the local file system, this file must be saved as insight-server.xml. Where the Insight server is running as a web service the file name can vary.

issue-reporting.xml

Semmle Core: optional configuration file, used to define integration with Atlassian JIRA. This XML file defines the details required for the buildDashboard tool to connect to JIRA and raise or update issues for violations found by Semmle analysis.

Default path: SEMMLE_HOME/dashboards/<dashboard>/issue-reporting.xml

project file

Semmle Core: defines the language, checkout commands, build commands and snapshot retention policy for a project. For Team Insight analysis, the build elements of this file control whether: light analysis is performed; or whether the code base is built and a snapshot database created for full Team Insight analysis.

project-layout format

Project Insight: optional configuration file, used to filter areas of code or change the way that a code base is displayed in Project Insight. Changes affect the dashboard web archive generated by the buildDashboard tool.

Default path:  SEMMLE_HOME/projects/<project>/<project-layout>

project-summary-config.json

Project Insight: controls the layout and content of project summary pages for a single dashboard . Changes affect the dashboard web archive generated by the buildDashboard tool.

Default path: SEMMLE_HOME/dashboards/<dashboard>/dashboard-settings/project-summary-config.json

queries.xml file

Semmle Core: defines the language of associated query files. This is used to resolve the import statements in associated query files and also to determine whether or not a query is compatible with the snapshot database or dashboard database that is currently being analyzed.

Query configuration

Bamboo plug-in: custom template that defines one or more queries to run on a project. The query configuration uses the same format as the query configuration section of a dashboard Configuration file and as query suite files. For information about defining the query configuration for a dashboard generated using Semmle Core directly, see Configuration file.

Query suites

Semmle Core: used to define a logical group of queries in order to simplify query management during analysis.

repositories file

Team Insight: The input file used by the insightConfig tool to generate a team-insight configuration file.

repository-config.json

Project Insight: defines the layout and content of the Repository page for a single dashboard. Changes affect the dashboard web archive generated by the buildDashboard tool. 

Default path: SEMMLE_HOME/dashboards/<dashboard>/dashboard-settings/repository-config.json

server.log file

Team Insight and Enterprise Insight: Details of the log file created each time an Insight server is started as a local service, and how copies of this log file are archived.

servlet.properties

Project Insight: defines the date format, the font used in charts and limits on file sizes for code syntax highlighting for a single Project Insight dashboard. Changes affect the dashboard web archive generated by the buildDashboard tool. 

Default path: SEMMLE_HOME/dashboards/<dashboard>/dashboard-settings/servlet.properties

slaves.xml file

snapshot file

Semmle Core: configuration file created automatically when a snapshot is added a the project using the addSnapshot or addLatestSnapshot tool. It includes a definition of where the source code for the snapshot is stored and the commands required to build this code (copied from the project file or defined on the command line at the time of creation). The information is used when the snapshot is built and the snapshot is indexed to create a snapshot database. In normal operation there is no reason to edit the content of the snapshot file.

team-insight file

Team Insight: defines the analysis requirements of one instance of Team Insight. The details defined in this file control the data sources used by the attribution tool to collect data for Team Insight analysis.

Default path: SEMMLE_HOME/team-insight/<instance>/team-insight

workers.xml file

Team Insight: The workers.xml file (which can also be called slaves.xml) is used to facilitate the setup and maintenance of data collection/analysis. The file is optional but recommended if you are using Linux or OS X as running Team Insight analysis without it requires additional manual steps—for example, starting the individual worker processes manually. The file defines worker nodes for the attribution tool to use for Team Insight data collection. Default path: SEMMLE_HOME/team-insight/<instance>/workers.xml.

Tools

CommandPurpose

addLatestSnapshot

Semmle Core: used to create a snapshot for the latest version of your code base. The checkout commands used to create the snapshot are defined by the project configuration file or entered on the command line.

addSnapshot

Semmle Core: used to create a snapshot for a historic version of the code base. The checkout commands used to create the snapshot are defined by the project configuration file or entered on the command line.

analyzeSnapshot

Semmle Core: optional command used to analyze a Semmle snapshot and generate data for display in a code review tool.

applySnapshotPolicy

Semmle Core: optional command to apply the snapshot deletion policy defined in a project configuration file to the snapshots stored for a project. During standard operation this task is performed by the addSnapshot and  addLatestSnapshot commands, but on distributed systems it may be necessary to carry this task out separately. For example, when snapshots are stored on a separate server from the server where the addSnapshot and addLatestSnapshot commands are run.

archiveSnapshot

Semmle Core: optional command to compact the snapshot database generated by the buildSnapshot tool manually. During standard operation this task is performed by the buildDashboard command, but on distributed systems it may be necessary to carry this task out separately.

attribution

Team Insight: Use this command to start data collection and analysis for Team Insight, using the configuration defined in a team-insight file. Work is carried out by the "workers," each of which is a separate process running either remotely or on localhost.

autoBuild

Semmle Core: autoBuild makes configuring a compiled-language project for Semmle analysis easier by automatically working out the best build commands to use.

bitbucketReview

Semmle Core: optional command used to integrate Semmle analysis directly into the Bitbucket server review process.

bootstrap

Semmle Core: this command guides users through the process of creating a project configuration file for a new code base. Users may also choose to add a snapshot to the project, run a set of analyses, and export the snapshot.

buildDashboard

Project Insight: command used to analyze snapshots and build a dashboard database. The snapshots to include and analysis performed are defined by the dashboard Configuration file. Optionally, a project-layout file can be used to alter the code base layout and files in the dashboard-settings directory can be edited to customize the layout of Project Insight. The resulting dashboard can be viewed using Project Insight or Mend, and the data can be published to Enterprise Insight using the eiPublish command.

buildSnapshot

Semmle Core: command may be used to build and index a single snapshot of source code for a project. When the tool has finished the code index is ready for analysis. The build commands used are defined by the snapshot configuration file (copied from the project configuration on creation of the snapshot).

createInsightArchive

Insight server: command used to create an archive file containing all of the existing analysis data for the specified repositories/projects. The resulting archive file can then be uploaded to an Insight server using the uploadInsight or uploadHttpInsight command with the --from-archive flag.

createOdasaJavaHome

Semmle Core: legacy command used for installations where Java builds are analyzed using a customized JAVA_HOME directory (the default behavior in Semmle 1.7.6 and earlier versions).

createProject

Semmle Core: this command enables you to create a new project by copying the project configuration file for an existing project or a predefined template. When the command finishes, you need to edit the new project configuration file, configure the checkout command and verify that the build steps are correct. If you want to create an entirely new project then you should use either the insightConfig command or the bootstrap tool.

credentials

Semmle Core: This tool is used to create and manage credentials files which are used to store passwords, SSL private keys, and trusted certificates. The credentials stored in this file can be accessed by tools that need to authenticate with source code repositories, an Insight server or other secure systems.

deleteSnapshot

Semmle Core: You can use this tool to delete a snapshot from a project. This is most useful when you are setting up a new project and experimenting with options. During normal processing, this tool is run automatically and controlled by the snapshot deletion policy defined in the  project file.

duplicateCode

Semmle Core: used to analyze a snapshot for the presence of duplicate code. Usually called automatically by a build command defined in the project file created by the bootstrap or insightConfig tools.

editProject

Project Insight: optional command enables you to modify one or more properties of an existing project configuration file. This is an alternative to editing the project file manually and is typically used in a script to modify the checkout or build command part way through the process. 

editSnapshot

Project Insight: command enables you to modify one or more properties of a snapshot, in addition you can add, change or delete metadata values associated with a snapshot. Often used in a build step in the project configuration file. When the command is run, the metadata set by the command is automatically incorporated into the snapshot database.

editUsers

Project Insight: used to create and maintain users for a specific dashboard. Users who have logged into a dashboard can dismiss defects using Semmle Project Insight or Semmle Mend.

eiPublish

Enterprise Insight: command collects data from a Project Insight dashboard and pushes the data to an Insight server service. The behavior of the command is controlled by the ei-client.xml file stored in the configuration directory of the dashboard that you specify.

export

QL plugins and extensions: command used to export a snapshot generated using the QL command line tools for use in the QL plugins and extensions. The resulting snapshot can be imported into the QL plugins and extensions and used to write new queries.

exportDashboard

Studio: command used to export a dashboard database from a Semmle Core project for use in the Studio application. The resulting snapshot can be imported into Studio and used to write new queries.

exportMendConfig

Mend: command used to export Mend configuration files (.sdc files) with custom query sets for use with local analysis. In Mend you can import an .sdc file to customize the queries run by an analysis configuration of type Local. When run on a dashboard Configuration file, an .sdc file is generated containing a full definition of the analysis carried out for each project included in the dashboard.

exportQueryMetadata

Semmle Core: optional command used to process a set of queries and produce an output file describing the metadata associated with each query.

findObsoleteSnapshots

Team Insight: this command is run on the Team Insight master server to identify and delete snapshots that are no longer required.

getSnapshot

Semmle Core: report the absolute path for a specific snapshot data directory.

getSnapshots

Semmle Core: report the absolute path for all snapshots in a specific project configuration (optionally, report all built or all unbuilt snapshots).

githubReview

Semmle Core: optional command used to integrate Semmle analysis directly into the Github review process.

insightClient

Insight server: command used to communicate with a locally running Semmle Insight server set up for Enterprise Insight or Team Insight. Used to manually upload data, ping the server or to stop the server.

insightConfig

The insightConfig configuration tool reads a repositories file and generates a team-insight file and the individual project configuration files for each project.

insightHttpClient

Insight server: The command used to communicate with an HTTPS-based Insight server web service set up for Enterprise Insight or Team Insight. Used to manually upload data or ping the service.

insightServer

Insight server: command used to start an SSH-based service to process data published by client servers for Enterprise Insight or Team Insight. Data is processed using the configuration defined in the specified insight-server.xml file. Can also be used to upgrade the database for Enterprise Insight or Team Insight via an SSH Insight server.

insightStatus

Team Insight: command used to monitor the progress of Team Insight build and/or attribution operations.

insightTableau

Team Insight: this tool generates a Tableau workbook that you can use to interactively explore the information collected by a Semmle Team Insight installation. It should be called after the successful execution of the updateMetadata command.

lightExtractor

Team Insight light analysis: this tool is run to perform light analysis of a code base. Called automatically by a build command defined in the project file created by the insightConfig (Team Insight light analysis) tool, when the attribution tool runs. It indexes the files and folders under a specific directory and determines basic metrics such as lines of code, lines of comment, lines of test code.

lint

Semmle Core: optional command used to analyze a Semmle snapshot and generate data for display in a code review tool. See also analyzeSnapshot.

matchAuthorAliases

Insight server: command used to perform automatic detection and deduplication of authors with multiple identities in the source code repository, prior to publishing the data to a Semmle Insight server.

overview

Semmle Core: used to provide a quick assessment of the number of lines of code and comment in a source code directory.

prepareQueries

Semmle Core: command may be used to compile new or updated query files to speed up future dashboard generation. The buildDashboard command automatically calls this before analyzing snapshots. All queries in the specified directory are reviewed and all new or updated queries are compiled (creating a .qlo file for each query).

qltest

Semmle Core: command provides a simple way to run regression tests for custom QL queries. The tool runs QL queries against source code that you provide, and it checks that the output of each query matches the expected results.

runQuery

Semmle Core: run queries on a Semmle database from the command line. Particularly useful as part of a script to generate CSV reports or if you want to test a new query for a system before you add it to the dashboard configuration.

selfTest

Semmle Core: used to report details of the current Semmle Core including the version and values of the SEMMLE_DIST, SEMMLE_HOME, and SEMMLE_DATA environment variables.

semflip

Semmle Core: a simple shell function to toggle the current working directory between SEMMLE_HOME and SEMMLE_DATA.

serve

Semmle Core: simple command for hosting an individual dashboard or war file locally. It is intended for ad-hoc usage—if you want to host many dashboards, you should use a web application sever like Tomcat.

updateExternalData

Semmle Core: this tool enables you to update the external data stored in a snapshot database from the data stored in the external/data subdirectory of the snapshot directory without rebuilding the entire snapshot database.

updateMetadata

Team Insight: Use this tool to complete data collection for Team Insight. You should run it after successful execution of the attribution command. Note that if you want to update the metadata stored in a Semmle Project Insight snapshot database, you should use updateExternalData and not this command.

upgrade

Semmle Core: can be used to upgrade snapshot databases that were created with an earlier version of Semmle to enable you to analyze them using the latest version. The database schema of the snapshot databases is updated so that it is compatible with current version of Semmle analysis.

uploadHttpInsight

Insight server: command used to publish per-revision data deliveries to a Semmle Insight server that is running as a webs service, in place of the standard process of uploading a single zip file containing all analysis data. The command prepares zip files containing incremental data updates and uploads these using the insightHttpClient command.

uploadInsight

Insight server: command used to publish per-revision data deliveries to a locally running Semmle Insight server, in place of the standard process of uploading a single zip file containing all analysis data. The command prepares zip files containing incremental data updates and uploads these using the insightClient command.

worker-daemon.jar

This tool performs Semmle Team Insight data collection and analysis as a daemon process, connecting to the Team Insight master server over HTTPS. The tool fetches details of a job to run from the master server, runs the job, passes results back to the master server, and then fetches details of the next job. Multiple instances of this tool can run on each worker machine.

Scripts

ScriptPurpose

add-dashboard script

Used to add a new or updated dashboard to the legacy Apache Tomcat web server. On completion the web server is started or restarted so that the new/updated dashboard is accessible to Project Insight and other client applications.

add-dashboard-no-start script

Used to add a new or updated dashboard to the legacy Apache Tomcat web server, without starting the server. The new/updated dashboard will not be accessible to Project Insight and other client applications until the server is started.

remove-dashboard script

Used to remove a named dashboard from the legacy Apache Tomcat web server. The server is stopped, the dashboard is deleted and then the server is restarted. The dashboard is no longer accessible to Project Insight and other client applications.

start-server script

Used to start the Apache Tomcat web server included in the legacy Tomcat extension pack. Once the server has started, any dashboards added to the server are accessible to Project Insight and other client applications.

stop-server script

Used to stop the Apache Tomcat web server included in the legacy Tomcat extension pack. When the server is stopped, any dashboards added to the server are not accessible to Project Insight and other client applications.

Other reference topics

FilePurpose

Enterprise Insight database

Enterprise Insight: overview of the default database used by Enterprise Insight to store data.

Environment variables

Semmle Core: used to simplify build commands and can be customized to override the default locations and behavior of Semmle Core. See Semmle variables for configuration variables.

Qhelp files

Query help: this topic describes the structure and storage requirements for query help files (.qhelp).

Query directives

Semmle Core: used in a dashboard  Configuration or query suite file to override the default properties of a query or to provide additional information for the calculation.

Query file requirements

Overview of the basic structure of query files ( .ql ) and the properties supported. For information about the QL language and writing your own queries, see  Semmle QL .

Semmle variables

Semmle variables are used in configuration files to simplify the configuration of Semmle analysis and to reduce the use of absolute paths (which make it difficult to relocate a project). Semmle variables can be used almost everywhere in the various configuration files.

Team Insight database

Team Insight: overview of the default database used by Team Insight to store data.

variables file

Semmle Core: The  variables  file allows you to define custom variables that can be used to augment a configuration in various ways. You can define variables for either a Team Insight instance, a particular project, or a particular snapshot.