Semmle 1.18
Skip to end of metadata
Go to start of metadata

This topic describes how to publish data to a Semmle Insight service from Project Insight dashboards.

Prerequisites

Before you can publish data from Project Insight to a Semmle Insight service you must ensure that communication between the servers is set up correctly. The requirements vary according to the communication protocol defined for the Semmle  Insight service. For example, the server that hosts Project Insight

You can check that the Semmle Insight service is up and responsive by running the following command on the Semmle Insight Master (which hosts the Semmle Insight service):

  • HTTPS - run: odasa insightHttpClient --ping --url https://localhost:8443/insight-server/upload/ei-server --keystore client.jks --keystore-password password
    where:
    • https://localhost:8443/insight-server/upload/ei-server is the URL of the service, 
    • client.jks is the keystore and truststore for the client key
    • password is the password for this store.
  • SSH service - run: odasa insightClient --ping /ei-server-working
    where /ei-server-working is the location of the workspace directory used by the service. 

Overview

You can publish data from a Project Insight dashboard to an Enterprise Insight service by creating an ei-client.xml configuration file and running the eiPublish command. The data received by the service is automatically imported into internal tables in the Enterprise Insight database and processed according to the data model defined in the Insight server configuration file.

Configuring the Project Insight dashboard

For each Project Insight dashboard that you want to publish data for, you need to create an ei-client.xml file.

To define the data to export from a Project Insight dashboard
  1.  In the directory where the Project Insight dashboard Configuration file is stored, create a copy of the Semmle Core templates/enterprise-insight/ei-client.xml file.

    Default ei-client.xml file
    <enterprise-insight>
      <!-- Pick either -->
      <ssh host="host" odasahome="/path/to/odasa" workspace="/path/to/insight-server/config/directory" />
    
      <!-- or -->
      <https url="https://server:port/insight-server/upload/CONFIG-FILENAME-MINUS-EXTENSION" 
          keystore="keystore.jks" keystorePassword="password"/>
    
      <dashboard-queries>
        <query>semmle-insight-queries/queries/enterprise/ProjectViolations.ql</query>
        <query>semmle-insight-queries/queries/enterprise/ViolationDelta.ql</query>
        <query>semmle-insight-queries/queries/enterprise/ProjectMetrics.ql</query>
        <query>semmle-insight-queries/queries/enterprise/ProjectScores.ql</query>
        <query>semmle-insight-queries/queries/enterprise/ProjectQueries.ql</query>
        <query>semmle-insight-queries/queries/enterprise/QueryMetadata.ql</query>
        <query>semmle-insight-queries/queries/enterprise/QueryCategories.ql</query>
      </dashboard-queries>
    </enterprise-insight>
  2. Check which protocol the Semmle Insight service that you want to publish results to uses and delete the unwanted protocol definition. That is, if you want to publish results to an HTTPS service then delete the ssh element (line 3) and if you want to publish to an SSH service then delete the https element (line 6).

  3. Update the remaining protocol definition with the details of the service that you want to publish results to:
    HTTPS service - update the https element with the service URL, name/location of the client keystore file and the password for this keystore. For example:

    Example HTTPS connection
    <https url="http://my.server.com:8080/insight-server/upload/ei-server" keystore="client.jks" keystorePassword="passw@rd"/>

    SSH service - update the ssh element with the server host name, location of the installation of Semmle Core (main odasa directory), workspace of the service and the location of the key to use. For example:

    Example SSH connection
    <ssh host="my.server.com:8080" odasahome="/home/user/odasa" workspace="/home/user/data/ei-server">
      <param>-i</param>
      <param>/home/user/keys/server-key</param>
    </ssh>
  4. The default queries defined in the dashboard-queries element should be suitable for most projects. Define additional query elements to define any custom queries that you want to run on the dashboard database to extract data to publish to the Enterprise Insight service. For example:

    Example dashboard queries
    <dashboard-queries>
       <query>semmle-insight-queries/queries/enterprise/QueryMetadata.ql</query>
       <query>semmle-insight-queries/queries/enterprise/QueryCategories.ql</query>
       <query>semmle-insight-queries/queries/enterprise/ProjectViolations.ql</query>
       <query>semmle-insight-queries/queries/enterprise/ViolationDelta.ql</query>
       <query>semmle-insight-queries/queries/enterprise/ProjectMetrics.ql</query>
       <query>semmle-insight-queries/queries/enterprise/ProjectScores.ql</query>
     </dashboard-queries>
  5. Save the ei-client.xml file.

Publishing data to the Enterprise Insight service

Now that you have defined a configuration file for Project Insight, you can use the eiPublish command to publish data to the Enterprise Insight service defined in that file.

To publish data to the Enterprise Insight service

The precise form of the command you need to run depends on your operating system, your current location and the location of the dashboard that you want to publish. For a Project Insight dashboard stored in odasa/dashboards/mydashboard, on a Linux or Apple OS X server you would run:

  • odasa eiPublish dashboards/mydashboard

from the odasa directory.

When the eiPublish command runs it:

  1. Runs each query defined in the dashboard-queries element of the configuration against the Project Insight dashboard database.
  2. The result of each query is stored in a CSV file with a file name derived from the query file name.
  3. When all the queries have run, the CSV files are added to a zip archive.
  4. The zip archive is published to the Enterprise Insight service using the connection details defined in the ssh or https element.

When the data is received by the service, it is automatically imported into tables and columns in the internal schema of the database. Then the SQL queries defined in the Insight server configuration file are run on the database.

What next?

If you have already set up a dashboard for Enterprise Insight, then you can display this dashboard and see the new data displayed. You may want to update any automated scripts that you use to update Project Insight dashboards to publish data to Enterprise Insight as the final step.

Otherwise you may be ready to use your chosen business intelligence tool to create a dashboard for Enterprise Insight.

  • No labels