Semmle 1.19
Skip to end of metadata
Go to start of metadata
Setting up Semmle Insight server: Part 3

This topic is third part of a series of topics that step you through the process of setting up Semmle Insight server. In this topic you will learn how to set up a web service that the Semmle Insight server uses to receive Enterprise Insight and/or Team Insight data over the HTTPS protocol.

-

An alternative way of running the Insight server—instead of using a web service—is to run it as a local service, see Starting an SSH-based Insight server service.

-

Prerequisties

  • You must have worked through the first two parts of the Insight server setup process:
    1. Creating a database for Insight data
    2. Defining an Insight server configuration file
  • The Semmle environment must have been set up, in the command console you will be working in, by running the setup script—see Setting up the environment.
  • A Java web application server must be available—for example, Apache Tomcat—to which the Insight server service's webapp file (insight-server.war) can be deployed. 

    -

    Configuring the Tomcat environment

    Semmle recommends that you use a logback file to improve the logging of any SQL exceptions. If you are using Apache Tomcat web server, you can enable logging of SQL exceptions by using the logback.configurationFile Java system property to specify a logback file.

    It is also useful to define the variable SEMMLE_DIST for the Tomcat environment, setting it to the path to your Semmle distribution.

     Click here to show more detailed information...

    On Tomcat you can set up a logback file, and define SEMMLE_DIST within Tomcat's environment, by creating a file called setenv.sh (on Linux and OS X) or setenv.bat (on Windows). Create the file in Tomcat's bin directory (in the CATALINA_HOME directory—for example /usr/share/tomcat8/bin). Enter the following environment variable definitions in the file:

    • Linux and Apple OS X:
      CATALINA_OPTS="-Dlogback.configurationFile=${CATALINA_BASE}/conf/logback.xml"
      SEMMLE_DIST="<path-to-the-base-directory-of-the-Semmle-distribution>"
    • Windows:
      CATALINA_OPTS="-Dlogback.configurationFile=%CATALINA_BASE%\conf\logback.xml"
      SEMMLE_DIST="<path-to-the-base-directory-of-the-Semmle-distribution>"

    Then create the logback.xml file in the location specified by the Dlogback.configurationFile option with the following content:

    Example logback.xml file
    <configuration>
     <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
       <!-- encoders are assigned the type
            ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
       <encoder>
       <!--
         <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
       -->
         <pattern>%d{HH:mm:ss.SSS} %-5level %logger{50} - %msg%n</pattern>
       </encoder>
     </appender>
     <root level="INFO">
       <appender-ref ref="STDOUT" />
     </root>
     
     <logger name="com.semmle" level="INFO" additivity="false">
      <appender-ref ref="STDOUT" />
     </logger>
    </configuration> 

    You can raise the logging level to debugging output by replacing the  string "INFO" with "DEBUG" in the line <logger name="com.semmle" level="INFO" additivity="false">

    For more information, see the Tomcat documentation: https://tomcat.apache.org/tomcat-7.0-doc/RUNNING.txt, section 3.4.

    -

  • The server used to host the Insight server web service must have the HTTPS protocol enabled. The steps listed below include instructions for doing this on a Tomcat web server.
  • The server and all clients must have a Java keystore file containing a private key and associated signed certificate (Java 7 keytool). If someone else creates the keystores for you then you must know the user name and password used to generate each key pair.

    For details of how to create Java keystores for testing purposes, see How can I get SSL certificates for testing an Insight server. If you decide to use a keystore generated in this way, step through this topic, entering each command in the order shown. A number of files are created. The ti-server.jks file must be saved to the web server, as described below. The client-server.jks file must be saved to the client computer, also described below.

Setting up an HTTPS service

When you have created one or more Insight server configuration files, and the Java keystore files required for the server and client machines, you are ready to set up an Insight server web service.

  • The following steps assume that you are using Apache Tomcat, but the steps are similar for other web servers.
  • To simplify the following instructions, environment variables have been written without the operating-system-specific notation—for example, CATALINA_BASE rather than $CATALINA_BASE (Linux/OS X) and %CATALINA_BASE% (Windows).
To set up an Insight Server service using Tomcat
  1. Insight server setup

    1. Add the insight-server.war file to your instance of Tomcat by copying:
        SEMMLE_DIST/tools/insight-server.war
      to:
        CATALINA_BASE/webapps/insight-server.war.
      Where CATALINA_BASE is the base directory of the instance of Tomcat—for example, /var/lib/tomcat8.

    2. Store the Java keystore for the Insight server (for example, ti-server.jks ) in the CATALINA_BASE/conf directory.

    3. Edit CATALINA_BASE/conf/server.xml and define a Connector element for SSL connections, using the Java keystore for the server. For example, see lines 6 onward:

      Example Connector element
      <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
              connectionTimeout="20000"
              redirectPort="8443" />
      
      <!-- Connector element for the Semmle Insight server -->
      <Connector SSLEnabled="true"
              keystoreFile="PATH-TO-TOMCAT/conf/ti-server.jks" 
              keystorePass="PASSWORD" 
              maxThreads="150" 
              port="8443"
              protocol="org.apache.coyote.http11.Http11NioProtocol" 
              scheme="https"
              secure="true" 
              sslProtocol="TLSv1"
              truststoreFile="PATH-TO-TOMCAT/conf/ti-server.jks" 
              truststorePass="PASSWORD" 
              truststoreType="JKS" />  

      In the above example, replace:

      • Both instances of PATH-TO-TOMCAT/conf/ti-server.jks with the location of the server's keystore.
      • Both instances of PASSWORD with the password for this keystore.
    4. In CATALINA_BASE/conf, create a file called semmle-users.xml containing the following:

      Example base semmle-users.xml file
      <tomcat-users>
         <role rolename="insight"/>
         <!-- Define one <user> element for each client keystore. The username must match the -dname
              flag used to generate the client key pair. --> 
      </tomcat-users>

      This file will contain user details—that is, details of the clients that are permitted to connect to the service. You will add details of each client to this file in a later step.

    5. Create a <hostname> directory in the CATALINA_BASE/conf/Catalina directory. Note that <hostname> must match the value that you used when you created the server keystore.
    6. In the <hostname> directory, create a file called insight-server.xml. This will be the Tomcat configuration file to control the insight-server.war web application.

    7. In this file, add a Context root element containing three child elements, as follows:

      Example: conf/Catalina/<hostname>/insight-server.xml file
      <Context>
         <Realm className="org.apache.catalina.realm.MemoryRealm" allRolesMode="authOnly" pathname="USERS-CONFIG-FILE" />
         <Parameter name="work-dir" value="IS-SERVICE-WORKSPACE" override="false" />
         <Parameter name="query-dir" value="SEMMLE-DISTRIBUTION-DIR/queries" />    
      </Context>

      Change the attribute values as required:

      • Realm: change USERS-CONFIG-FILE to the path to the semmle-users.xml file (see step d).
      • Parameter name="work-dir": change IS-SERVICE-WORKSPACE to the path to the Insight server service's workspace directory (which you created in the previous part of these setup instructions).
      • Parameter name="query-dir": change SEMMLE-DISTRIBUTION-DIR to the absolute path to the Semmle distribution directory.

        The query-dir parameter defines a path to a directory that contains the SQL scripts to be run when data is delivered to the service. This path is used as a fallback if scripts defined in the data model (either directly within the Insight server configuration file or in the referenced datamodel.xml file) are not found in the workspace directory (as defined by the work-dir parameter). Typically the query-dir value is the queries directory in the Semmle distribution directory.

        Data is not loaded into the database if the SQL scripts are not found.

       Click for an example of an insight-server.xml file
      insight-server.xml file from a Windows system
      <Context>
         <Realm className="org.apache.catalina.realm.MemoryRealm" allRolesMode="authOnly" pathname="C:\Program Files\Apache Software Foundation\Tomcat 8.5\conf\semmle-users.xml" />
         <Parameter name="work-dir" value="C:\semmle\semmleHOMEdir\IS-workspace" override="false" />
         <Parameter name="query-dir" value="C:\semmle\odasa\queries" />   
      </Context>
  2. For each Insight server web service
    1. If you have already defined an Insight server configuration file, move this file to the Insight server service's workspace directory.

      Make sure you have changed the file name from the name of the template file (insight-server.xml) to something else—for example, TeamInsight-serverConnection.xml.

       If you have not yet created a configuration file...
      1. Copy the SEMMLE_DIST/templates/enterprise-insight/insight-server.xml or SEMMLE_DIST/templates/team-insight/insight-server.xml file to the workspace directory you created previously.
      2. Rename the file to avoid confusion with the web server's insight-server.xml file—for example, TeamInsight-serverConnection.xml
      3. Edit the newly created file and define the database connection. If you have not set the environment variable SEMMLE_DIST for Tomcat, you have to replace the expression ${semmle_dist} in the databaseconfig and datamodel elements with the full path to the SEMMLE_DIST directory.

      4. If you want to use one server to update multiple databases, repeat steps a and b, giving each file a unique name.

    2. Edit the CATALINA_BASE /conf/semmle-users.xml file and define a user element for each client that will publish to the server, for example:

      Example semmle-users.xml file with user defined
      <tomcat-users>
         <role rolename="insight"/>
         <!-- Define one <user> element for each client keystore. The username must match the -dname
              flag used to generate the client key pair. --> 
         <user username="CN=CompanyX, O=CompanyX Ltd, L=London, C=UK" roles="insight"/>
      </tomcat-users>

      where the username values for CNOL and C match the values in the client certificate, defined by the -dname parameter when the client Java keystore was created.

      The values shown in the example above are those used in the topic How can I get SSL certificates for testing an Insight server .

  3. Client setup
    If you have not already done so, save the client keystore file (for example, ti-client.jks ) to the computer that will publish to the Insight server (typically this is the Semmle master server). For example, save it to the SEMMLE_HOME directory on the master server.

  4. Note the URL
    Semmle applications will now be able to access the Insight server web service at the following URL:
    https://<hostname>:<port>/insight-server/upload/<InsightServer-config-nameonly>
    where:
    • <hostname> is the host name of the server
    • <port> is the port in the CATALINA_BASE/conf/server.xml file
    • <InsightServer-config-nameonly> is the name of the Insight server configuration file, minus the .xml file name extension. This is the configuration file you saved to the service's workspace directory. It defines the database to update. For example, if the configuration file is called ti-insight-server.xml then the final part of the URL is ti-insight-server.
      If you need to check the location of the workspace directory it is specified by the work-dir attribute in the configuration file for the Insight server web application (for example, for Tomcat: CATALINA_BASE/conf/Catalina/<hostname>/insight-server.xml).

    Example URL: https://localhost:8443/insight-server/upload/ti-server
    Make a note of this URL as you will use it in the final part of the setup process, when you publish data to the web server at this location.

Important

The Java keystore containing the client part of the certificate must be available on computers that communicate with the Insight server web service. For Enterprise Insight, this location is defined in the ei-client.xml file. For Team Insight, this location is defined in the call to the insightHttpClient command.

What next?

Having set up the web service for the Insight server, you are now ready to configure client data providers to publish data to the new service.

For the final part of the Insight server setup process, choose one of the following topics: