Semmle 1.19
Skip to end of metadata
Go to start of metadata

This topic describes the infrastructure required to set up an installation of Enterprise Insight.

Overview

Enterprise Insight consists of three main components:

  1. Semmle Insight server—hosts one or more Insight server services to receive data and import it into one or more databases. 
  2. Data providers—publish data to the Insight server service - either from Project Insight dashboards or from third-party applications.
  3. Visualization server—runs the data visualization software.

An overview of the standard server configuration:

Servers

The servers shown in the example configuration above perform the following actions:

ServerActions
Data providers
  • Push data to an Insight server service
Insight server
  • Hosts one or more Insight server services and their associated SQL databases

  • Receives data from data providers and stores it in the appropriate SQL database

  • Imports data received into the designated Enterprise Insight database
  • On request, passes data to the visualization server
Visualization server
  • Hosts the visualization software (for example, Tableau or QlikView)
  • Requests data from the Enterprise Insight database for display by the visualization tool
  • Delivers data to browser or application-based clients belonging to users

Requirements

Enterprise Insight is based on Semmle Core technology (see System requirements for general requirements). For each project that you want to include in Enterprise Insight, you need to configure a working Project Insight dashboard. The additional requirements for Enterprise Insight are described in more detail below.

Data requirements

The data providers require passwordless public/private key connections to the Insight server to enable data to be passed as shown in the diagram above. This can be configured using either SSH (OpenSSH or Tectia SSH preferred) or HTTPS. For third-party applications, the main requirement is that any data that you want to include in Enterprise Insight can be exported into well-defined CSV files. You can then push a zip archive of the exported CSV files to the Insight server service ready to be imported into the Enterprise Insight database.

ServerSoftware required to be installedOther requirements
Data providers
  • Build prerequisites for the software projects to be built: build tools (compilers, etc.), build dependencies
  • Client software for the version control systems (VCS) required
  • Good network access to Insight server
  • Good network access to VCS servers
Insight server
  • Semmle Core

  • SQL database—either PostgreSQL (9.1 or above), or Microsoft SQL Server (Express 2014 or above)
  • Good network access to the visualization server
Visualization server
  • Visualization software—for example, QlikView (11 or above) or Tableau (8 or above)
  • Must have fast network access to the Insight server
  • Meet the requirements for the chosen visualization software

If Enterprise Insight is configured for a weekly update, then all builds and analysis must be able to complete in a timely fashion, allowing time for the aggregation and integration of data and update of the visualization server. For companies with many projects, this requires substantial resources and, in practice, Enterprise Insight is often configured for monthly updates in line with the availability of new data from financial reporting systems.

Data storage requirements

The data storage requirements for the visualization server and Semmle Insight server are fairly small.

The data providers are set up to run Project Insight and need enough space to obtain a full copy of the source code, build it and save the results of analysis. In practice, they typically need a minimum of disk space equivalent to about 10 times the size of the largest project configured for analysis, multiplied by the number of threads that you intend to run on that server. 

Visualization requirements

The Insight server converts the data received from the data providers into a form suitable for the nominated business intelligence reporting tool and stores it in the database. This SQL database is normally hosted by either the Semmle Insight server or the visualization server.

The two business intelligence visualization tools currently supported are: Tableau and Qlikview. Please refer to the product documentation for details of the server requirements for your chosen application.

Server maintenance and backup

The standard server maintenance and file system backup responsibilities are as follows:

ServerMaintenanceBackup
Semmle Insight server

Semmle:

  • Update the Semmle Core software as necessary1

Customer:

  • Semmle Core
  • Configuration files and any associated scripts
  • Enterprise Insight database
Data providers

Customer:

  • Update the SCM client software as necessary
  • Update the build prerequisites as necessary

Customer:

  • Any custom dependencies and setup
Visualization server

Customer or Semmle, as appropriate2:

  • Update the business intelligence software as necessary

Customer:

  • Business intelligence data/workbooks etc.

1 Semmle Core software is usually updated on a regular schedule agreed with the customer.

2 Where an existing visualization application is used to serve Enterprise Insight data, the customer normally takes responsibility for the maintenance of the software. If Semmle installs a new visualization application then they are normally responsible for its maintenance. Visualization software typically needs to be updated less frequently, unless there are specific reasons to upgrade to a new revision of the software.

Minimum backup requirements

The data stored on the Semmle Insight server and visualization server is derived and so it is not essential to back up all analysis and results—providing that, in the event of a server failure, you are prepared to take the time/resources to regenerate all the data again. The minimum backup requirements are as follows:

BackupServerRestore to enable
Visualization data/workbooksVisualizationUsers to view existing data and workbooks

Semmle SQL database

Insight

Users to view existing data and workbooks
Insight server to process new data received
Semmle Core, configuration and any associated scriptsInsightInsight server to process new data received
Configuration and build environmentData providersAnalysis of new versions of code

Planning for a deployment

The number and specification of servers required for an installation of Enterprise Insight varies widely according to the size and number of projects selected for analysis. Semmle recommends that you identify a potential data provider and then select a representative sample of projects for testing and configure them for analysis by Project Insight on this server. This will enable you to verify the:

  • Current build prerequisites for each project
  • Time taken to build/analyze the code
  • Size of a Project Insight snapshot and associated results for a single revision of the source code

This information can be used to confirm the build prerequisites for Project Insight, and help you determine a more accurate estimate of the number and specification of servers required to provide Project Insight data.

  • No labels