Semmle 1.19
Skip to end of metadata
Go to start of metadata

This tutorial helps you use the bootstrap command to create a new project for a Python code base.

Overview

In this tutorial you will create a snapshot to allow you to analyze the SCons code base. SCons is an open-source "software construction" tool, written in Python 2.

The tutorial covers basic use of the odasa bootstrap tool to retrieve source files, start code analysis and export a snapshot of the project. 

Time needed: 10–15 minutes

Prerequisites

This tutorial assumes:

Create a new project with odasa bootstrap

The quickest way to create a new Semmle analysis project is to use the odasa bootstrap command. The bootstrap tool steps you through the process and generates configuration files that you can reuse later. The process described here gives the steps required to analyze a particular Python project. Other projects will require different responses to some of the bootstrap prompts.

The bootstrap tool's on-screen instructions indicate what you should enter at each step. However, the first time you run it, you may find it useful to refer to the following procedure for some additional information.

  1. Open a command console.

  2. Change to the directory where Semmle Core is installed—for example, /opt/odasa.
  3. Enter the following command to make sure the environment is set up correctly:

    Linux and OS X:  source setup.sh 
    Windows: setup.bat 

    See Setting up the environment.

  4. Enter the following command to check that you are using Python 2.7:

    python --version

  5. Enter the following command to start the bootstrap tool:

    odasa bootstrap

    The tool guides you through the rest of the process. The remainder of the steps provided below give some additional guidance that you may find useful the first time you run the bootstrap tool.

    For additional on-screen information, enter ? at any of the prompts.

  6. Project name

    Output from the bootstrap command
     *** ODASA Bootstrap ***
     (c) Semmle ltd. 
    
    Welcome to the ODASA bootstrap utility, which is designed to help you get started by generating a basic configuration file for a new project. 
    
    You will be prompted for a series of choices, which will determine the configuration that will be generated. Any time you are prompted for input, you can just enter '?' to see a more detailed explanation of the possible inputs. 
    
    Please enter the project name: >

    Enter: SCons

  7. Project language



    Output from the bootstrap command
    What is the project language?
    	- Java [j]:                   The project is implemented using Java.
    	- C/C++ [c]:                  The project is implemented using C or C++.
    	- C# [C]:                     The project is implemented using C#.
        - COBOL [b]:				  The project is implemented using COBOL.
    	- Python [p]:                 The project is implemented using Python.
    	- JavaScript/Typescript [J]:  The project is implemented using JavaScript and/or TypeScript.
    
    [j|c|C|p|J|?] > 
    The SCons project is written in Python.
    Enter: p

    Notes

    The languages listed here depend on your Semmle license and the version of the Semmle software you have installed. If you do not see the Python option, check that your Semmle license includes Python analysis.

  8. Project source

    Output from the bootstrap command
    How should ODASA obtain the project source?
    	- Subversion [s]: The source is in Subversion.
    	- git [g]:        The source is in git.
    	- Mercurial [m]:  The source is in Mercurial.
    	- Detached [d]:   The build should be done in a detached source directory.
    
    [s|g|m|d|?] > 

    The SCons project uses Git as the version control system for its source code.
    Enter: g

  9. Source location

    Output from the bootstrap command
    Git repository URL: > 

    The SCons project stores its source code in GitHub. 

    Enter:  https://github.com/SConsProject/scons  

    Providing the bootstrap tool with the URL of the repository makes it possible to automatically download the latest version of the software each time an analysis is triggered.

  10. Branch

    Output from the bootstrap command
    Git branch or ref: [default: <empty>] > 

    Press Enter without specifying a branch name.

    The bootstrap tool will default to cloning the code from the "master" branch for the project.

  11. Location of Python modules

    Output from the bootstrap command
    Enter the search path for finding Python modules:

    Enter: ? to see more information about what to enter.

    Output from the bootstrap command
    Enter the path list for finding Python modules, separated by ';'s.
    The standard python search path will be included automatically.
    If you do not specify a path, the source directory for the project will also be included.

    Press Enter without specifying a path.

  12. Paths to exclude

    Output from the bootstrap command
    Enter any paths to exclude: > 

    Sometimes you may also need to exclude paths from analysis. For example, if you have installed SCons, then you want to ensure that this version of SCons is not mistakenly analyzed instead of the newly downloaded snapshot. To prevent this we can exclude the paths that contain the installed version of SCons. On Linux, you might enter  /usr/lib/python2.7/dist-packages/:/usr/lib/python2.6/dist-packages/

    If you do not want to exclude any paths, press Enter here without giving a path.

  13. Main module

    Output from the bootstrap command
    Main module path: >

    Enter:  ${src}/src/engine/SCons/Script/Main.py

    This is the path for the main module for SCons.

  14. Creating a snapshot

    You have now supplied all the information needed for the project file that is used each time a snapshot is generated. You can now go ahead and create a snapshot. 

    Output from the bootstrap command
    Do you want to automatically add a snapshot to the project?
     - Yes [y]: Add the latest snapshot to the project.
     - No [n]:  Do not add the latest snapshot to the project.
    
    [y|n|?] >

    Enter: y

    This tells the bootstrap tool that you want to clone the current SCons source files from GitHub and generate files needed to build a Semmle snapshot. This command will be run at the end of the bootstrap process.

  15. Running a set of analyses 

    Output from the bootstrap command
    Do you want to run a set of analyses against the snapshot?
     - Yes [y]: Run an analysis suite on the snapshot.
     - No [n]:  Do not analyze the snapshot at this time.
    
    [y|n|?] >

    Enter: y

    This tells the bootstrap tool that you want to run a set of queries to analyze the snapshot. This command will be run at the end of the bootstrap process. 

  16. Specifying an analysis suite 

    Output from the bootstrap command
    Which analysis suite should be applied? Leave blank for a default set. [default: <empty>] >

    Press Enter without specifying a suite. The bootstrap tool will run the standard set of Python queries.

  17. Exporting the database archive

    Output from the bootstrap command
    Do you want to export a database for a freshly created snapshot?
     - Yes [y]: Export a database archive for the snapshot.
     - No [n]: Do not export a database archive for the snapshot.
    
    [y|n|?] >

    Enter: y

    This tells the bootstrap tool to create a zip archive containing a snapshot in your project directory. This command will be run at the end of the bootstrap process. 

    The bootstrap tool now:

    • Clones the current SCons source files from GitHub, and generates files needed to build the snapshot.
    • Indexes the code base, builds a snapshot, performs Python analysis, and saves the results of the analysis in a SARIFv2 file in the project directory.
    • Exports the snapshot as a zip archive. You can use this snapshot to run queries in your IDE, for example using the QL for Eclipse plugin.

    Completing these operations will take a few minutes.

    When the bootstrap tool finishes, it displays a message like this:

    Output from the bootstrap command
    ...
    [2018-05-15 10:44:44] [analysis] Completed analysis for SCons - revision-2018-May-15--10-32-51 in 8 minutes.
    [2018-05-15 10:44:44] [analysis] SCons - revision-2018-May-15--10-32-51: 166 total, 166 succeeded (8:56 minutes)
    [2018-05-15 10:44:44] [analyzeSnapshot] Interpreting query results into analysis results
    Analysis results written to C:\odasa\projects\SCons\python2.sarif.
    [2018-05-15 10:44:55] [export] Extracting source
    ...
    Snapshot database exported to C:\odasa\projects\SCons\SCons_db.zip.
    Project configuration complete. 

    Now you can view the analysis results in a SARIF viewer, or import the snapshot database into your IDE and run additional analyses.

Problem solving

If error messages are reported, you can investigate the problem by reviewing the log files:

  • ...\projects\SCons\log.log
  • ...\projects\SCons\revision--<date>-<time>\log\build.log
  • ...\projects\SCons\revision--<date>-<time>\log\import.log
  • ...\projects\SCons\revision--<date>-<time>\log\analysis.log

Before rerunning the bootstrap process, enable prototyping mode. This will help with any further troubleshooting you need to do.

What next?

Spend some time investigating the analysis of the SCons code base. This will help you familiarize yourself with the standard rules available for Python analysis.

Query the snapshot in your IDE

If you install a QL plugin or extension, you can easily write custom queries to analyze snapshots and view the results directly in your IDE. 

Create more snapshots

Use the bootstrap tool to create a project for your own code base.

Work through the tutorial on advanced project creation to see some examples of the available configuration options.