Semmle 1.19
Skip to end of metadata
Go to start of metadata

   

Overview

In this tutorial you will create a snapshot to allow you to analyze LuaJIT, which is an open source Just-In-Time compiler for the Lua programming language. 
 

The tutorial covers basic use of the odasa bootstrap tool to retrieve source files, start code analysis and export a snapshot of the project. 

Time needed: 10–15 minutes

Prerequisites

This tutorial assumes:

Create a new project with odasa bootstrap

The quickest way to create a new Semmle analysis project is to use the odasa bootstrap command. The bootstrap tool steps you through the process and generates configuration files that you can reuse later. The process described here gives the steps required to analyze a particular C project built using gcc (Linux) or Visual Studio (Windows). Other projects will require different responses to some of the bootstrap prompts.

The bootstrap tool's on-screen instructions indicate what you should enter at each step. However, the first time you run it, you may find it useful to refer to the following procedure for some additional information.

  1. Open a command console.

    Important - Windows command prompt

    On Windows, the LuaJIT build process requires that Visual Studio's environment variables are set. You must therefore either run odasa bootstrap in a Visual Studio Command Prompt window, or run the following batch script in a standard Windows command console before running odasa bootstrap:

    <Visual Studio installation directory>\Common7\Tools\vsdevcmd.bat

    For example:

    c:\odasa>"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\vsdevcmd.bat"

    If you are using an old version of Visual Studio (2014 or earlier), you may need to replace `vsdevcmd.bat` with `vsvars32.bat`.

    Note

    The requirement to use the Visual Studio Command Prompt (or to run vsdevcmd.bat) is specific to LuaJIT build process, and not a general requirement for building C/C++ projects on Windows.
  2. Change to the directory where Semmle Core is installed – for example, /opt/odasa.
  3. Make sure the ODASA environment is set:

    Linux and OS X source setup.sh 
    Windows setup.bat
     
    See Setting up the environment.
  4. Enter the following command to start the bootstrap tool:

    odasa bootstrap

    The tool guides you through the rest of the process. The remainder of the steps provided below give some additional guidance that you may find useful the first time you run the bootstrap tool.

    For additional on-screen information, enter ? at any of the prompts.


  5. Project name

    Output from the bootstrap command
    *** ODASA Bootstrap *** (c) Semmle ltd. 
     
    Welcome to the ODASA bootstrap utility, which is designed to help you get started by generating a basic configuration file for a new project. 
     
    You will be prompted for a series of choices, which will determine the configuration that will be generated. Any time you are prompted for input, you can just enter '?' to see a more detailed explanation of the possible inputs. 
     
    Please enter the project name: > 

    Enter:  LuaJIT

  6. Project language



    Output from the bootstrap command
    What is the project language?
    	- Java [j]:                   The project is implemented using Java.
    	- C/C++ [c]:                  The project is implemented using C or C++.
    	- C# [C]:                     The project is implemented using C#.
        - COBOL [b]:				  The project is implemented using COBOL.
    	- Python [p]:                 The project is implemented using Python.
    	- JavaScript/Typescript [J]:  The project is implemented using JavaScript and/or TypeScript.
    
    [j|c|C|p|J|?] > 

    The project LuaJIT is written in C.
    Enter: c 

    Notes

    • Make sure to enter a lowercase c for C/C++ and not an uppercase C for C#.
    • The languages listed here depend on your Semmle license and the version of the Semmle software you have installed. You may, therefore, see a different selection of languages than the list shown here.
  7. Project source

    Output from the bootstrap command
    How should ODASA obtain the project source?
    	- Subversion [s]: The source is in Subversion.
    	- git [g]:        The source is in git.
    	- Mercurial [m]:  The source is in Mercurial.
    	- Detached [d]:   The build should be done in a detached source directory.
    
    [s|g|m|d|?] > 

    The LuaJIT project uses Git as the version control system for its source code.
    Enter: g

  8. Source location

    Output from the bootstrap command
    Git repository URL: > 

    The LuaJIT project stores its source code in Git. 

    Enter: https://github.com/LuaJIT/LuaJIT

    Providing the bootstrap tool with the URL of the repository makes it possible to automatically download the latest version of the software each time an analysis is triggered.

  9. Branch

    Output from the bootstrap command
    Git branch or ref: [default: <empty>] > 

    Press Enter without specifying a branch name.

    The bootstrap tool will default to cloning the code from the "master" branch for the project.

  10. Build method

    The Semmle software determines which files to analyze (and which libraries are needed) by monitoring the build process. This means you need to provide the bootstrap tool with information about how the software is built.

    1. OS X:

      Output from the bootstrap command
      How is the code built?
       - Xcodebuild [x]: xcodebuild (Xcode project)
       - Other [o]:   Other
       
       [x|o|?] >
      

      Most C/C++ projects are built on OS X using an Xcode project file. LuaJIT does not provide an Xcode project file. Instead, it is built using gcc and make.
      Enter: o

    2. Linux:
      LuaJIT project is built on Linux using gcc and make. The bootstrap process automatically selects the "other" option and proceeds to the next step. 

    3. Windows:

      Output from the bootstrap command
      How is the code built?
       - Msbuild [m]: msbuild (Visual Studio project or solution - Visual Studio 2010 or later)
       - Vcbuild [v]: vcbuild (Visual Studio project or solution - Visual Studio 2008 or earlier)
       - Other [o]:   Other
       
       [m|v|o|?] >
      

      Most C/C++ projects are built on Windows using a Visual Studio project or solution file. LuaJIT does not provide a Visual Studio project/solution file. Instead, it provides a batch script to build the project. 
      Enter: o

  11. Working directory

    Output from the bootstrap command
    Does the build require a custom working directory?
     - Yes [y]: A custom working directory is needed.
     - No (default) [n]: Running in the root of the source location is fine.
    
     [y|n|?] >
    

    Enter:
    - n
    if building on Linux or OS X 

    - y if building on Windows

  12. Windows only: Name of the working directory

    Output from the bootstrap command
    Working directory: > 

    Enter: src

  13. Clean step

    Output from the bootstrap command
    You will be prompted to enter a build command that, when executed, will compile all relevant source code. This can be an actual executable (e.g. 'make', if running make with the default target is enough), a more complex command line (e.g. 'make clean gen default'), or an arbitrary script that will run the build ('./build.sh').
    
    By default, the build command runs in the source location (identified as ${src} in the project configuration), but, if necessary, you have the option of overriding this.
     
    Should a clean command be run before the build? Note that if a build is run on a project source location that already has built artifacts, ODASA will most likely miss the source files used to build those artifacts if a clean command is not run.
     
    Should existing build artifacts be removed (cleaned) before building?
     - Yes [y]: A 'clean' step should be performed before building.
     - No (default) [n]: A 'clean' step is not necessary.
    
     [y|n|?] > 

    Enter: n

    Info

    When building projects using the make command, usually an incremental build is performed to avoid recompilation of previously compiled source files. Hence, projects built using make typically provide a clean command (such as make clean) to remove the existing build artifacts so that a full non-incremental build can be performed. Some features of Semmle Core (such as building multiple snapshots from the same source code) require a full build to ensure correct behavior. A clean step is essential when using such features.

    Since this tutorial performs a full non-incremental build on newly checked-out source code, a clean step is not essential. On Windows, the LuaJIT project always performs a non-incremental build.

  14. Build command

    Output from the bootstrap command
    Build command: > 

    Enter:
    - make if building on Linux or OS X
    - msvcbuild if building on Windows

  15. Creating a snapshot

    You have now supplied all the information needed for the project file that is used each time a snapshot is generated. You can now go ahead and create a snapshot. 

    Output from the bootstrap command
    Do you want to automatically add a snapshot to the project?
     - Yes [y]: Add the latest snapshot to the project.
     - No [n]:  Do not add the latest snapshot to the project.
    
    [y|n|?] >

    Enter: y

    This tells the bootstrap tool that you want to clone the current LuaJIT source files from GitHub and generate files needed to build a Semmle snapshot. This command will be run at the end of the bootstrap process.

  16. Running a set of analyses 

    Output from the bootstrap command
    Do you want to run a set of analyses against the snapshot?
     - Yes [y]: Run an analysis suite on the snapshot.
     - No [n]:  Do not analyze the snapshot at this time.
    
    [y|n|?] >

    Enter: y

    This tells the bootstrap tool that you want to run a set of queries to analyze the snapshot. This command will be run at the end of the bootstrap process. 

  17. Specifying an analysis suite

    Output from the bootstrap command
    Which analysis suite should be applied? Leave blank for a default set. [default: <empty>] >

    Press Enter without specifying a suite. The bootstrap tool will run the standard set of C/C++ queries.

  18. Analysis

    Output from the bootstrap command
    Select the analysis set to run.
     - C [c]:   Run C analyses.
     - C++ [C]: Run C++ analyses.
     
     [c|C|?] > 

    Enter: c

  19. Exporting the database archive

    Output from the bootstrap command
    Do you want to export a database for a freshly created snapshot?
     - Yes [y]: Export a database archive for the snapshot.
     - No [n]: Do not export a database archive for the snapshot.
    
    [y|n|?] >

    Enter: y

    This tells the bootstrap tool to create a zip archive containing a snapshot in your project directory. This command will be run at the end of the bootstrap process. 

    The bootstrap tool now:

    • Clones the current LuaJIT source files from GitHub, and generates files needed to build the snapshot.
    • Indexes the code base, builds a snapshot, performs C/C++ analysis, and saves the results of the analysis in a SARIFv2 file in the project directory.
    • Exports the snapshot as a zip archive. You can use this snapshot to run queries in your IDE, for example using the QL for Eclipse plugin.

    Completing these operations will take a few minutes.

    When the bootstrap tool finishes, it displays a message like this:

    Output from the bootstrap command
    ...
    [2018-05-18 16:53:49] [analysis]  Completed analysis for LuaJIT - revision-2018-May-18--16-49-23 in 2 minutes.
    [2018-05-18 16:53:49] [analysis] LuaJIT - revision-2018-May-18--16-49-23: 109 total, 109 succeeded (2:45 minutes)
    [2018-05-18 16:53:49] [analyzeSnapshot] Interpreting query results into analysis results
    Analysis results written to C:\odasa\projects\LuaJIT\c.sarif.
    [2018-05-18 16:53:58] [export] Extracting source
    ...
    Snapshot database exported to C:\odasa\projects\LuaJIT\LuaJIT_db.zip.
    Project configuration complete.

    Now you can view the analysis results in a SARIF viewer, or import the snapshot database into your IDE and run additional analyses.

Problem solving

If error messages are reported, you can investigate the problem by reviewing the log files:

  • ...\projects\LuaJIT\log.log
  • ...\projects\LuaJIT\revision--<date>-<time>\log\build.log
  • ...\projects\LuaJIT\revision--<date>-<time>\log\import.log
  • ...\projects\LuaJIT\revision--<date>-<time>\log\analysis.log

Before rerunning the bootstrap process, enable prototyping mode. This will help with any further troubleshooting you need to do.

What next?

Spend some time investigating the analysis of the LuaJIT code base. This will help you familiarize yourself with the standard rules available for C analysis.

Query the snapshot in your IDE

If you install a QL plugin or extension, you can easily write custom queries to analyze snapshots and view the results directly in your IDE. 

Create more snapshots

Use the bootstrap tool to create a project for your own code base.

Work through the tutorial on advanced project creation to see some examples of the available configuration options.