LGTM Enterprise 1.24

Checking progress for added projects

The Add tab of the Projects administration page has a History section. You can use this to check the status of the projects that you added to LGTM.

The History section shows every batch addition that an LGTM administrator adds using that tab. The most recent batch is listed first.

Viewing progress for a batch

To view the progress information for a batch addition, click the appropriate arrow button in the History section.

The Add projects page shows the following status information at the top of the page:

  • Batch creation—date and time the batch of projects was added to LGTM.
  • Last updated—date and time the information was most recently updated.

Initially this page is set to auto-reload the progress of the jobs that are running for the listed project(s). The details on the page change as jobs complete. If you leave the page and then return to it later, auto-reloading is turned off. Click Auto-reload if you want the page to update automatically.

The page also shows one or both of the following sections:

  • Attempted—this section has a filter control followed by a table. There's one row in the table for every repository URL that matched an integration, and for which attempt-build jobs have been queued.
  • Could not be attempted—where present, this section contains a table with one row for every repository URL where it wasn't possible to start the attempt-build jobs. For example, because the host failed to respond.

Monitoring attempt-build jobs

The table in the Attempted section contains one row for each project you included in this batch. The language columns include all of the languages you asked LGTM to attempt to build. The table shows you the status of each attempt-build job, for each language, for each project:

Click to open the repository URL in a new browser tab.

The status is indicated as follows:

  • job queued or currently running
  • no code of this language was found
  • job completed successfully
  • job failed

If at least one language was analyzed successfully, the repository is registered as an LGTM project. You can see the full log information for completed jobs by clicking the green check mark, red cross, or gray dash.

In the example above, you can see that LGTM has successfully analyzed JavaScript code for the first project. It is still running the attempt-build job for Python. None of the other languages were found in the repository for this project. The project name (yarnpkg/yarn) has been changed to a link. Click this to display the newly created project administration page. LGTM is currently running, or has not yet started, attempt-build jobs for the other three projects in this batch.

Code not found

Where projects use only one or two programming languages, you'll see a lot of gray dashes in the table, unless you cleared the selection of some languages in the Advanced options section when you added the projects. For example, the table below shows a batch of four projects. LGTM did not find any C/C++, C#, or Go code in any of these projects. All of the projects have been added to LGTM as they all contain at least one language that could be analyzed. One error is shown. The apache/ant contains Java which, in this example, LGTM has not been able to analyze.

Failed attempt-build jobs

The main reasons for attempt-build jobs to fail are:

  1. There was a problem checking out the code (all jobs for that repository will fail).
  2. The environment isn't set up correctly for that project and some dependencies are missing.
  3. For compiled languages, LGTM is unable to build the project using the global configuration.

Filtering the list for failed jobs

If you have a long list of projects and you just want to see those where an attempt-build job failed, you can use the Failure classification filter. When you make a selection from the drop-down list, additional controls are displayed. Select a language to filter on, if required, then click Apply.

For example, choose Any failure from the classification list, and a specific language to see all failures for a specific language:

An attempt-build job may fail for a language even where LGTM did not find any lines of code written in that language. You can hide these failures by clearing the selection of the Include errors for 0 estimated LOC languages check box, as shown above.

Retrying failed jobs

Click the red cross for a failed job to display the full log information. The Log details page is displayed (for general information about this page, see Exploring work pool logs).

The stdout and stderr generated by the job are displayed (truncated if the output is long). Click the Show link to see the full output as raw text. You can retry any project using the Retry button on the History page (see image above).

  1. On the History page, click the Retry button for the attempted project to display a Retry project page.

  2. In the Manual configuration section, add a YAML extraction block to customize building and analyzing the code. For details of what to enter here, see the topics on customizing extraction code and the lgtm.yml configuration file in the user help.

  3. When you've finished defining the configuration, click Retry.

  4. A new set of attempt-build jobs is queued for the project, using the customized project configuration, and the History page is redisplayed.

Alternatively, if you have filtered the list to show only projects with failures, and you want to retry all of these, click Retry all. LGTM will attempt to add the projects again, using the default project configuration.

When you click Retry or Retry all, LGTM performs a new attempt-build job for each language that was selected for analysis when the project, or batch of projects, was originally added—including languages that were built successfully at the previous attempt.

Example: Java build failure

In the example above, the Java code for the Apache Ant project failed to build. The stdout includes the following error:

[2019-06-28 09:31:37] [build] [2019-06-28 09:31:37] [autobuild] > ant -noinput -f "build.xml" clean "main"
[2019-06-28 09:31:38] [build] [2019-06-28 09:31:38] [autobuild] Picked up JAVA_TOOL_OPTIONS: -Djava.io.tmpdir="/var/cache/lgtm/workers/1/private"
[2019-06-28 09:31:38] [build] [2019-06-28 09:31:38] [autobuild] Running '/var/cache/lgtm/workers/1/downloads/setup_dist/fdfd7276708505ad41e0c268bed67a37.unzipped/odasa/tools/preload_tracer ant -noinput -f build.xml clean main' with tracer /var/cache/lgtm/workers/1/downloads/setup_dist/fdfd7276708505ad41e0c268bed67a37.unzipped/odasa/tools/preload_tracer
[2019-06-28 09:31:38] [build] [2019-06-28 09:31:38] [autobuild] Runner failed to start 'ant': No such file or directory
[2019-06-28 09:31:38] [build] [2019-06-28 09:31:38] [autobuild] [2019-06-28 09:31:38] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/var/cache/lgtm/workers/1/downloads/setup_dist/fdfd7276708505ad41e0c268bed67a37.unzipped/odasa/tools/preload_tracer, ant, -noinput, -f, build.xml, clean, main])

The Java autobuild process failed to find and start ant. The Ant project can only be built using the latest version of the Ant build system. Even if Ant is already installed on the worker machines, this job will fail. You need to use a YAML extraction block in the project configuration to specify a build command that will bootstrap the Ant code.

For example:

extraction:
  java:
    before_index:
      - export ANT_HOME="${LGTM_SRC}/bootstrap"
      - export PATH="${PATH}:${ANT_HOME}/bin"
      - ./bootstrap.sh

This block adds an extra step before the standard LGTM index step. The before_index step runs the following commands:

  1. Define the ANT_HOME environment variable as the bootstrap folder in the directory where LGTM has downloaded the source code. This tells LGTM where to find Ant.
  2. Add the Ant executable to the path.
  3. Run the bootstrap.sh script. This generates a folder called bootstrap that contains a basic version of Ant .

Now when the index step runs, it will use the bootstrapped version of Ant to build the full codebase.

In this example, the worker daemons are Linux-based so the commands are written to run under bash. For Windows-based workers, the commands run in the Windows command shell. They also need to set Windows environment variables and to use the bootstrap.bat script instead of bootstrap.sh. That is:

extraction:
  java:
    before_index:
      - set ANT_HOME="%LGTM_SRC%/bootstrap"
      - set PATH="%PATH%;%ANT_HOME%/bin"
      - ./bootstrap.bat

Related topicsRelated topics