LGTM Enterprise 1.22

Defining which workers can build and analyze a project

LGTM uses daemon processes, known as "workers," to build and analyze each revision of a project. This produces a variety of data, such as code quality alerts (see Code analysis in the user help). Each worker needs to run in an environment in which it can successfully build the project revision it is being asked to analyze.

All projects have dependencies, and these vary from project to project. Dependencies can include: a specific build tool (or a specific version of that build tool), a specific set of libraries, a particular operating system, or a minimum amount of available memory. If these requirements are not met then the code will not build. If you have projects that have very different, perhaps conflicting, dependencies then you can set up specific environments on individual worker host machines. You can then use labels to specify that code from a particular project—or all code written in a particular language—is processed by the workers running on a specific machine.

What are labels?

Labels are simply arbitrary text strings. They have no inherent meaning in themselves; they're just a way of matching up projects to workers. When you're choosing the text to use as a label, you should try to come up with something that will clearly indicate a feature that determines where a project should be processed—for example, maven3-6 for projects that will only build successfully using Apache Maven 3.6.

Labels can contain the following characters:

a–z     0–9    - (hyphen)    _ (underline)

Labels cannot contain uppercase letters.

Where are labels applied?

Labels can be applied to:

  • Workers running on a particular host machine
  • Individual programming languages
  • Code in a specific language from an individual project

You define labels for particular worker host machines in the cluster configuration file.

You apply language-wide labels and project-specific labels in the LGTM administration interface.

Before defining language-wide and project-level labels you must check what labels are available on workers (see below). Applying a label that has not been assigned to any worker, will prevent code from being analyzed.

The code of a specific language in a project gains all of the labels defined at the project level plus any labels defined globally for that language.

How labels work

Workers build and analyze revisions of the code in a particular language from an individual project—one revision at a time. Each time a worker is free to process another such job it requests one from the LGTM coordinator server. LGTM then compares the labels possessed by the worker with the labels of the jobs that are waiting to be processed, and passes an appropriate job to the worker.

For a worker to be assigned a job to process, the worker must have all of the labels associated with the code in that job (that is, all of the labels directly assigned at the project level, plus all of the labels assigned at the global language level). The worker can have additional labels that are not associated with the code in that job, but if it does not have all of the project/language labels it will not be used to process the code of that language from that project.

Assigning labels to workers

Each worker has an automatically assigned label to indicate its operating system and, optionally, labels assigned as part of the configuration of worker host machines.

Automatically assigned worker labels

Each worker is automatically assigned one of the following labels:

  • windows—if the host machine is running Windows
  • unix—if the operating system is not Windows

A typical use-case for labels is to specify that certain projects can only be built—or cannot be built—on Windows. Automatic worker labeling means that all you need to do to achieve this is to assign the windows or unix labels at the language-wide level and/or project level in the administration interface.

Assigning worker labels manually

You can assign labels to workers by editing the cluster configuration file (lgtm-cluster-config.yml) and then deploying the updated configuration.

For full details of the cluster configuration file, see lgtm-cluster-config.yml file.

For details of how to edit and redeploy your cluster configuration, see Editing the cluster configuration file.

The code sample below shows an extract from a cluster configuration file. It shows the workers section of the configuration, in which four host machines are specified. The first of these, with the host name lgtmcontrol, runs a single worker daemon to process queries submitted in LGTM's query console. Each of the other three machines runs two workers.

  • The workers on workerhostA are not assigned labels manually.
  • The workers on workerhostB get the labels highmemory and maven3-6.
  • The workers on workerhostC get a single label: pythonenv.
workers:
  hosts:
  - hostname: "lgtmcontrol"
    specs:
    - type: "QUERY"
      copies: 1
  - hostname: "workerhostA"
    specs:
    - type: "GENERAL"
      copies: 2
  - hostname: "workerhostB"
    specs:
    - type: "GENERAL"
      copies: 2
    labels:
    - highmemory
    - maven3-6
    environment:
      LGTM_RAM: "8400"
  - hostname: "workerhostC"
    specs:
    - type: "GENERAL"
      copies: 2
    labels:
    - pythonenv

After editing the cluster configuration file and adding labels for worker hosts, you must generate and deploy updated files for the configuration to the machines in the work pool. For details, see Editing and deploying the cluster configuration.

Checking which labels have been assigned to workers

The Worker management tab on the Infrastructure page of the administration interface shows the labels that are currently assigned to the worker daemons on each worker host.

Applying a label to a language

To ensure that all analysis of a particular language is carried out by workers that have a particular label, add a label to the language:

  1. Go to the Code analysis administration page.

  2. In the relevant language section, type the name of a label into Worker labels field.

    The labels you add must match labels assigned to one or more workers.

  3. Add more labels, if required.

  4. Click Update <language> settings.

    A confirmation message is displayed at the top of the page: "Labels updated."

    Code written in this language will only be processed by workers that have the labels you specified.

Applying a label directly to an individual project

To ensure that the code from a specific project is processed on a machine where particular build requirements are met, you can assign one or more labels directly to the project.

You can specify labels for an existing project in LGTM Enterprise, or when you add projects using the Add tab in the Projects administration page.

Applying labels when adding projects

The Worker labels box in the Advanced options section of the Add tab, allows you to apply labels to projects. The projects you add will only be processed by workers that have all of the labels you specify (plus any relevant language labels that you have configured).

For example, if you specify the labels highmemory and maven3-6 when you bulk add some projects, then those projects will only be processed by workers that have both of these labels.

For more information about bulk adding projects, see Adding projects.

Applying labels to existing projects

  1. Go to the Projects administration page.

  2. Click the project to which you want to assign a label.

    If your system analyzes a lot of projects, you can search for the project by its name.

  3. Click the Analysis settings tab.

    This tab contains a section for adding labels. There is a label field for each language in the project that is analyzed by LGTM.

    If you used the Projects administration page to add the project, and you specified worker labels, then those labels will be displayed in the field for any languages that have been successfully analyzed.

  4. In the language field that corresponds to the code in this project that you want to label, type the name of a label.

  5. Add more labels, if required.

  6. Click Save.

    A confirmation message is displayed at the top of the page: "Updated worker labels for: <languages>."

    Code from this project, in the relevant language, will only be processed by workers that have been assigned all of the labels you specified (plus any labels applied globally across this language).

When labels take effect

Changes to the labels defined in the cluster configuration file are applied to worker hosts when you generate and deploy the files for the updated configuration. For details, see Editing and deploying the cluster configuration.

Changes to the language-wide and project-specific labels defined in the administration interface are applied to all new jobs that get queued for processing after you save the label changes. Jobs that are already queued for processing when you add or remove labels are not affected by the changes.

Example

The diagram below represents a situation where three virtual machines have been set up to run workers. The job of the workers is to build and analyze revisions of seven projects, each of which contains code written in a single programming language. Labels have been assigned to determine which workers can be used to process revisions for each project.

In this example, the three virtual machines each run two worker daemons. Worker host B has a lot of RAM and, consequently, its worker daemons are assigned the label highmemory in the cluster configuration file. Worker host C has been set up with the dependencies necessary for Python projects, so its worker daemons are labeled pythonenv.

There are seven projects, named P1 to P7. Projects P4 and P7 have been labeled highmemory, in the LGTM administration interface, to ensure they are only processed on worker host B.

Also in the administration interface, the label windows has been assigned to C# to ensure that all C# projects are processed on worker host A. This occurs because worker host A is running Windows, so all workers on this machine are automatically given the label windows. Python has been given the label pythonenv to ensure that all Python projects are processed on worker host C.

* Project P4 has the label highmemory and, being a Python project, also gets the label pythonenv. However, none of the worker daemons has both of these labels, so attempt-builds, build and build-export jobs for this project will fail. The Jobs tab of the Logs page will list these jobs with the status "Returned From Queue (NO_SUITABLE_WORKER)," and the Labels field will be displayed, showing the labels that the job required a worker to possess:

To fix this problem, either remove the highmemory label from this project, or reconfigure the workers so that at least one of them has both of these labels. For more details, see Fixing NO_SUITABLE_WORKER errors.

The following table provides a quick view of the machines on which projects, from the example above, are built and analyzed.

Project Language Project + language labels Processed by workers on host(s)
P1 C# windows Worker host A
P2 C unix Worker host B
Worker host C
P3 Python pythonenv Worker host C
P4* Python highmemory
pythonenv
*(Jobs for this project will not be processed and will be logged as having failed.)
P5 JavaScript Worker host A
Worker host B
Worker host C
P6 JavaScript Worker host A
Worker host B
Worker host C
P7 Java highmemory Worker host C

Related topicsRelated topics