LGTM Enterprise 1.21.1

Assessing worker resourcing

LGTM uses worker daemons—"workers" for short—to perform the "heavy lifting" involved in building each revision of a project and running analysis on it.

Use the charts on the Infrastructure administration page to assess whether you have too few, or too many, of these workers.

Alternatively, you can use the API to check the current number and status of workers. For more information, see Monitoring the health of the application.

How many workers are currently running?

The Current workers table at the top of the General infrastructure tab shows you how many workers are running and what they are currently doing. Typically workers are either executing jobs or they are idle, awaiting an appropriate job to process (see Worker types and Defining which workers can build and analyze a project).

How busy have the workers been?

The three charts on the General infrastructure tab show you the status of your workers over the past seven days. There's a separate chart for each of the three types of worker. The status of each worker (booting, executing or idle) is recorded at regular intervals, giving a clear picture of how busy the workers have been.

If LGTM is processing historical commits, the general workers are likely to be executing almost all of the time. However, the charts for on-demand and query workers should not show extended periods where all of the workers were busy. Results from queries submitted in the Query console are likely to take longer to be displayed if your query workers are constantly busy. Similarly, if your on-demand workers show little or no idle time (or if you have no on-demand workers and your general workers are constantly busy) this may result in code review analysis results taking longer than necessary to be posted to pull requests.

Do you need to add more workers?

In addition to the worker status charts (described above), the charts on the Job queues tab provide information that lets you assess whether you're running enough worker daemons.

LGTM uses a "job dealer" to queue job instructions for the workers. When a worker starts, or when it finishes a job, it polls the job dealer for the details of a job to process. The job dealer assigns an appropriate job to a worker based on a combination of the worker type and any labeling you have configured (see Defining which workers can build and analyze a project). If a job remains queued for a long time it indicates that there were no suitable workers available to process that job. Adding more workers of an appropriate type, or with an appropriate label, is likely to reduce the time that jobs wait before being processed. See Adding and removing workers.

Job queue waiting times

Use the Job latency chart to see how long jobs have waited in a queue before being picked up by a worker for processing.

The chart shows information about waiting times for the last seven days. Data is added every hour, covering an hour-long period. The data relates to jobs that finished during the period. Note that a job "finishes" when a worker stops processing the job, regardless of whether the processing was successful.

Three measures are calculated: 

  • The number of jobs that finished during the hour represented by the data point.

  • The maximum time any of those jobs was queued before a worker began to process it.

  • The mean amount of time those jobs were queued before processing began.

Hover the mouse pointer over this chart to see the specific details of each data point. The time displayed is the start of the hour-long period that the data represents.

Job queue sizes

Use the Pending jobs chart to see the number of on-demand and query jobs that were queued for processing each hour during the past seven days.

Different types of worker are used to process on-demand and query jobs.

Example

For example, let's suppose the Job latency chart shows long delays for some jobs and the Pending jobs chart shows on-demand jobs being queued for an extended period of time. If you now check the On-demand workers chart on the General infrastructure tab and it shows that your on-demand workers have been constantly busy executing jobs, this strongly indicates that your system is under-resourced for processing this type of work. Adding more on-demand workers should reduce the time it takes to see analysis results.

See Adding and removing workers.

Related topicsRelated topics