jimklimov

NUT CI farm: where the time goes

Where time goes on the NUT CI farm (ci.networkupstools.org, job nut/nut), from the builds that started in the 45 days to 2026-10-28. Every number here comes from nutci-stats run on that data; its full output is in report-2026-10-28.txt. These issues came out of it:

• networkupstools/jenkins-dynamatrix#56: the git-cache lock covers the whole workspace checkout
• networkupstools/nut#3810: skip allowed-failure cells on ordinary PR builds
• networkupstools/nut#3811: a Jenkins restart fails every queued and running build
• networkupstools/nut#3812: the Jenkins spellcheck and shellcheck quick tests vs GitHub Actions
• a comment on networkupstools/nut#3725 (new hosting) with capacity numbers

jimklimov

nutci-stats

Where the time goes on a jenkins-dynamatrix CI farm, such as the NUT CI farm at ci.networkupstools.org. It fetches build data for one multibranch job from Jenkins, then reports on it. One file, Python 3.8+, standard library only.

nutci-stats.py fetch --until 2026-10-28 --days 45     # into ./nutci-data
nutci-stats.py queue      # orchestrator wait, and how builds ended
nutci-stats.py capacity   # agent-hours and builds a day per agent pool
nutci-stats.py entries    # agent-hours per slowBuild entry
nutci-stats.py unstash # time in "Unstash sources", and how much is paused

jimklimov

Kudos to https://www.reddit.com/r/linux/comments/1y8qg2/can_i_force_a_process_to_flush_a_file_descriptor/

Re-confirmed the trick with Ubuntu 26.04 - answer is still relevant, thanks!

FWIW, reproduced with a simple program that prints a message without \n so it is not flushed until the process is closed (or buffer fills up), and sleeps:

// ppp.c
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

jimklimov

Completeness Audit of Client Implementations

The client implementations in libupsclient (C), libnutclient (C++), PyNUT (Python), Nut.pm (Perl), and jNut (Java) were audited against the NUT network protocol defined in docs/net-protocol.txt.

1. C Library: clients/upsclient.c (libupsclient)

The C library is the most mature and complete implementation, used by most core NUT utilities (upsc, upscmd, upsrw, upsmon).

• Commands: Supports all protocol commands: GET, LIST, SET, INSTCMD, LOGIN, LOGOUT, USERNAME, PASSWORD, VER, NETVER, HELP, STARTTLS, PRIMARY/MASTER, FSD.
• Sub-commands: Full support for GET (VAR, TYPE, DESC, CMDDESC, UPSDESC, NUMLOGINS) and LIST (UPS, VAR, RW, CMD, ENUM, RANGE, CLIENT).
• SSL/TLS: Comprehensive support for both OpenSSL and Mozilla NSS backends via upscli_sslinit and upscli_tryconnect.
• Tracking: Implements protocol 1.3 TRACKING support (GET/SET TRACKING).

jimklimov

#!/bin/sh

# Toggle Jenkins agent(s) running on local system on and off
# (e.g. free up CPU when needed for something else temporarily).
#
# (C) 2026 Jim Klimov <jimklimov+nut@gmail.com>
#
# Posted at https://gist.github.com/jimklimov/e8841ff5bbcf6a23a8ece7fe7cb53eb9
# Inspired by discussion at
# https://stackoverflow.com/questions/61389821/get-running-build-status-on-a-given-jenkins-worker-node

jimklimov

Specifying a zero size for filesystem lets LXC backend use the "subvol" directories rather than disk images. The latter were problematic for "pxvirt" 9.0.10-2 on Raspberry/trixie with ZFS (there raw ext4 images are okay for direct fsck/mount, until PVE mounts them and the system locks up with timed-out processes).

GUI may forbid specifying 0 as disk size, CLI to the rescue then.

Thanks to https://forum.proxmox.com/threads/proxmox-4-lxc-chroot-instead-of-raw-images.22881/ for CLI examples:

pct create 110 /var/lib/vz/template/cache/ubuntu-14.04-standard_14.04-1_amd64.tar.gz -hostname servername -password 'p@ssw0rd' -rootfs local:0

Comparing two setups:

jimklimov

<!DOCTYPE html><html>

<link rel="stylesheet" href="https://cdn.oaistatic.com/assets/root-dxedo8gg.css"/><link rel="stylesheet" href="https://cdn.oaistatic.com/assets/conversation-small-c80izxqg.css"/>

<body style="background-color:#404040">

<div style="background-color:#404040" class="flex flex-col text-sm thread-xl:pt-header-height pb-25"><article class="text-token-text-primary w-full focus:outline-none [--shadow-height:45px] has-data-writing-block:pointer-events-none has-data-writing-block:-mt-(--shadow-height) has-data-writing-block:pt-(--shadow-height) [&amp;:has([data-writing-block])&gt;*]:pointer-events-auto scroll-mt-(--header-height)" tabindex="-1" dir="auto" data-turn-id="76b36e2a-fced-457d-9e2b-7435b6d37214" data-testid="conversation-turn-1" data-scroll-anchor="false" data-turn="user"><h5 class="sr-only">You said:</h5><div class="text-base my-auto mx-auto pt-3 [--thread-content-margin:--spacing(4)] thread-sm:[--thread-content-margin:--spacing(6)] thread-lg:[--thread-content-margin:--spacing(16)] 

jimklimov

// From https://community.jenkins.io/t/random-403-invalid-crumb-messages-through-out-jenkins/35619/4
// by @poddingue - Bruno Verachten, GSoC Mentors 

import jenkins.model.Jenkins
import hudson.security.csrf.CrumbIssuer
import jenkins.security.s2m.AdminWhitelistRule

def j = Jenkins.instance

println "=== Jenkins Security & CSRF Diagnostics ==="

jimklimov

// requires Simple Queue Plugin : https://plugins.jenkins.io/simple-queue/
// run this in Jenkins console to generate shell code
// create a token in sysadmin account, save as e.g. `~/.sq` in a trusted workstation
// and in a shell session define JENKINS_URL="https://USERNAME:`cat ~/.sq`@SERVERNAME"
// then paste the result of script below (tweak "it" matcher to your needs)

def q = Jenkins.instance.queue
def pattern = ~/.*jobname.*1234.*/
def ACTION = "DOWN_FAST"
//def ACTION = "UP_FAST"

jimklimov

#!/bin/sh

# https://superuser.com/questions/1582234/make-ip-address-of-wsl2-static/1808525#1808525
# https://superuser.com/a/1723531/433945
# Add to /etc/wsl.conf as boot/command="/root/network-start.sh" for autostart
# (C) 2023-2025 by Jim Klimov (and other SO/GH/blog authors)
#
# This script assigns a fixed IP to the VM, fixes up DNS,
# allows fixed port on host for SSH dial-back into the VM
# and starts the SSH server.