# Run sent to worker: doanac-reckless == 2026-03-18 15:51:00.419911: Setting up runner on worker == 2026-03-18 15:51:02.434249: Steps to recreate inside simulator wget -O simulate.sh https://api.gavelci.us/projects/jobserv/builds/503/runs/flake8//.simulate.sh # wget'ing the file may require the --header flag if the # jobserv API requires authentication. sh ./simulate.sh == 2026-03-18 15:51:03.686065: Pulling container: python:3.11-alpine3.19 3.11-alpine3.19: Pulling from library/python Digest: sha256:135b0db9b7c3e33d654ba8893b63cbe9394768cc64fc2a2f283148125ae8aa1f Status: Image is up to date for python:3.11-alpine3.19 docker.io/library/python:3.11-alpine3.19 == 2026-03-18 15:51:06.993344: Preparing bind mounts 2026-03-18 15:51:07.420932: INFO Default docker configuration detected 2026-03-18 15:51:07.844540: INFO Creating secret: githubtok 2026-03-18 15:51:08.278528: INFO Creating secret: webhook-key 2026-03-18 15:51:08.722731: INFO Creating secret: milosz == 2026-03-18 15:51:09.693688: Creating container .netrc file 2026-03-18 15:51:10.120328: INFO Creating token for jobserv run access 2026-03-18 15:51:10.577446: INFO Creating a github token entry == 2026-03-18 15:51:11.444926: Preparing script == 2026-03-18 15:51:12.362558: Cloning git repository 2026-03-18 15:51:12.815740: INFO Clone_url: https://github.com/foundriesio/jobserv.git 2026-03-18 15:51:13.265497: INFO Checking to see if https://github.com/foundriesio/jobserv.git requires authentication. 2026-03-18 15:51:14.270631: INFO Server does not appear to need credentials for cloning 2026-03-18 15:51:14.477441: INFO Adding githubtok to .gitconfig 2026-03-18 15:51:14.717850: INFO Git install supports submodules Cloning into '/srv/gavelci-worker/runs/tmpohqx36vk/repo'... 2026-03-18 15:51:16.730866: INFO Checking out: 00e1fb042c1fce714c4a412d6420278d03d2db25 Switched to branch 'jobserv-run' == 2026-03-18 15:51:17.779461: Setting up container environment 2026-03-18 15:51:18.113101: INFO Container environment variables: GH_PRNUM=155 GH_OWNER=foundriesio GH_REPO=jobserv GH_BRANCH=master GH_STATUS_URL=https://api.github.com/repos/foundriesio/jobserv/statuses/00e1fb042c1fce714c4a412d6420278d03d2db25 GH_TARGET_REPO=https://github.com/foundriesio/jobserv.git GIT_URL=https://github.com/foundriesio/jobserv.git GIT_SHA_BASE=d287b065c8bec8035e1ec51f75b386b5e08c0ea1 GIT_OLD_SHA=d287b065c8bec8035e1ec51f75b386b5e08c0ea1 GIT_SHA=00e1fb042c1fce714c4a412d6420278d03d2db25 H_PROJECT=jobserv H_BUILD=503 H_RUN=flake8 H_RUN_URL=https://api.gavelci.us/projects/jobserv/builds/503/runs/flake8/ H_WORKER=doanac-reckless == 2026-03-18 15:51:18.881331: Running script inside container 2026-03-18 15:51:19.405251: INFO Running with --memory=14639596339 WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. + apk add musl-dev gcc fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz (1/14) Upgrading musl (1.2.4_git20230717-r4 -> 1.2.4_git20230717-r5) (2/14) Installing libgcc (13.2.1_git20231014-r0) (3/14) Installing jansson (2.14-r4) (4/14) Installing libstdc++ (13.2.1_git20231014-r0) (5/14) Installing zstd-libs (1.5.5-r8) (6/14) Installing binutils (2.41-r1) (7/14) Installing libgomp (13.2.1_git20231014-r0) (8/14) Installing libatomic (13.2.1_git20231014-r0) (9/14) Installing gmp (6.3.0-r0) (10/14) Installing isl26 (0.26-r1) (11/14) Installing mpfr4 (4.2.1-r0) (12/14) Installing mpc1 (1.3.1-r1) (13/14) Installing gcc (13.2.1_git20231014-r0) (14/14) Installing musl-dev (1.2.4_git20230717-r5) Executing busybox-1.36.1-r19.trigger OK: 166 MiB in 50 packages + pip3 install -r ./requirements-lint.txt Collecting black==26.3.1 (from -r ./requirements-lint.txt (line 1)) Downloading black-26.3.1-py3-none-any.whl.metadata (91 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.9/91.9 kB 67.2 kB/s eta 0:00:00 Collecting flake8-junit-report==2.1.0 (from -r ./requirements-lint.txt (line 2)) Downloading flake8-junit-report-2.1.0.tar.gz (3.1 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting click>=8.0.0 (from black==26.3.1->-r ./requirements-lint.txt (line 1)) Downloading click-8.3.1-py3-none-any.whl.metadata (2.6 kB) Collecting mypy-extensions>=0.4.3 (from black==26.3.1->-r ./requirements-lint.txt (line 1)) Downloading mypy_extensions-1.1.0-py3-none-any.whl.metadata (1.1 kB) Collecting packaging>=22.0 (from black==26.3.1->-r ./requirements-lint.txt (line 1)) Downloading packaging-26.0-py3-none-any.whl.metadata (3.3 kB) Collecting pathspec>=1.0.0 (from black==26.3.1->-r ./requirements-lint.txt (line 1)) Downloading pathspec-1.0.4-py3-none-any.whl.metadata (13 kB) Collecting platformdirs>=2 (from black==26.3.1->-r ./requirements-lint.txt (line 1)) Downloading platformdirs-4.9.4-py3-none-any.whl.metadata (4.7 kB) Collecting pytokens~=0.4.0 (from black==26.3.1->-r ./requirements-lint.txt (line 1)) Downloading pytokens-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl.metadata (3.8 kB) Downloading black-26.3.1-py3-none-any.whl (207 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 207.5/207.5 kB 1.3 MB/s eta 0:00:00 Downloading click-8.3.1-py3-none-any.whl (108 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.3/108.3 kB 792.6 kB/s eta 0:00:00 Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) Downloading packaging-26.0-py3-none-any.whl (74 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.4/74.4 kB 708.2 kB/s eta 0:00:00 Downloading pathspec-1.0.4-py3-none-any.whl (55 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.2/55.2 kB 217.3 kB/s eta 0:00:00 Downloading platformdirs-4.9.4-py3-none-any.whl (21 kB) Downloading pytokens-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl (264 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 264.1/264.1 kB 2.3 MB/s eta 0:00:00 Building wheels for collected packages: flake8-junit-report Building wheel for flake8-junit-report (setup.py): started Building wheel for flake8-junit-report (setup.py): finished with status 'done' Created wheel for flake8-junit-report: filename=flake8_junit_report-2.1.0-py3-none-any.whl size=3736 sha256=c128109ea579b40de16ec1347d7dbdd0cbbadb568a837bbf012419d14fe8344c Stored in directory: /root/.cache/pip/wheels/61/49/33/d490b7f9a42bd5abfb880d2af6c9e97ab4c0e6eb3441a38c19 Successfully built flake8-junit-report Installing collected packages: flake8-junit-report, pytokens, platformdirs, pathspec, packaging, mypy-extensions, click, black Successfully installed black-26.3.1 click-8.3.1 flake8-junit-report-2.1.0 mypy-extensions-1.1.0 packaging-26.0 pathspec-1.0.4 platformdirs-4.9.4 pytokens-0.4.1 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 24.0 -> 26.0.1 [notice] To update, run: pip install --upgrade pip + black --check --diff ./jobserv ./runner ./tests ./jobserv_worker.py --- /repo/jobserv/storage/__init__.py 2026-03-18 15:51:16.723961+00:00 +++ /repo/jobserv/storage/__init__.py 2026-03-18 15:51:52.452498+00:00 @@ -3,7 +3,6 @@ from importlib import import_module from jobserv.settings import STORAGE_BACKEND - Storage = import_module(STORAGE_BACKEND).Storage would reformat /repo/jobserv/storage/__init__.py --- /repo/jobserv/worker_jwt.py 2026-03-18 15:51:16.723961+00:00 +++ /repo/jobserv/worker_jwt.py 2026-03-18 15:51:52.864825+00:00 @@ -18,11 +18,10 @@ Encoding, ) import jwt from jobserv.settings import WORKER_JWTS_DIR - _keys: Dict[str, Certificate] = {} def _keyid(cert: Certificate) -> str: would reformat /repo/jobserv/worker_jwt.py --- /repo/jobserv/notify.py 2026-03-18 15:51:16.723961+00:00 +++ /repo/jobserv/notify.py 2026-03-18 15:51:52.865275+00:00 @@ -142,13 +142,11 @@ stats = _get_build_stats(build) body += """Build history for last {total} builds: pass rate: {pass_rate}% (newest->oldest): {pass_fails} - """.format( - **stats - ) + """.format(**stats) msg = MIMEText(body) msg["Subject"] = subject msg["From"] = SMTP_USER msg["To"] = to_list would reformat /repo/jobserv/notify.py --- /repo/jobserv/api/run.py 2026-03-18 15:51:16.723961+00:00 +++ /repo/jobserv/api/run.py 2026-03-18 15:51:53.135323+00:00 @@ -359,13 +359,11 @@ {rundef} EIEIO wget -O runner {runner} PYTHONPATH=./runner python3 -m jobserv_runner.simulator -w `pwd` rundef.json - """.format( - rundef=rundef_str, runner=runner - ) + """.format(rundef=rundef_str, runner=runner) return script, 200, {"Content-Type": "text/plain"} @blueprint.route("//", methods=("GET",)) def run_get_artifact(proj, build_id, run, path): would reformat /repo/jobserv/api/run.py --- /repo/runner/jobserv_runner/handlers/simple.py 2026-03-18 15:51:16.727961+00:00 +++ /repo/runner/jobserv_runner/handlers/simple.py 2026-03-18 15:51:54.230917+00:00 @@ -445,13 +445,11 @@ msg = """ wget -O simulate.sh {run}/.simulate.sh # wget'ing the file may require the --header flag if the # jobserv API requires authentication. sh ./simulate.sh -""".format( - run=self.rundef["run_url"] - ) +""".format(run=self.rundef["run_url"]) log._write(msg) def prepare_mounts(self): """Prepare the directories we will bind mount by docker.""" with self.log_context("Preparing bind mounts") as log: would reformat /repo/runner/jobserv_runner/handlers/simple.py --- /repo/jobserv/models.py 2026-03-18 15:51:16.723961+00:00 +++ /repo/jobserv/models.py 2026-03-18 15:51:54.247197+00:00 @@ -530,21 +530,17 @@ # run to RUNNING. If we see this change, we can mark it FAILED. # If we don't see the change, its RUNNING and needs CANCELLING. conn = db.session.connection().connection cursor = conn.cursor() - rows = cursor.execute( - """ + rows = cursor.execute(""" UPDATE runs SET _status = 2 WHERE id = {run_id} - """.format( - run_id=self.id - ) - ) + """.format(run_id=self.id)) db.session.commit() if rows == 1: # It was us self.set_status(BuildStatus.FAILED) else: @@ -573,12 +569,11 @@ # So we first find a suitable Run: conn = db.session.connection().connection cursor = conn.cursor() # Find queued(status=1), running(status=2), and uploading(status=6) runs - cursor.execute( - """ + cursor.execute(""" SELECT runs.id, runs.build_id, runs._status, projects.id, projects.synchronous_builds, runs.host_tag FROM runs JOIN builds on builds.id = runs.build_id @@ -586,12 +581,11 @@ WHERE runs._status in (1, 2, 6) ORDER BY runs._status DESC, runs.queue_priority DESC, runs.build_id ASC, runs.id ASC - """ - ) + """) tags = [worker.name] + [x.strip() for x in worker.host_tags.split(",")] # By ordering the query above by Run._status, we'll get the active # runs first so that we can build up this list of build ids that are # active for synchronous projects runs with these build ids are okay @@ -629,21 +623,17 @@ # fight the race condition where two threads might schedule the same # run to two different workers. The first worker will get the run, # the second worker won't see a row change, and won't schedule anything # This means the worker will have to check in again to find work # (if any) - rows = cursor.execute( - """ + rows = cursor.execute(""" UPDATE runs SET _status = 2 WHERE id = {run_id} - """.format( - run_id=run_id - ) - ) + """.format(run_id=run_id)) db.session.commit() if rows == 1: # Critical Section! # If any of this fails - we'll have a run in RUNNING, # but no assigned worker. It will be blocked from working. would reformat /repo/jobserv/models.py --- jobserv_worker.py 2026-03-18 15:51:16.723961+00:00 +++ jobserv_worker.py 2026-03-18 15:51:55.018694+00:00 @@ -362,23 +362,21 @@ ) _create_systemd_service() p = HostProps() args.server.create_host(p.data) p.cache() - print( - """ + print(""" A SystemD service can be enabled with: sudo cp jobserv.service /etc/systemd/system/ sudo systemctl enable jobserv sudo systemctl start jobserv You also need to add a sudo entry to allow the worker to clean up root owned files from CI runs: echo "$USER ALL=(ALL) NOPASSWD:/bin/rm" | sudo tee /etc/sudoers.d/jobserv -""" - ) +""") def cmd_unregister(args): """Remove worker from server""" args.server.delete_host() would reformat jobserv_worker.py Oh no! 💥 💔 💥 7 files would be reformatted, 56 files would be left unchanged. Script completed with error(s) == 2026-03-18 15:51:57.665059: Finding artifacts to upload Uploading 0 items 0 bytes == 2026-03-18 15:51:58.138581: Runner has completed ________ | o o| Thumbs Down |___/\_|________ | _____|__| | | || |______| | | | | | | | | |_| |_|