Clone of mesa.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Erik Faye-Lund 3d529c1739 gitlab-ci: also build Zink on CI 5 vuotta sitten
..
README.md freedreno: Introduce gitlab-based CI. 6 vuotta sitten
arm.config gitlab-ci/lava: Test Lima driver with dEQP 5 vuotta sitten
arm64.config gitlab-ci/lava: Test Lima driver with dEQP 5 vuotta sitten
create-rootfs.sh gitlab-ci: Move LAVA-related files into top-level ci dir 5 vuotta sitten
cross-xfail-i386 ci: Run tests on i386 cross builds 6 vuotta sitten
debian-arm64-install.sh gitlab-ci: Enable llvmpipe in ARM build jobs 5 vuotta sitten
debian-install.sh gitlab-ci: also build Zink on CI 5 vuotta sitten
debian-stretch-install.sh gitlab-ci: Use newer packages from backports by default 6 vuotta sitten
deqp-default-skips.txt gitlab-ci: Run the GLES2 CTS on llvmpipe. 6 vuotta sitten
deqp-freedreno-a307-fails.txt freedreno/a3xx: Mostly fix min-vs-mag filtering decisions on non-mipmap tex. 5 vuotta sitten
deqp-freedreno-a630-fails.txt gitlab-ci/a630: Drop the MSAA expected failure. 6 vuotta sitten
deqp-freedreno-a630-skips.txt freedreno/ci: Add failing tests to skip list 5 vuotta sitten
deqp-lima-fails.txt st/mesa: Map MESA_FORMAT_RGB_UNORM8 <-> PIPE_FORMAT_R8G8B8_UNORM 5 vuotta sitten
deqp-lima-skips.txt gitlab-ci/lava: Test Lima driver with dEQP 5 vuotta sitten
deqp-llvmpipe-fails.txt gitlab-ci: Run the GLES2 CTS on llvmpipe. 6 vuotta sitten
deqp-panfrost-t760-fails.txt panfrost/ci: Update expectations list 5 vuotta sitten
deqp-panfrost-t760-skips.txt gitlab-ci/lava: Use files to list tests to skip 5 vuotta sitten
deqp-panfrost-t860-fails.txt panfrost/ci: Update expectations list 5 vuotta sitten
deqp-panfrost-t860-skips.txt gitlab-ci/lava: Use files to list tests to skip 5 vuotta sitten
deqp-runner.sh gitlab-ci: Make the test job fail when bugs are unexpectedly fixed. 6 vuotta sitten
deqp-softpipe-fails.txt gitlab-ci: Enable the GLES2/3 CTS on softpipe. 6 vuotta sitten
generate_lava.py gitlab-ci/lava: Test Lima driver with dEQP 5 vuotta sitten
lava-debian-install.sh gitlab-ci: Update kernel for LAVA jobs to 5.4-rc4 5 vuotta sitten
lava-deqp-runner.sh gitlab-ci/lava: Test Lima driver with dEQP 5 vuotta sitten
lava-deqp.yml.jinja2 gitlab-ci/lava: Test Lima driver with dEQP 5 vuotta sitten
lava-gitlab-ci.yml gitlab-ci: refactor out some common stuff for Windows and Linux 5 vuotta sitten
meson-build.bat gitlab-ci: Add a job for meson on windows 5 vuotta sitten
meson-build.sh gitlab-ci: Update the meson cross file for LLVM_VERSION as well 5 vuotta sitten
run-shader-db.sh gitlab-ci: Set the prefix to ./install instead of the DESTDIR. 6 vuotta sitten
scons-build.sh scons: Print a deprecation warning about using scons on not windows 5 vuotta sitten
x86_64-w64-mingw32 gitlab-ci: Add a pkg-config for mingw 5 vuotta sitten

README.md

Mesa testing using gitlab-runner

The goal of the “test” stage of the .gitlab-ci.yml is to do pre-merge testing of Mesa drivers on various platforms, so that we can ensure no regressions are merged, as long as developers are merging code using the “Merge when pipeline completes” button.

This document only covers the CI from .gitlab-ci.yml and this directory. For other CI systems, see Intel’s Mesa CI or panfrost’s LAVA-based CI (src/gallium/drivers/panfrost/ci/)

Software architecture

For freedreno and llvmpipe CI, we’re using gitlab-runner on the test devices (DUTs), cached docker containers with VK-GL-CTS, and the normal shared x86_64 runners to build the Mesa drivers to be run inside of those containers on the DUTs.

The docker containers are rebuilt from the debian-install.sh script when DEBIAN_TAG is changed in .gitlab-ci.yml, and debian-test-install.sh when DEBIAN_ARM64_TAG is changed in .gitlab-ci.yml. The resulting images are around 500MB, and are expected to change approximately weekly (though an individual developer working on them may produce many more images while trying to come up with a working MR!).

gitlab-runner is a client that polls gitlab.freedesktop.org for available jobs, with no inbound networking requirements. Jobs can have tags, so we can have DUT-specific jobs that only run on runners with that tag marked in the gitlab UI.

Since dEQP takes a long time to run, we mark the job as “parallel” at some level, which spawns multiple jobs from one definition, and then deqp-runner.sh takes the corresponding fraction of the test list for that job.

To reduce dEQP runtime (or avoid tests with unreliable results), a deqp-runner.sh invocation can provide a list of tests to skip. If your driver is not yet conformant, you can pass a list of expected failures, and the job will only fail on tests that aren’t listed (look at the job’s log for which specific tests failed).

DUT requirements

DUTs must have a stable kernel and GPU reset.

If the system goes down during a test run, that job will eventually time out and fail (default 1 hour). However, if the kernel can’t reliably reset the GPU on failure, bugs in one MR may leak into spurious failures in another MR. This would be an unacceptable impact on Mesa developers working on other drivers.

DUTs must be able to run docker

The Mesa gitlab-runner based test architecture is built around docker, so that we can cache the debian package installation and CTS build step across multiple test runs. Since the images are large and change approximately weekly, the DUTs also need to be running some script to prune stale docker images periodically in order to not run out of disk space as we rev those containers (perhaps this script).

Note that docker doesn’t allow containers to be stored on NFS, and doesn’t allow multiple docker daemons to interact with the same network block device, so you will probably need some sort of physical storage on your DUTs.

DUTs must be public

By including your device in .gitlab-ci.yml, you’re effectively letting anyone on the internet run code on your device. docker containers may provide some limited protection, but how much you trust that and what you do to mitigate hostile access is up to you.

DUTs must expose the dri device nodes to the containers.

Obviously, to get access to the HW, we need to pass the render node through. This is done by adding devices = ["/dev/dri"] to the runners.docker section of /etc/gitlab-runner/config.toml.

HW CI farm expectations

To make sure that testing of one vendor’s drivers doesn’t block unrelated work by other vendors, we require that a given driver’s test farm produces a spurious failure no more than once a week. If every driver had CI and failed once a week, we would be seeing someone’s code getting blocked on a spurious failure daily, which is an unacceptable cost to the project.

Additionally, the test farm needs to be able to provide a short enough turnaround time that people can regularly use the “Merge when pipeline succeeds” button successfully (until we get marge-bot in place on freedesktop.org). As a result, we require that the test farm be able to handle a whole pipeline’s worth of jobs in less than 5 minutes (to compare, the build stage is about 10 minutes, if you could get all your jobs scheduled on the shared runners in time.).

If a test farm is short the HW to provide these guarantees, consider dropping tests to reduce runtime. VK-GL-CTS/scripts/log/bottleneck_report.py can help you find what tests were slow in a results.qpa file. Or, you can have a job with no parallel field set and:

  variables:
    CI_NODE_INDEX: 1
    CI_NODE_TOTAL: 10

to just run 1/10th of the test list.

If a HW CI farm goes offline (network dies and all CI pipelines end up stalled) or its runners are consistenly spuriously failing (disk full?), and the maintainer is not immediately available to fix the issue, please push through an MR disabling that farm’s jobs by adding ‘.’ to the front of the jobs names until the maintainer can bring things back up. If this happens, the farm maintainer should provide a report to mesa-dev@lists.freedesktop.org after the fact explaining what happened and what the mitigation plan is for that failure next time.