Commit Graph

407 Commits

Author SHA1 Message Date
Thomas Perale
95c0e5ca95 utils/generate-cyclonedx: add support for 'resolved_with_pedigree'
The CycloneDX specification for vulnerabilities defines four analysis
states ([1]) for cases where a vulnerability does not affect a component:

* resolved
* resolved_with_pedigree
* not_affected
* false_positive

Currently, the metadatas present in Buildroot does not allow an accurate
mapping of ignored CVEs to the appropriate CycloneDX vulnerability
categories. As a result, all ignored CVEs are currently marked as
'in_triage' by default.

This default analysis was established during the introduction of the
'generate-cyclonedx' script. The reasoning at the time was that SBOM
consumers might want to re-evaluate ignored vulnerabilities, as the
Buildroot infrastructure could not reliably determine their actual
state.

This patch adds support for automatically marking vulnerabilities as
'resolved_with_pedigree' when a Buildroot patch includes a 'CVE:''
tag in its header referencing the CVE identifier.

The 'CVE:' tag appears alongside the already required 'Upstream:', if
the patch address a security vulnerability and may be repeated if a
patch addresses multiple vulnerabilities.

If a vulnerability is addressed by multiple patches, each patch will need to
reference the vulnerability identifier.

For details on how CycloneDX handles 'resolved_with_pedigree', see
[1][2].

As an example, the CVE-2025-3198 from the binutils package will result
in the following pedigree for the binutils component:

```
{
    "type": "unofficial",
    "diff": {
        "text": {
            "content": "..."
        }
    },
    "resolves": [
        {
            "type": "security",
            "name": "CVE-2025-3198"
        }
    ]
},
```

The `resolves` property is an array of issue the pedigree resolves. If
multiple are addressed by the same patch, then multiple identifier will be
present in this array.

In the listed vulnerabilities the entry for the CVE-2025-3198 looks like
this:

```
{
    "id": "CVE-2025-3198",
    "analysis": {
        "state": "resolved_with_pedigree",
        "detail": "The CVE 'CVE-2025-3198' has been marked as ignored by Buildroot"
    },
    "affects": [
        {
            "ref": "binutils"
        }
    ]
}
```

[1] https://cyclonedx.org/docs/1.6/json/#vulnerabilities_items_analysis_state
[2] https://cyclonedx.org/docs/1.6/json/#components_items_pedigree_patches_items_resolves

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 9415529923c9f7eaeec44c8fd3eecca79bf6b8d2)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-12-18 16:56:17 +01:00
Peter Korsgaard
6b1d9ab4e7 utils/check-package: add a check for 'default n' in Config.in files
Boolean Config.in symbols default to 'n', so we typically do not add such
redundant lines.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 279cbbdb64)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-11-26 09:25:29 +01:00
Vincent Stehlé
d7515eefc8 utils/brmake: filter output for parallel build
When building in parallel with per-package directories
(BR2_PER_PACKAGE_DIRECTORIES=y), brmake output is often garbled:

  2025-10-08T18:39:10 >>> host-dtc 1.7.2 Building
  2025-10-08T18:39:11 checking for stdint.h... >>> host-dtc 1.7.2 Installing to host directory
  2025-10-08T18:39:12 checking for limits.h... >>> host-gmp 6.3.0 Installing to host directory

Remove the spurious string between the timestamp and the ">>>" marker to
fix this.
We need some extra care to preserve the preceding "term bold" special
characters sequence.

We also prevent grep and sed to buffer their output too much. This leads to
more frequent output even when we might not be connected to a terminal; for
example: when brmake's output is piped to another program or when running
in CI.

Reviewed-by: Marcus Hoffmann <buildroot@bubu1.eu>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit c9dca7f4c3)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-10-30 08:56:13 +01:00
James Hilliard
5b42ae0e05 utils/scanpypi: remove python six module
We dropped support for python2 a while back in [1], as such we
can remove the python six module which was only needed for
backwards comaptibility with python2.

[1] 2743ce00ca

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[Julien: add commit ref removing python2 support]
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 53eb75ef53)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-09-25 22:07:15 +02:00
Gero Schwäricke
89f5bf0406 utils/bump-stable-kernel-versions: new tool
This tool helps to update Linux stable releases.

The script uses the versions found in linux/linux.hash. For each of the
versions it downloads the related hash list and tries to find an updated
stable release. If found it updates all related files and hashes.

Signed-off-by: Gero Schwäricke <gero.schwaericke@grandcentrix.net>
[Arnout: commonalize the sed expression for linux and linux-headers]
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit a87b1800b9)
Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>
2025-09-04 13:57:24 +02:00
Titouan Christophe
8af66709c7 utils/test-pkg: add new option -T/--toolchain-name
Recently when trying to reproduce a build error with test-pkg on the toolchain
br-arm-basic, I had to create a new CSV file with that toolchain only to be
able to run the test in that single case.

I believe having a command-line option right away in test-pkg would make this
much easier, as I can now run: `./utils/test-pkg -T br-arm-basic -p redis`

Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Arnout Vandecappelle <arnout@rnout.be>
(cherry picked from commit f3be464f7a)
Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>
2025-09-04 13:57:24 +02:00
Fabien Lehoussel
01e97b6f5c utils/generate-cyclonedx: sort dependencies
The SBOM is easier to read if the dependencies are sorted alphabetically.

Signed-off-by: Fabien Lehoussel <fabien.lehoussel@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 4b15707056)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-08-14 09:26:01 +02:00
Fabien Lehoussel
2745db5d2a utils/generate-cyclonedx: add project name and version options
Add options to customize the project name and version in the generated SBOM
metadata and set buildroot generate-cyclonedx as a tool in the metadata
section.

Signed-off-by: Fabien Lehoussel <fabien.lehoussel@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 9cbbc47762)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-08-14 09:25:57 +02:00
Fabien Lehoussel
f65bbd34f8 utils/generate-cyclonedx: move metadata section to top level
This makes it more readable and easier to quickly identify basic information.

Signed-off-by: Fabien Lehoussel <fabien.lehoussel@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 6098cc45d6)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-08-14 09:25:55 +02:00
James Knight
e0275af534 utils/test-pkg: stop on sigint
When SIGINT is issued for a package test run, it will abort the active
toolchain run then proceed to the next. If a user is running the entire
default toolchain set (`-a`), they can be required to invoke SIGINT
multiple times to stop a run.

This commit uses a SIGINT hook to flag a shutdown state and stop further
attempts to run anymore toolchain tests.

Signed-off-by: James Knight <git@jdknight.me>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit 8f09106e81)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-07-10 11:23:42 +02:00
James Knight
074e005e42 utils/check-package: handle missing files
When running check-package before completing commits for a change, if
any files are setup for removal, check-package will throw
FileNotFoundError exceptions instead of generating a warning state. For
example:

 $ utils/docker-run make check-package
 Traceback (most recent call last):
   ...
 FileNotFoundError: [Errno 2] No such file or directory: 'package/.../0001-some-removed-patch.patch'
 make: *** [Makefile:1264: check-package] Error 1

This commit will now catch FileNotFoundError and populate a warning
message:

 $ utils/docker-run make check-package
 package/.../0001-some-removed-patch.patch: missing; unstaged file removal?
 package/.../0002-another-removed-patch.patch: missing; unstaged file removal?
 427843 lines processed
 3 warnings generated
 make: *** [Makefile:1264: check-package] Error 1

Signed-off-by: James Knight <git@jdknight.me>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
(cherry picked from commit c41a06bbd9)
Signed-off-by: Thomas Perale <thomas.perale@mind.be>
2025-07-10 11:23:38 +02:00
Thomas Devoogdt
1713472621 utils/generate-cyclonedx: use indent 2 by default
By default, use an indent of 2, which is the same as what 'jq' uses.
This omits the need for 'jq' in the example usage. Also, add a new
line to the output while at it.

Signed-off-by: Thomas Devoogdt <thomas.devoogdt@barco.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-03-06 21:49:53 +01:00
Thomas Devoogdt
66a0513e0e utils/generate-cyclonedx: fix wrong example usage
The real file is utils/generate-cyclonedx,
not utils/generate-cyclonedx.py.

Signed-off-by: Thomas Devoogdt <thomas.devoogdt@barco.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2025-03-05 19:55:43 +01:00
Heiko Thiery
eec0f25734 utils/generate-cyclonedx: fix detecting of tty
Check if the script shall read from stdin if data is piped into.
Otherwise read from the input file or if not specified print usage and
exit.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2025-02-28 14:56:00 +01:00
Thomas Perale
dbab39e2d9 support/scripts/generate-cyclonedx.py: add script to generate CycloneDX-style SBOM
There is a growing need to generate software bill of materials (SBOM) from
buildroot configurations. Right now there are different solutions available
for buildroot users `show-info`, `legal-info` and `pkg-stats`.
They all generate similar information but in a format that is specific
to buildroot.

CycloneDX is a SBOM specification that can be consumed by different services.

This patch introduces a Python script, that converts the JSON output of the
show-info Makefile target to a CycloneDX-style SBOM.
The script output contains the following information.
    - A list of all packages, or "components" with information about
      version, cpe (if available), applied patches.
    - By default virtual packages are not listed in the SBOM.
    - Additional information is added to the component 'properties' to
      specify wheter the component is present on the target or the host
      under the `BR_TYPE` property name.
    - An overview of the licenses applicable to each package. If possible,
      the names of these licenses have been matched to known SPDX license
      identifiers.
    - Per package, a list of (recursive) dependencies on other packages.
    - A list of ignored CVE and their associated component.

More information on CycloneDX at https://cyclonedx.org/.

Usage:
    make show-info | utils/generate-cyclonedx.py | jq '.'

Example output:

```
{
  "bomFormat": "CycloneDX",
  "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json",
  "specVersion": "1.6",
  "components": [
    {
      "bom-ref": "busybox",
      "type": "library",
      "name": "busybox",
      "version": "1.36.1",
      "licenses": [
        {
          "license": {
            "id": "GPL-2.0"
          }
        },
        ...
      ],
      "cpe": "cpe:2.3🅰️busybox:busybox:1.36.1:*:*:*:*:*:*:*",
      "pedigree": {
        "patches": [
          {
            "type": "unofficial",
            "diff": {
              "text": {
                "content": "..."
              }
            }
          }
        ]
      }
      "properties": [
        {
          "name": "BR_TYPE",
          "value": "target"
        }
      ]
    },
    ...
  ],
  "dependencies": [
    {
      "ref": "busybox",
      "dependsOn": [
        "host-skeleton",
        "skeleton",
        "skeleton-init-sysv",
        "skeleton-init-common",
        ...
    }
    ...
  ],
  "vulnerabilities": [
    {
      "id": "CVE-2022-28391",
      "analysis": {
        "state": "in_triage",
        "detail": "The CVE 'CVE-2022-28391' has been marked as ignored by Buildroot"
      },
      "affects": [
        {
          "ref": "busybox"
        }
      ]
    },
    ...
  ],
  "metadata": {
    "component": {
      "bom-ref": "buildroot",
      "name": "buildroot",
      "type": "firmware",
      "version": "2024.02-4744-gafea667f00-dirty"
    }
  }
}
```

Signed-off-by: Thomas Perale <thomas.perale@mind.be>
Co-authored-by: Matthias Swiggers <matthias.swiggers@mind.be>
Reviewed-by: Vincent Jardin <vjardin@free.fr>
[Arnout:
 - alphabetically order imports;
 - use endswith instead if 'in' to check suffix;
 - add usage to help text;
 - remove .py suffix.
]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-22 18:35:30 +01:00
Yann E. MORIN
25b6a6ecee utils/docker-run: allow explicitly setting of docker/podman/...
Currently, our docker-run helper will decide on its own whether it
should use docker (preferred) or podman (fallback), as introduced in
9a629f5129 (utils/docker-run: allow running with Podman). In case
both are installed, it is not possible to exercise the podman case.
Often, 'docker' is just an alias for 'podman' when both are available,
but this is not always true - and in the latter case, the user needs to
be able to choose which one they want.

Allow the user to force the one to use, by setting the BR2_DOCKER
environment variable. If that is set and it doesn't exist, exit with
an explicit error message (rather than relying on the failure when
eventually exec-ing the specified command).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Cc: Julien Olivain <ju.o@free.fr>
Cc: Fiona Klute <fiona.klute@gmx.de>
Reviewed-by: Fiona Klute (WIWA) <fiona.klute@gmx.de>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-18 21:18:48 +01:00
Yann E. MORIN
d21aeb44e0 utils/brmake: simplify redirection
We want to redirect both stdout and stderr to the log, so just do both
redirections at once. This simplifies the call to make, which was
inherited from afdb545b28 (tools: new tool to filter the output of
make) when brmake was using unbuffer (and even then it was not entirely
waranted to do the redirection that way).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-18 20:58:16 +01:00
Yann E. MORIN
5494efe61b Revert "utils/brmake: add option to run in docker"
All our documented uses of utils/docker-run, besides running it
standalone to get an interactive shell, is to use it as a prefix
to the otherwise standard command to run, e.g.:
  - ./utils/docker-run make foo_defconfig
  - ./utils/docker-run make menuconfig
  - ./utils/docker-run make
  - ./utils/docker-run make check-package

Commit 8aad67f157 (utils/brmake: add option to run in docker) departed
from that usual convention, by hiding the call to docker-run inside
brmake, conditioned by an environment variable. The only reason is that
brmake internally used ubuffer. This is no longer the case, so we can
now use brmake together with docker-run in the usual manner.

This basically reverts commit 8aad67f157,
after resolving the conflict due to the removal of unbuffer in te
previous commit.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-18 20:50:54 +01:00
Yann E. MORIN
3aacb38098 utils/brmake: drop use of unbuffer
When it was introduced in afdb545b28 (tools: new tool to filter the
output of make), brmake made use of unbuffer, so as to try and ensure
that no output would get lost between make printing it, and the logger
loop time-stamping it and storing it in the logfile, in case the user
would interrup the build (SIGINT, ^C)

However, unbuffer is missing in our reference build image (we could have
added it, but we missed the occasion to do so every time we updated the
reference build image).

Furthermore, the rationale for using unbuffer, although reasonable, is
not so practical: indeed, when the user hits Crtl-C, this is inherently
asynchronous, and they can't expect everything to be entirely cleanly
terminated, especially buffering. Using unbuffer is thus slightly
superfluous.

The timestamps will be slightly off because of buffering, but a build
generates so much output that this won't be a problem in practice.

Drop use of unbuffer.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-18 20:45:00 +01:00
Romain Naour
ef80b710f2 utils/config: fix shellcheck errors
SC2086 is now reported for missing Double quote around
$FN since shellcheck 0.9.0:

In utils/config line 175:
                if grep -q "# ${BR2_PREFIX}$ARG is not set" $FN ; then
                                                            ^-^ SC2086 (info): Double quote to prevent globbing and word splitting.

In utils/config line 178:
                        if V="$(grep "^${BR2_PREFIX}$ARG=" $FN)"; then
                                                           ^-^ SC2086 (info): Double quote to prevent globbing and word splitting.

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2025-02-09 15:39:46 +01:00
Romain Naour
4f78d362aa utils/checkpackagelib: fix shellcheck tests after update to shellcheck 0.9.0
shellcheck tests needs to be updated after shellcheck 0.9.0 update due
to changes in errors/warning reporting:

  utils/checkpackagelib/test_tool.py::test_Shellcheck[missing shebang-empty.sh--expected0] FAILED [ 98%]
  utils/checkpackagelib/test_tool.py::test_Shellcheck[2 warnings-unused.sh-unused=""-expected3] FAILED [ 99%]
  utils/checkpackagelib/test_tool.py::test_Shellcheck[tab-tab.sh-\t#!/bin/sh-expected4] FAILED [100%]

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2025-02-09 12:53:35 +01:00
Julien Olivain
fa98239074 utils/check-symbols: allow BR2_BROKEN to be unused
Commit [1] introduced the BR2_BROKEN hidden symbol. The intent of
this symbol is to temporarily mark a configuration known to be broken.
This commit was added to fix an undefined usage of the symbol, in a
previous commit [2].

With this somewhat special symbol, it is also perfectly normal to
have no use of it.

Running the command "utils/check-symbols" in a state there is no usage
of the BR2_BROKEN, the script fails with output:

    Config.in:15: BR2_BROKEN defined but not referenced

This commit adds an exception for this symbol, allowing it to be
unused in the whole Buildroot tree.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/8925433382

[1] 2ddc5808cd
[2] 11a8cdd2bb

Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
2025-02-05 11:01:05 +01:00
Brandon Maier
80d26f091f utils/check-package: support finding files from patches
For the `b4` tool to support check-package, check-package must support
reading patch files from stdin.

It would be complicated to make check-package actually run checks on
patch files. So instead we search the patch files to figure out what
files are modified in the repo, then run check-package on the modified
files directly.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
[Arnout: rename variable "files" to "files_to_check"]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2025-02-04 18:44:35 +01:00
Yann E. MORIN
717f1fdaeb utils/get-developers: add option to report Cc: lines
It is very common to use the output of get-developers to add cc: lines
in the commit log.

Add an option so that get-developers reports Cc: lines ready to be
pasted in a commit log. That new option behaves similarly to the
existing -e option: it only affects the output when parsing a patch.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:20 +01:00
Yann E. MORIN
3177ecd260 utils/get-developers: sort reported developers
The list of reported developers is not ordered: that may leave the
impression (when receiving a patch) that a Cc is more important than
another, by virtue of being earlier in the list.

Also, the ordering changes on every call.

Report the developers in an alphabetically order, so that there is no
confusion anymore, and so the ordering is reproducible across calls.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:17 +01:00
Yann E. MORIN
d10d22221f utils/get-developers: read patch from stdin when it's not a tty
It is very common that get-developers be used with its stdin a pipe from
git-show:
    git show |./utils-get-developers -

In this case, the '-' is superfluous: we can very easily deduce that the
user wants to read stdin as the patch.

So, if no other action was requested, and stdin is not a tty, use it as
the source of the patch, and thus '-' is then no longer required.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:13 +01:00
Yann E. MORIN
35f381b93e utils/get-developers: use parser.error() rather than canned print+return
parser.error() reports a nice error message, that also displays a short
reminder of the available options.

Adapt the test-suite accordingly: previously, the error string was an
exact string in the stdout list, while it now is a substring in one of
the strings in stderr. The exit code changes, too.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Julien Olivain <ju.o@free.fr>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:10 +01:00
Yann E. MORIN
cdcb3f56e8 utils/get-developers: don't offload parse_args()
Offloading parser.parse_args() to a helper function does not bring much,
if at all; it even is restrictive: indeed, we can't use parser.error()
to report errors and thus have to resort to a canned print+return
sequence...

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2025-01-05 14:09:06 +01:00
Thomas Petazzoni
f97b67be74 utils/genrandconfig: more igh-ethercat tweaks
utils/genrandconfig already had some logic to disable building
igh-ethercat drivers since they are highly kernel version
specific. Commit
eec3a1108a ("package/igh-ethercat: add
configuration options") recently added a number of additional config
options to build more igh-ethercat drivers, which causes a number of
autobuilder failures. This commit therefore updates genrandconfig to
also not build those new drivers.

Fixes:

  https://autobuild.buildroot.org/results/9489b68967a35aa398e608d663824b2d2be9cae5/ (stmmac driver)
  https://autobuild.buildroot.org/results/ec2c24f16b365a3e7de8964fc8d73cb570c74abe/ (igc driver)
  https://autobuild.buildroot.org/results/65e17f8e03ab14dd9f39386f2176629677b8ba8c/ (genet driver)

Cc: Jannik Becher <becher.jannik@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
2024-12-17 00:13:08 +01:00
James Hilliard
0205f4c2b8 package/pkg-python.mk: add poetry setup type
As we have many python packages using the poetry(poetry-core) build
system we should add a setup type for it so that we don't have to
manually specify the host-python-poetry-core dependency.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-11-04 23:19:55 +01:00
Thomas Petazzoni
b07d21ab98 utils/genrandconfig: handle BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y
When BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y,
BR2_LINUX_KERNEL_IMAGE_TARGET_NAME is supposed to be non-empty. But in
the context of genraconfig, we don't know to what value
BR2_LINUX_KERNEL_IMAGE_TARGET_NAME can be set, so let's avoid cases
where BR2_LINUX_KERNEL_IMAGE_TARGET_CUSTOM=y. By dropping this option,
kconfig will revert back to the default image format for the selected
architecture.

Fixes:

  http://autobuild.buildroot.net/results/1d104a051c83bb31e98565369a2ec7badfa21eca/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-27 19:48:06 +01:00
Brandon Maier
6ffcdb52e8 utils/check-package: add inline script requirements
The check-package tool requires some PyPi package to be installed before
it can run. This is typically done by manually installing them into the
user's global Python environment or setting up a virtual environment,
then manually installing each dependency.

Python recently defined a format for managing script dependencies as
inline metadata[1]. This can be used with the `uv` tool to run a Python
script and automatically install the minimum required version of Python
and PyPi dependencies.

With this change, it's now possible to run check-package with

  uv run -s ./utils/check-package

Note that, because check-package does not have the '.py' file extension
we must specify the `-s` or `--script` argument. That argument was added
very recently in release 0.4.19[2].

I set the minimum python to 3.9 as that is the oldest version still
supported[3]. I verified 3.9 works by running

  uv run -p 3.9 -s ./utils/check-package `git ls-tree -r --name-only HEAD` --ignore-list=.checkpackageignore

[1] https://packaging.python.org/en/latest/specifications/inline-script-metadata/#script-type
[2] https://github.com/astral-sh/uv/releases/tag/0.4.19
[3] https://devguide.python.org/versions/

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-26 13:37:39 +02:00
James Hilliard
1459c0a6f0 package/pkg-python.mk: add hatch setup type
As we have many python packages using the hatch(hatchling) build
system we should add a setup type for it so that we don't have to
manually specify the host-python-hatchling dependency.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-23 23:00:03 +02:00
Arnout Vandecappelle
8aad67f157 utils/brmake: add option to run in docker
At the moment, it is difficult to combine brmake with docker-run.
`docker-run brmake ...` doesn't work because our docker image doesn't
have unbuffer. In addition, inside the container the timezone is UTC,
but you probably want the timestamps added by brmake to be in local
time. Therefore, it's better to have the call to docker-run nested
inside brmake.

Currently, brmake doesn't have any way to pass parameters, all of "$@"
is passed unchanged to the `make` invocation. Thus, there is no
established way to pass in the option whether or not to use docker. We
choose to use an environment variable to pass in the option. The
convention is that such a buildroot-specific environment variable should
start with BR2_, so we choose BR2_DOCKER.

Run make inside docker-run if the BR2_DOCKER environment variable is
set. Update utils/readme.txt (the only existing documentation of brmake)
with this information.

Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-23 22:09:50 +02:00
James Hilliard
74764683f5 utils/scanpypi: add missing setuptools fallback
Some projects use pyproject.toml without setting a build backend, per
pep517 we should simply fallback to using setuptools when no build
backend is specified.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-10-10 23:17:16 +02:00
James Hilliard
699b3c38fd utils/scanpypi: refactor setuptools handling to not use imp
The imp module is deprecated as of python verison 3.12.

Refactor setuptools handling to remove monkeypatching hack and
instead do pep517 metadata generation and dependency resolution.

This is effectively done by implementing the minimal neccesary
pep517 frontend hooks needed for dependency resolution in scanpypi.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Fiona Klute (WIWA) <fiona.klute@gmx.de>
Reviewed-by: Fiona Klute (WIWA) <fiona.klute@gmx.de>
[Arnout: add license info]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-10-08 22:07:28 +02:00
Peter Korsgaard
f1ceb3687f utils/update-rust: fix 'following' typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:39:10 +02:00
Peter Korsgaard
aa79ae24a8 utils/scanpypi: fix typos
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[yann.morin.1998@free.fr: s/contents/content/]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:38:59 +02:00
Peter Korsgaard
37574ab12a utils/readme.txt: fix 'typos' typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:19:00 +02:00
Peter Korsgaard
ed2cb54252 utils/getdeveloperlib.py: fix retrieve typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:18:56 +02:00
Peter Korsgaard
6bcdbccb7f utils/docker-run: fix symmetry typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:18:52 +02:00
Peter Korsgaard
b3e5bcec65 utils/config: fix don't typo
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-09-21 20:18:48 +02:00
Yann E. MORIN
f9e6d39e8e utils/check-package: require exactly 1 TAB and 2 SP on help text 1st line
Test that the first line starts exactly with one TAB and exactly two
spaces before the text.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-09-14 22:24:55 +02:00
James Hilliard
ea6bb507b1 utils/genrandconfig: allow overriding KCONFIG_PROBABILITY
Tweaking this variable should allow us to get better coverage of
packages with larger dependency trees.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-08-23 19:40:12 +02:00
Yann E. MORIN
4dbb87bb66 utils/genrandconfig: fix fallout of dropping toolchain CSV
Commit 2f260084d5 (utils/genrandconfig: remove support for toolchain
CSV) kept the --no-toolchains-csv option, but in the rework forgot to
keep it as a bool, while argparse default is to expect a string.

Rather than re-introduce the action="store_true" which implies the
argument is a bool, explicit make it a bool.

Fixes: 2f260084d5

Reported-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-08-20 23:22:59 +02:00
Thomas Petazzoni
ce3dedc26b utils/genrandconfig: improve logging
Right now, genrandconfig just spits out the random messages from the
different make invocations, which isn't terribly useful. Instead,
let's redirect the output of make invocations to oblivion, and add
some more high level logging.

As part of this logging, we're interested to see how many iterations
were needed to find a valid configuration, so changed the loop logic
to count from 0 to 100 instead of from 100 to 0 so that we can easily
show the iteration number.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:08:10 +02:00
Thomas Petazzoni
da5c25c9f9 utils/genrandconfig: do not use BR2_BACKUP_SITE for some builds
In order to test that upstream sites are still working, we need to NOT
fallback to sources.buildroot.net for some builds.

As there is anyway a local cache in the autobuilder instances, we need
to do quite a lot of builds without any BR2_BACKUP_SITE configured to
have a chance to catch issues, which is why a 50% chance is used to
unset BR2_BACKUP_SITE.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:08:05 +02:00
Thomas Petazzoni
3d33d394c2 utils/genrandconfig: rework fine-tuning logic
Before calling randpackageconfig/randconfig, we were pre-generating a
snippet of .config with:

 (1) minimal.config
 (2) BR2_CURL/BR2_WGET settings
 (3) some random selection of init system, debug, runtime debug, etc
 (4) enabling BR2_REPRODUCIBLE=y when diffoscope was found

Now that we only use randconfig, this whole fine-tuning is completely
irrelevant, as it gets overridden by "make randconfig".

(1) and (3) above are useless, as randconfig does all the
randomization that is needed.

However, we want to preserve (2) and (4) above, so we re-implement
those fixups, but *after* randconfig has done its job.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:07:56 +02:00
Thomas Petazzoni
2f260084d5 utils/genrandconfig: remove support for toolchain CSV
Now that the support for generating a fully random configuration has
been well-tested, the whole mechanism based on a toolchain CSV isn't
really useful anymore, so let's drop it to simplify the logic.

Note that the autobuilder code still uses --{,no-}toolchains-csv, so we
can't remove those or the autobuilders would fail. Once all supported
branches no longer use those argumetns, we can drop them from the
autobuilder code, then ask people to update their runners, and we will
finally be able to drop those arguments. Eventually.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: keep --{,no-}toolchains-csv and explain why]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 12:07:21 +02:00
Thomas Petazzoni
9e33882568 utils/genrandconfig: remove fixups related to untested CT-NG toolchains
We have accumulated a whole bunch of very old fixups to avoid issues
with super old CT-NG toolchains, which we are not testing anymore, so
remove those fixups.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-08-18 11:56:30 +02:00