[RESEND PATCH] kconfig: Proposed language extension for multiple builds

[RESEND PATCH] kconfig: Proposed language extension for multiple builds

[Simon Glass]

(I am sending this again to get more feedback)

In the case of Linux, only one build is produced so there is only a
single configuration. For other projects, such as U-Boot and Zephyr, the
same code is used to produce multiple builds, each with related (but
different) options enabled.

This can be handled with the existing kconfig language, but it is quite
verbose, somewhat tedious and very error-prone, since there is a lot of
duplication. The result is hard to maintain.

Describe an extension to the Kconfig language to support easier handling
of this use case.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 Documentation/kbuild/kconfig-language.rst | 134 ++++++++++++++++++++++
 1 file changed, 134 insertions(+)

diff --git a/Documentation/kbuild/kconfig-language.rst b/Documentation/kbuild/kconfig-language.rst
index 858ed5d80defe..73fb016a5533f 100644
--- a/Documentation/kbuild/kconfig-language.rst
+++ b/Documentation/kbuild/kconfig-language.rst
@@ -228,6 +228,24 @@ applicable everywhere (see syntax).
   enables the third modular state for all config symbols.
   At most one symbol may have the "modules" option set.
 
+- phase declaration: "defphase"
+  This defines a new build phase. See `Build Phases`_.
+
+- default phase: "phasedefault"
+  This indicates the default build phase. See `Build Phases`_.
+
+- add entries for phases: "addphases"
+  This creates new phase-specific entries based on a template entry and adds
+  the same attributes to it. See `Build Phases`_.
+
+- set entries for phases: "setphases"
+  This sets the phases which need an entry. This allows creating an entry that
+  only has a primary phase. See `Build Phases`_.
+
+- indicate a phase-specific attribute: "forphases"
+  This marks an attribute as being applicable only to a particular phase or
+  group of phases.  See `Build Phases`_.
+
 Menu dependencies
 -----------------
 
@@ -319,6 +337,119 @@ MODVERSIONS directly depends on MODULES, this means it's only visible if
 MODULES is different from 'n'. The comment on the other hand is only
 visible when MODULES is set to 'n'.
 
+Build Phases
+------------
+
+Some projects use Kconfig to control multiple build phases, each phase
+resulting in a separate set of object files and executable. This is the
+case in U-Boot [12]_. Zephyr OS seems to be heading this way too [13]_.
+
+Generally the phases are related, so that enabling an entry in the primary
+phase also enables it by default in the others. But in some cases it may
+be desirable to use separate conditions for each phase.
+
+All phases have a phase name, for example `SPL`. This name is used as a
+prefix to each entry used in that phase, with an underscore in between.
+So if FOO is the primary entry, the equivalent entry for the SPL phase
+is SPL_FOO. The primary phase is marked with a "phasedefault" entry.
+
+Phases are declared like any other menu entry except that also have a
+"defphase" keyword. Phase entries are normally hidden so do not have a
+prompt::
+
+    config PPL
+        bool
+        defphase "Primary Program Loader"
+        phasedefault
+        help
+          This is the primary bootloader.
+
+    config SPL
+        bool
+        defphase "Secondary Program Loader"
+        help
+          This is used to set up memory and load the primary bootloader.
+
+The default phase (here PPL) is assumed for all entries, in the sense that
+all entries are present in PPL by default and no prefix is needed on these
+entries. So FOO means that it applies to PPL. There must be exactly one
+default phase.
+
+The resulting menu entries can be used normally throughout the Kconfig. With
+this technique, the different build phases can be fully and individually
+controlled from Kconfig.
+
+However it is not ideal. Often the secondary phases have far fewer entries than
+the primary phase, since they offer fewer features. Even so, each FOO that is
+needed in a phase must have an SPL_FOO, etc. To avoid an explosion of entries,
+it is possible to indicate which are enabled, as a shortcut for creating new
+entries::
+
+    config FOO
+        bool "Enable foo feature"
+        addphases SPL
+        depends on %BAR
+        depends on QUX
+        forphases SPL depends on FIZZ
+
+Note that "%" expands to the phase, so this is equivalent to (ignoring BAR)::
+
+    config FOO
+        bool "Enable foo feature"
+        depends on BAR
+        depends on QUX
+
+    config SPL_FOO			   # Phase is prepended
+        bool "Enable foo feature (SPL)"    # Suffix is added
+        depends on SPL_BAR                 # "%" dependency is expanded
+        depends on QUX
+        depends on FIZZ                    # Added only for SPL
+        depends on SPL                     # Added automatically
+
+Attributes declared in the primary symbol FOO (such as "depends on BAR") also
+apply to the secondary ones.
+
+An entry without any 'addphases' attribute applies to all phases. Individual
+phase entries are not available in that case. If the entry is enabled, then
+it is enabled for all phases. Only one entry appears in the resulting Kconfig.
+
+In the case where an entry should apply only to the primary phase (or a
+particular set of phases), you can uses "setphases" instead of "addphases"::
+
+    config FOO
+        bool "Enable foo feature"
+        setphases PPL
+
+This means that even if the option is enabled, it will not be active outside
+the primary-phase build, here named "PPL".
+
+Internally, phases are implemented simply by creating new entries. These
+appear in the Kconfig as per normal. It would be possible for a Kconfig
+editor to show the entries just for a particular phase, leaving out the
+entries not applicable to that phase.
+
+When phases are used, the Kconfig tool outputs separate auto.conf files for
+each phase (e.g. auto_spl.conf), so that if SPL_FOO is enabled, then
+`CONFIG_FOO=y` is present in the file. This makes it easy for the build system
+to build the correct code, use IS_ENABLED(), etc.
+
+To ensure that the correct options are enabled for each build, in addition to
+the normal CONFIG_FOO option in the file, phase symbols are added too. For
+example, if FOO is enabled in the SPL phase, then auto_spl.conf contains::
+
+    CONFIG_FOO=y
+    CONFIG_SPL_FOO=y
+
+The phase-specific line (CONFIG_SPL_FOO) is seldom needed, but it allows one
+phase to access the symbols from another phase. For example, if the primary
+phase needs to receive boot timings from SPL, then support for boot timings must
+be added in both phases. If the timings are at a fixed address, this can be in a
+shared symbol (like CONFIG_SPL_TIMING_BASE) that both phases can access.
+
+Technical note: the grammar definition of <symbol> in this documeent does not
+include the "%" prefix at present. It can be used with any attribute. It cannot
+be used with any top-level items, like "config", "menuconfig", "if" and "menu".
+
 
 Kconfig syntax
 --------------
@@ -744,3 +875,6 @@ https://kernelnewbies.org/KernelProjects/kconfig-sat
 .. [9] https://www4.cs.fau.de/Publications/2011/tartler_11_eurosys.pdf
 .. [10] https://paulgazzillo.com/papers/esecfse21.pdf
 .. [11] https://github.com/paulgazz/kmax
+
+.. [12] https://u-boot.readthedocs.io/en/latest/develop/spl.html
+.. [13] https://docs.zephyrproject.org/latest/build/sysbuild/index.html
-- 
2.40.0.rc1.284.g88254d51c5-goog

Re: [RESEND PATCH] kconfig: Proposed language extension for multiple builds

[Randy Dunlap]

Hi--

On 3/10/23 18:37, Simon Glass wrote:
> (I am sending this again to get more feedback)
> 
> In the case of Linux, only one build is produced so there is only a
> single configuration. For other projects, such as U-Boot and Zephyr, the
> same code is used to produce multiple builds, each with related (but
> different) options enabled.
> 
> This can be handled with the existing kconfig language, but it is quite
> verbose, somewhat tedious and very error-prone, since there is a lot of
> duplication. The result is hard to maintain.
> 
> Describe an extension to the Kconfig language to support easier handling
> of this use case.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

IMO Masahiro has already answered this multiple times and I agree with his answers.

For others, the full previous thread is at
  https://lore.kernel.org/all/20230219145453.1.Idaaf79c3e768b85750d5a7eb732052576c5e07e5@changeid/


> ---
> 
>  Documentation/kbuild/kconfig-language.rst | 134 ++++++++++++++++++++++
>  1 file changed, 134 insertions(+)


-- 
~Randy

Re: [RESEND PATCH] kconfig: Proposed language extension for multiple builds

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]

On Fri, Mar 10, 2023 at 09:39:15PM -0800, Randy Dunlap wrote:
> Hi--
> 
> On 3/10/23 18:37, Simon Glass wrote:
> > (I am sending this again to get more feedback)
> > 
> > In the case of Linux, only one build is produced so there is only a
> > single configuration. For other projects, such as U-Boot and Zephyr, the
> > same code is used to produce multiple builds, each with related (but
> > different) options enabled.
> > 
> > This can be handled with the existing kconfig language, but it is quite
> > verbose, somewhat tedious and very error-prone, since there is a lot of
> > duplication. The result is hard to maintain.
> > 
> > Describe an extension to the Kconfig language to support easier handling
> > of this use case.
> > 
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> 
> IMO Masahiro has already answered this multiple times and I agree with his answers.
> 
> For others, the full previous thread is at
>   https://lore.kernel.org/all/20230219145453.1.Idaaf79c3e768b85750d5a7eb732052576c5e07e5@changeid/

Well, I think what was unclear, or maybe we just wanted to confirm the
answer was "none at all", was this. As good community neighbors, we see
a generic issue in the Kconfig language, a tool used frequently outside
of just the Linux kernel, and would like to contribute back. Ideally
without having first gone off, designed and implemented something, and
then been told it's all wrong and to rewrite it first. So what level of
interest is there in this?

As I pointed out in that thread, I believe barebox has examples where
some keyword like we're proposing here would help them (and yes, there's
only a dozen or so symbols so it's also manageable without anything
special), and Simon believes Zephyr will be in a similar situation soon
enough (which doesn't use the kernel's implementation of the language).
Frankly, I keep going back to "tristate" is just the original example of
what we're talking about here (CONFIG_FOO=n, CONFIG_FOO_MODULE=y), not
that I'm suggesting we would remove the tristate word.
So we would really like to make sure as many people and projects are
aware, as possible.

And as Simon asked in the thread, what about code refactoring that makes
further maintenance easier? Clearly, such patches would need to be
against the current appropriate tree.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [RESEND PATCH] kconfig: Proposed language extension for multiple builds

[Boris Kolpackov]

Tom Rini <trini@konsulko.com> writes:

> On Fri, Mar 10, 2023 at 09:39:15PM -0800, Randy Dunlap wrote:
> > Hi--
> > 
> > On 3/10/23 18:37, Simon Glass wrote:
> > > (I am sending this again to get more feedback)
> > > 
> > > In the case of Linux, only one build is produced so there is only a
> > > single configuration. For other projects, such as U-Boot and Zephyr, the
> > > same code is used to produce multiple builds, each with related (but
> > > different) options enabled.
> > > 
> > > This can be handled with the existing kconfig language, but it is quite
> > > verbose, somewhat tedious and very error-prone, since there is a lot of
> > > duplication. The result is hard to maintain.
> > > 
> > > Describe an extension to the Kconfig language to support easier handling
> > > of this use case.
> > > 
> > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > 
> > IMO Masahiro has already answered this multiple times and I agree with his answers.
> > 
> > For others, the full previous thread is at
> >   https://lore.kernel.org/all/20230219145453.1.Idaaf79c3e768b85750d5a7eb732052576c5e07e5@changeid/
> 
> So what level of interest is there in this?

Unlike Masahiro & co I am interested in generalizing Kconfig to be usable
outside of the Linux kernel (for example, I've integrated it into the
build2 build system[1]). However, in this case, I tend to agree with Randy
and Masahiro: this feels like a very niche use-case (which I am still not
100% clear on, after reading the description 3 times) that would add
quite a bit of complexity.

One thing that did cross my mind during those 3 reads is that maybe the
essence of the feature you are looking for here is to be able to use a
value from the previous phase as "initial" (i.e., stronger than Kconfig
default but weaker than user-specified) when configuring the next phase.
It probably won't allow you to solve your problem in an auto-magical way
like your proposal, but perhaps you could get close enough while not
complicating the Kconfig language significantly.


> And as Simon asked in the thread, what about code refactoring that makes
> further maintenance easier?

>From my experience, there is little interest in patches that make Kconfig
more general or improve the compatibility of the implementation. As a
result, I am maintaining my patch set[2] out of tree.


[1] https://build2.org/libbuild2-kconfig/doc/build2-kconfig-manual.xhtml
[2] https://github.com/build2-packaging/kconfig/tree/upstream-5.15-rc7

Re: [RESEND PATCH] kconfig: Proposed language extension for multiple builds

[Masahiro Yamada]

On Sun, Mar 12, 2023 at 1:55 AM Tom Rini <trini@konsulko.com> wrote:
>
> On Fri, Mar 10, 2023 at 09:39:15PM -0800, Randy Dunlap wrote:
> > Hi--
> >
> > On 3/10/23 18:37, Simon Glass wrote:
> > > (I am sending this again to get more feedback)
> > >
> > > In the case of Linux, only one build is produced so there is only a
> > > single configuration. For other projects, such as U-Boot and Zephyr, the
> > > same code is used to produce multiple builds, each with related (but
> > > different) options enabled.
> > >
> > > This can be handled with the existing kconfig language, but it is quite
> > > verbose, somewhat tedious and very error-prone, since there is a lot of
> > > duplication. The result is hard to maintain.
> > >
> > > Describe an extension to the Kconfig language to support easier handling
> > > of this use case.
> > >
> > > Signed-off-by: Simon Glass <sjg@chromium.org>
> >
> > IMO Masahiro has already answered this multiple times and I agree with his answers.
> >
> > For others, the full previous thread is at
> >   https://lore.kernel.org/all/20230219145453.1.Idaaf79c3e768b85750d5a7eb732052576c5e07e5@changeid/
>
> Well, I think what was unclear, or maybe we just wanted to confirm the
> answer was "none at all", was this. As good community neighbors, we see
> a generic issue in the Kconfig language, a tool used frequently outside
> of just the Linux kernel, and would like to contribute back. Ideally
> without having first gone off, designed and implemented something, and
> then been told it's all wrong and to rewrite it first. So what level of
> interest is there in this?

Sorry, no interest.
If you want to get a clear answer, NACK.

>
> As I pointed out in that thread, I believe barebox has examples where
> some keyword like we're proposing here would help them (and yes, there's
> only a dozen or so symbols so it's also manageable without anything
> special),

Barebox keeps PBL in very limited, ad-hoc implementation.
PBL has no more than 10 user-configurable options.
Sascha Hauer designed it this way.



Linux kernel also has a small loader (a.k.a decompressor) in
arch/*/boot/decompress/.

For example, CONFIG_KERNEL_GZIP is a CONFIG option
for the decompressor instead of the main kernel.

In this sense, you could apply your theory,
"Linux kernel is also multi build-phases, so Kconfig should have this
extension to move CONFIG_KERNEL_GZIP to another build phase".
No, no. The main kernel and the decompressor are well separated
and the latter is small and simple.

Barebox is the same - the main Barebox and PBL are well separated
and PBL is really small and simple.

The problems you are suffering from do not exist in Barebox.



> and Simon believes Zephyr will be in a similar situation soon
> enough (which doesn't use the kernel's implementation of the language).

Zephyr does not share any Kconfig code with Linux.
They use Python implementation, a.k.a. Kconfiglib.
It is up to the Zephyr community, but this requires extra effort.

> Frankly, I keep going back to "tristate" is just the original example of
> what we're talking about here (CONFIG_FOO=n, CONFIG_FOO_MODULE=y), not
> that I'm suggesting we would remove the tristate word.
> So we would really like to make sure as many people and projects are
> aware, as possible.

This is on the boundary.
We can make the tristate optional if it does not make the code too ugly.

But, if you do not add CONFIG_MODULES in your Kconfig file,
users will not see 'm' in the first place.

I know some help messages still mention 'm', but is this the problem
you want to solve?


> And as Simon asked in the thread, what about code refactoring that makes
> further maintenance easier? Clearly, such patches would need to be
> against the current appropriate tree.

If such patches clean up the code, they will be appreciated.

-- 
Best Regards
Masahiro Yamada