platformio.ini advanced build configurations are not yet compatible with PlatformIO/SCons flag workflows

[zackees]

Title: platformio.ini advanced build configurations are not yet compatible with PlatformIO/SCons flag workflows

Summary

fbuild advertises compatibility with platformio.ini, but today that compatibility is mostly limited to flat config values like build_flags / build_src_flags and environment inheritance. Standard PlatformIO patterns for advanced build configurations are not modeled yet, so a project that uses normal PlatformIO debug/custom build techniques can silently build with different semantics under fbuild.

What this means in practice

A lot of PlatformIO projects define advanced build flavors by combining:

• build_type = debug|release|test
• debug_build_flags
• build_unflags
• extra_scripts that mutate SCons construction variables such as CPPDEFINES, CCFLAGS, CXXFLAGS, LINKFLAGS, or CPPPATH

That is the standard PlatformIO/SCons workflow for �advanced build types�: the INI selects the build flavor, and either declarative keys or SCons script hooks add/remove the actual compiler/linker flags.

Right now fbuild does not appear to implement that model. It reads ordinary build_flags / build_src_flags, has its own internal BuildProfile abstraction (release vs quick), and supports one-off caller-injected flags, but it does not map the PlatformIO build-type layer onto those internal profiles.

Repo evidence

• PlatformIOConfig::get_build_flags() only reads build_flags, and get_build_src_flags() only reads build_src_flags: crates/fbuild-config/src/ini_parser.rs:123-143
• BuildContext::new() only collects build_flags and build_src_flags before build orchestration: crates/fbuild-build/src/pipeline.rs:106-111
• Build orchestration has an internal profile system (release, quick) rather than a platformio.ini-driven build_type model: crates/fbuild-build/src/compiler.rs:24-29, crates/fbuild-build/src/lib.rs:138-166
• PLATFORMIO_BUILD_UNFLAGS is listed as a supported env override and has an accessor, but I could not find any point where it is applied to the effective flag set: crates/fbuild-config/src/pio_env.rs:15-21, crates/fbuild-config/src/pio_env.rs:111-121
• PLATFORMIO_EXTRA_SCRIPTS is explicitly recognized as warn-only / no-op today: crates/fbuild-config/src/pio_env.rs:43-59
• I could not find any repo references to build_type or debug_build_flags

Expected behavior

At minimum, fbuild should not silently accept a platformio.ini that depends on PlatformIO advanced build-type mechanics while producing a materially different build.

Preferred direction:

1. Parse and honor build_type, debug_build_flags, and build_unflags from platformio.ini
2. Define a compatibility mapping from PlatformIO build types to fbuild build profiles, including a real debug-oriented path rather than only release / quick
3. Apply build_unflags against framework/toolchain/default/user flags in the same stage where the final effective flag set is assembled
4. Detect extra_scripts / SCons flag mutation patterns and either:
   • support a constrained compatibility subset, or
   • fail fast with a clear diagnostic that advanced scripting is not yet supported
5. Add tests with representative fixtures for:
   • a plain build_type = debug environment
   • a build_unflags override (for example removing a default -std= or warning flag)
   • a project that depends on extra_scripts to mutate CPPDEFINES / CCFLAGS / LINKFLAGS

Why this matters

Without this, �compatible with platformio.ini� is true for basic environments but not for a common class of real-world PlatformIO projects. Those projects often rely on advanced build types specifically to control optimization/debug info, remove framework defaults, or inject profile-specific compile/link behavior. If fbuild ignores that layer, migration can succeed syntactically while diverging semantically.

PlatformIO references

• Build configurations (build_type, debug_build_flags): https://docs.platformio.org/en/stable/projectconf/build_configurations.html
• build_flags SCons scopes and dynamic flags: https://docs.platformio.org/en/stable/projectconf/sections/env/options/build/build_flags.html
• Section [env] includes build_type, build_flags, build_src_flags, build_unflags, build_src_filter: https://docs.platformio.org/en/latest/projectconf/sections/env/index.html
• Construction environments / extra_scripts mutating env and projenv: https://docs.platformio.org/en/latest/scripting/construction_environments.html

[zackees]

Investigation split into child issues:

• Fail fast on unsupported advanced PlatformIO build options in platformio.ini #35 Fail fast on unsupported advanced PlatformIO build options in platformio.ini
• Add a native compatibility shim for build_type=debug and debug_build_flags #36 Add a native compatibility shim for build_type=debug and debug_build_flags
• Apply PlatformIO build_unflags to the effective native flag scopes #37 Apply PlatformIO build_unflags to the effective native flag scopes
• Define native support boundaries for PlatformIO extra_scripts env/projenv mutations #38 Define native support boundaries for PlatformIO extra_scripts env/projenv mutations
• Research whether PlatformIO __idedata / project metadata can support a scripted-project bridge #39 Research whether PlatformIO __idedata / project metadata can support a scripted-project bridge

Current read from the research:

• Declarative features (build_type=debug, debug_build_flags, build_unflags) look like a good fit for native compatibility work.
• extra_scripts is the real boundary. Once arbitrary Python mutates SCons environments, full parity likely means either a very constrained shim or explicit fallback to --platformio.
• PlatformIO's metadata dump looks useful for editor/intellisense parity, but probably not rich enough by itself for full build/link parity.

That points toward option (2) unless the spike in #39 turns up something unexpectedly stronger.

[zackees]

Additional child issue from deeper investigation: #40 tracks the mismatch between declared-supported PLATFORMIO_* overrides and the config getters that actually consume them.

[zackees]

Added #41 for the concrete runtime-bridge experiment: execute PlatformIO/SCons just far enough to dump resolved build state, then feed that into native fbuild.

[zackees]

Runtime-bridge analysis

I think the promising path is not a parser, but a runtime bridge:

1. run PlatformIO/SCons far enough for PRE scripts, platform/framework scripts, library logic, and POST scripts to mutate the construction environments
2. inject one final dump step
3. serialize the resolved build state into a machine-readable spec
4. let fbuild perform the real native compile/link/deploy from that spec

Why this looks feasible

• PlatformIO already executes PRE scripts, the main build script, and POST scripts before the integration-dump path: .cache/platformio-core/platformio/builder/main.py:167-180, .cache/platformio-core/platformio/builder/main.py:214-227
• PlatformIO?s docs say POST scripts can access both env and projenv, which is exactly the point where project-vs-global flag mutations have settled: https://docs.platformio.org/en/latest/scripting/construction_environments.html
• PlatformIO already serializes part of this state today (build_type, defines, includes, cc_flags, cxx_flags, tool paths, targets, flash image metadata): .cache/platformio-core/platformio/builder/tools/piointegration.py:141-173
• The normal build path already resolves board flags, build flags, framework scripts, debug behavior, and build_unflags in a well-defined order: .cache/platformio-core/platformio/builder/tools/piobuild.py:94-128

What this should NOT be

• not a static Python/SCons parser
• not a giant reimplementation of SCons semantics inside Rust
• not __idedata as-is, because some scripts may branch on integration/service stages and that dump is editor-oriented, not build-replay-oriented

What the prototype should dump

• global env scopes: CPPDEFINES, CPPPATH, CCFLAGS, CFLAGS, CXXFLAGS, ASFLAGS, LINKFLAGS, LIBPATH, LIBS
• projenv scopes separately
• tool paths (CC, CXX, AR, linker, objcopy, gdb when present)
• BUILD_TYPE, linker script info, flash image metadata
• selected source/object graph (PIOBUILDFILES and related nodes if available)
• ideally exact compile and link argv after SCons expansion

Decision impact

• If this dump can capture exact compile/link commands or a sufficiently rich resolved build spec, then many ?simple SCons flag files? are realistically bridgeable without full SCons hosting.
• If not, then the right boundary is still native declarative support plus explicit --platformio fallback for scripted projects.

That makes #41 the key experiment before deciding whether the long-term answer is a stronger version of option (2) or a real option (3).

[zackees]

PR #42 is up with the native extra_scripts runtime bridge implementation. It covers the constrained runtime path for common env / projenv flag mutations, applies the resulting overlay to compile/link phases, and fails fast on unsupported scripted behavior instead of silently drifting.

[zackees]

Merged in PR #42: the implementation follows the expanded (2) path discussed here. fbuild now has a constrained extra_scripts runtime bridge that executes simple PlatformIO-style script mutations, captures the resulting compile/link overlays, applies them to the native build, and fails fast for unsupported dynamic behavior instead of silently drifting. This keeps native builds compatible with common SCons flag workflows without committing the project to full SCons re-hosting.

Key boundary: supported script behavior is limited to straightforward flag/path/library mutations on common scopes (CPPDEFINES, CPPPATH, CCFLAGS, CFLAGS, CXXFLAGS, ASFLAGS, LINKFLAGS, LIBPATH, LIBS) plus env / post-projenv access. More dynamic script behavior should still fall back to --platformio.

[zackees]

Status — closing umbrella; remaining scope is independently tracked

The original five asks in this issue are mostly delivered. The remaining surface lives in the two independent child issues #38 and #39, each actionable on its own without needing this umbrella to coordinate them.

Delivered

1. Parse and honor build_type, debug_build_flags, build_unflags from platformio.ini — ✅ done in crates/fbuild-build/src/pipeline.rs (BuildContext::new reads all three; build_type = debug triggers apply_debug_build_type; build_unflags is applied across user / src / overlay link scopes)
2. Map PlatformIO build types to fbuild profiles — ✅ debug path is wired via apply_debug_build_type; release falls back to the existing BuildProfile::Release. A future full mapping (including test) can be filed separately if needed.
3. Apply build_unflags against framework/toolchain/user flag sets — ✅ closed by Apply PlatformIO build_unflags to the effective native flag scopes #37 (PRs feat(build): apply build_unflags to framework compile flags, ESP32 wired (#37) #63 and feat(build): wire build_unflags through all remaining platform compilers (#37) #64, across all platforms)
4. Detect extra_scripts and handle them — ✅ partially done (script_runtime::resolve_extra_script_overlay in crates/fbuild-build/src/script_runtime.rs runs scripts via a Python harness); the remaining design work on what script patterns to support is tracked in Define native support boundaries for PlatformIO extra_scripts env/projenv mutations #38
5. Add tests — ✅ crates/fbuild-build/src/pipeline.rs and crates/fbuild-config/src/ini_parser.rs cover build_type, debug_build_flags, build_unflags; extra_scripts cases covered in script_runtime.rs tests

Remaining active scope

• Define native support boundaries for PlatformIO extra_scripts env/projenv mutations #38 — Define the native support boundary for extra_scripts env/projenv mutations (what to support, what to reject, how to diagnose)
• Research whether PlatformIO __idedata / project metadata can support a scripted-project bridge #39 — Research whether PlatformIO __idedata / project metadata can back a scripted-project bridge

Both are independent and actionable without needing this umbrella as a coordination point. Closing.