[vic1707]

Hi,
following up on #2298 latest idea @angelcerveraroldan @prestist if you want to take a look.
Keeping it as draft for illustration purposes, assuming this idea is kept I'll close the Gomplate focused PR in favor of this one.

I'm starting to have a number of open PRs, sorry about that

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Binary size report (bin/amd64/ignition)

	Size
Base (main)	33MiB
PR (#2327)	33MiB
Delta	+14KiB (0.04%)

[prestist]

Hey @vic1707, thanks for putting this together! This is much closer to what I had in mind. I like that it’s opt-in, keeps the templating dependencies out of Butane, and invokes the processor directly without a shell. The stdin/stdout approach and failing translation when the processor fails make sense to me too.

A few things I think we should work through before settling the interface:

• What exactly should get processed? The flag currently says “every input file,” but storage.trees reads bypass the hook. I don’t think we should necessarily process those blindly either, especially with binary files or files containing template syntax meant for something else.
• Should we give the processor the source filename somehow? Right now it only receives the contents on stdin. Filename context could help it decide which files to render versus pass through unchanged, and make errors easier to understand.
• Could we pass the reader through the translation options instead of using a package-global setter? That would keep separate library callers from affecting one another.

Once we’ve agreed on those pieces, it would be good to add tests for processor failures, missing executables, disabled processing, and local-file handling alongside the happy-path test.

I ran the Butane tests and vet locally, and those passed. There’s also a small lint failure on the unchecked file.Close(), but that’s straightforward cleanup.

All that to say, I like this direction. I think the main thing now is agreeing on which files the processor sees and what context we give it. What do you think?

[vic1707]

Hi @prestist 👋, long time no see, hope you're doing fine.

Pushed 3 changes on what doesn't need more discussion (imo), the lint fix, putting the reader in options rather than module-global, and add filename to errors (error examples available in the commit description).

Remaining subjects would be

---

1. which files are processed

Too many edgecases imo

1. inline-based imports will always be text but could contain special chars
2. path-based imported files could be bins or contain special chars
3. trees can contain anything and everything.

Introducing an extension allowlist for processing (either as env or proper arg --file-processor-allowed-ext=txt,bu,templated.yml) would work.
But it needs to also apply to the main bu file (only way to control inline-based imports), which begs the question of "how do we handle stdin-provided main bu file?"

---

2. processor tools may ask/want additional settings (that can't be expressed via env vars which are already inherited)

I'm mainly thinking of gomplate's config file if not in current dir
Adding --file-processor-arg=--config=/path/to/gomplate.yaml feels natural but quid of multiple arguments? one per? simple string with space separated args (either parsed as a string or an array of string)?

note: current implementation makes gomplate search for the config file in the cwd of butane's process

---

3. inter-file processor context?

Concern that may only apply to me, and that was an issue on the last PR too.

What if I want to define a variable at runtime (ie: a secret app random token) that I need in multiple imported files? currently this context is lost. I don't see a way to fix that other than remaking the source.bu with every file-based import being turned into an inline-based import.

I would consider this out-of-scope and users would have to figure that out on their own (with gomplate it simply implies generating these values as a config file and use that with butane).

---

4. tools that don't accept stdin but only a file path (either as named or unnamed arg)

Explicit no support at least for now? I can't think of any tool like that, but asking for futureproof-ness.

[vic1707]

Hi, may I ask for an update on this @prestist / @angelcerveraroldan ?

[angelcerveraroldan]

I think that it may be nice to keep the API really simple to start with, and add things later if we see the need for them. A good starting point IMO would be to assume that the pre-processor uses stdin-stdout, and not pass any args to it.

If someone wanted to pass args to their tool, or if their tool does not use stdin-stdout, they can always make a simple wrapper scripts that handles that.

1. which files are processed

Introducing an extension allowlist for processing (either as env or proper arg --file-processor-allowed-ext=txt,bu,templated.yml) would work.

I think that this is the harder question. I think that an extension whitelist is nice option, as it avoids accidentally processing binary files / files that were meant to be processed by someone else. As for cases when the main butane file is passed through stdin, I would lean towards processing that too, but I'll have to think more about this, I am not too sure if I am missing any edge cases / weird UX.