Skip to content

refactor[codegen]: refactor venom builtins for uniformity and cleanliness#5001

Open
banteg wants to merge 15 commits into
vyperlang:masterfrom
banteg:fix/venom-reduced-builtin-kwargs
Open

refactor[codegen]: refactor venom builtins for uniformity and cleanliness#5001
banteg wants to merge 15 commits into
vyperlang:masterfrom
banteg:fix/venom-reduced-builtin-kwargs

Conversation

@banteg

@banteg banteg commented May 21, 2026

Copy link
Copy Markdown
Contributor

What I did

Follow-up to #4984 after maintainer review. Refactored Venom builtin lowering so lower_builtin() constructs one BuiltinCall entry object and every HANDLERS target consumes that object instead of each builtin having its own (node, ctx) entry plumbing.

This keeps the PR Venom-only while replacing the copied _get_kwarg_value style with shared infrastructure for reduced kwarg validation, default-filled compile-time constants, and default-filled runtime kwargs. Runtime kwargs are lowered by walking node.keywords in source order, and raw_create / create_from_blueprint materialize positional constructor args before keyword expressions so side effects follow source order.

How I did it

  • Added vyper/codegen_venom/builtins/_kwargs.py with BuiltinCall, uniform kwarg validation, default-filled get_kwarg_ast_constants(), and source-order get_kwarg_values().
  • Changed the Venom builtin dispatcher to pass BuiltinCall(node, ctx) to all builtin handlers.
  • Updated every Venom builtin handler to take BuiltinCall, including non-kwarg builtins, so the entrypoint shape is uniform across the whole HANDLERS registry.
  • Kept literal-only kwargs reduced and validated as AST constants; runtime kwargs are only lowered when codegen runs.
  • Added regression coverage for folded constants, duplicate/unexpected kwargs, and source-order runtime kwarg evaluation.

How to verify it

  • uv run --python 3.12 make lint
  • uv run --python 3.12 pytest --experimental-codegen tests/unit/compiler/venom tests/functional/builtins/codegen tests/functional/syntax/test_print.py -q

Commit message

refactor[codegen]: refactor venom builtins for uniformity and cleanliness

Centralize Venom builtin call lowering around a single BuiltinCall entry object. Use shared kwarg helpers for validation, default-filled folded constants, and source-order runtime kwarg lowering so each builtin follows the same shape instead of open-coding kwarg lookup.

Description for the changelog

Refactor Venom builtin lowering to use a uniform dispatcher entry object and shared source-order keyword argument handling.

@charles-cooper

Copy link
Copy Markdown
Member

hmm, i think the point is actually that _get_kwarg_value is a code smell that args are getting evaluated out of order. what i would like to see ideally is some shared infrastructure which makes all of these conform to a uniform spec. like we should probably have:

  • get_kwarg_values (compiles kwargs in source order and returns a dict from kwarg name to ir value)
  • get_kwarg_constants or get_kwarg_ast_constants (fold foldable kwargs and return a dict from kwarg name to constant ast values)
  • each builtin should also have a list of allowed kwarg names, which makes the validation uniform rather than needing adhoc implementation for every builtin

the keys of get_kwarg_values() should always be a superset of get_kwarg_constants(). the only difference is when they run (during validation vs during codegen), and what they return -- get_kwarg_constants should return ast nodes which are guaranteed to be compile-time constants (and maybe even have valid get_folded(), not sure without reading all the code), vs get_kwarg_values should return things which live in the runtime (VyperValue or IROperand or the like)

@banteg

banteg commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a normal follow-up commit for source order, no force push.

Runtime kwargs are now lowered by walking source-order node.keywords, and raw_create/create_from_blueprint materialize positional constructor args before keyword expressions. I also converted the existing raw_create source-order xfails into Venom-passing tests and added raw_call gas/value kwarg order coverage.

Comment thread tests/functional/builtins/codegen/test_create_functions.py
Comment thread vyper/codegen_venom/builtins/create.py Outdated
@banteg

banteg commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 9e9070f.

Changes:

  • replaced the reduced-kwarg lookup with shared get_kwarg_ast_constants() and get_kwarg_values() helpers;
  • added explicit allowed kwarg lists for the Venom builtins that lower kwargs here: abi_encode/abi_decode, raw/create variants, raw_call, send, and print;
  • kept runtime kwargs lowered by walking node.keywords in source order; literal-only kwargs go through folded AST constants and are not runtime-lowered unnecessarily;
  • added unit coverage for folded constants, unexpected kwargs, and presence checks.

I also checked the Venom builtin tree after the refactor; the only direct node.keywords iteration left is inside _kwargs.py. extract32(output_type=...) is type-only in semantic analysis and has no Venom runtime lowering.

Validation:

  • uv run --python 3.12 pytest --experimental-codegen tests/unit/compiler/venom/test_builtin_kwargs.py tests/functional/builtins/codegen/test_abi_encode.py tests/functional/builtins/codegen/test_create_functions.py tests/functional/builtins/codegen/test_raw_call.py -q
  • uv run --python 3.12 pytest tests/functional/builtins/codegen/test_create_functions.py::test_raw_create_order_of_eval_of_kwargs tests/functional/builtins/codegen/test_create_functions.py::test_raw_create_eval_order tests/functional/builtins/codegen/test_raw_call.py::test_raw_call_runtime_kwargs_source_order -q
  • uv run --python 3.12 ruff check vyper/codegen_venom/builtins/_kwargs.py vyper/codegen_venom/builtins/abi.py vyper/codegen_venom/builtins/create.py vyper/codegen_venom/builtins/system.py vyper/codegen_venom/builtins/misc.py tests/unit/compiler/venom/test_builtin_kwargs.py
  • git diff --check

Comment thread vyper/codegen_venom/builtins/create.py
@charles-cooper charles-cooper changed the title fix[codegen]: reduce builtin kwargs before parsing refactor[codegen]: refactor venom pipeline for uniformity and cleanliness May 21, 2026
@charles-cooper charles-cooper changed the title refactor[codegen]: refactor venom pipeline for uniformity and cleanliness refactor[codegen]: refactor venom builtins for uniformity and cleanliness May 21, 2026
Comment thread vyper/builtins/functions.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/abi.py Outdated
Comment thread vyper/codegen_venom/builtins/abi.py
Comment thread vyper/codegen_venom/builtins/system.py Outdated
Comment thread vyper/codegen_venom/context.py
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/create.py Outdated
Comment thread vyper/codegen_venom/builtins/create.py Outdated
Comment thread vyper/codegen_venom/builtins/create.py Outdated
Comment thread vyper/codegen_venom/builtins/create.py Outdated
Comment thread vyper/codegen_venom/builtins/create.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
Comment thread vyper/codegen_venom/builtins/_kwargs.py
Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
@charles-cooper

Copy link
Copy Markdown
Member

@banteg please fix merge conflicts



@dataclass(frozen=True)
class BuiltinCall:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

represents a builtin callsite with additional metadata

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a BuiltinCall docstring describing it as the builtin callsite plus the codegen context used to lower it.

Comment thread vyper/codegen_venom/builtins/_kwargs.py Outdated
from vyper.codegen_venom.expr import Expr

arg_nodes = self.node.args if arg_nodes is None else arg_nodes
return [Expr(arg, self.ctx).lower() for arg in arg_nodes]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lower() or lower_value() should be called exactly once for every single arg/kwarg, in order, and it's not clear by inspection that (or why) this invariant holds

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified the invariant at the helper boundary: positional args are lowered in AST/source order, and explicit runtime kwargs are lowered once by iterating the collected keyword nodes in the user's keyword order rather than by allowed-name order.

Comment thread vyper/codegen_venom/context.py
@banteg

banteg commented May 24, 2026

Copy link
Copy Markdown
Contributor Author

Resolved the merge conflicts by merging current origin/master into fix/venom-reduced-builtin-kwargs and pushing without force-pushing.

I kept the refactored BuiltinCall.get_kwarg_values() path for raw_call and added a follow-up docs commit (3ea8f270b) to make the builtin callsite metadata and source-order/one-lowering-helper invariant explicit in _kwargs.py.

Validation:

  • .venv/bin/black --check -C --force-exclude=vyper/version.py vyper/codegen_venom/builtins/_kwargs.py vyper/codegen_venom/builtins/system.py
  • .venv/bin/pytest tests/unit/compiler/venom/test_builtin_kwargs.py tests/functional/builtins/codegen/test_raw_call.py -q (41 passed, 1 xfailed)
  • .venv/bin/pytest tests/functional/builtins/codegen/test_create_functions.py tests/functional/builtins/codegen/test_extract32.py tests/functional/builtins/codegen/test_as_wei_value.py tests/functional/builtins/codegen/test_convert.py -q (1583 passed, 2 xfailed)

@charles-cooper

Copy link
Copy Markdown
Member

Resolved the merge conflicts by merging current origin/master into fix/venom-reduced-builtin-kwargs and pushing without force-pushing.

I kept the refactored BuiltinCall.get_kwarg_values() path for raw_call and added a follow-up docs commit (3ea8f270b) to make the builtin callsite metadata and source-order/one-lowering-helper invariant explicit in _kwargs.py.

Validation:

  • .venv/bin/black --check -C --force-exclude=vyper/version.py vyper/codegen_venom/builtins/_kwargs.py vyper/codegen_venom/builtins/system.py
  • .venv/bin/pytest tests/unit/compiler/venom/test_builtin_kwargs.py tests/functional/builtins/codegen/test_raw_call.py -q (41 passed, 1 xfailed)
  • .venv/bin/pytest tests/functional/builtins/codegen/test_create_functions.py tests/functional/builtins/codegen/test_extract32.py tests/functional/builtins/codegen/test_as_wei_value.py tests/functional/builtins/codegen/test_convert.py -q (1583 passed, 2 xfailed)

re. the lowering invariant, i didn't really mean a docs clarification. i meant a code change so that the invariant is clear by inspection. like i think better would be if BuiltinCall calls lower()/lower_value() on all arguments as a pre-processing step before lower_<builtin> even gets called...

@banteg

banteg commented May 24, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a normal follow-up commit, 3e8fa3acf, for the code-side invariant.

BuiltinCall now pre-lowers runtime positional args and runtime kwargs before dispatching to lower_<builtin>, then handlers consume the cached VyperValues / unwrapped operands instead of calling Expr(...) again. I also wired create aliases into the runtime-kwarg map and updated the touched handlers so the source-order lowering path is visible at the call wrapper.

I left the explicit opt-outs (len, slice, raw_log) in the runtime-arg map because they have ad hoc lowering for msg.data/code or nested log topics; the opt-out is now centralized instead of implicit in each handler.

Validation:

  • .venv/bin/black --check -C --force-exclude=vyper/version.py vyper/codegen_venom/builtins
  • .venv/bin/ruff check vyper/codegen_venom/builtins
  • .venv/bin/pytest tests/unit/compiler/venom/test_builtin_kwargs.py tests/functional/builtins/codegen/test_raw_call.py -q (41 passed, 1 xfailed)
  • .venv/bin/pytest tests/functional/builtins/codegen/test_create_functions.py tests/functional/builtins/codegen/test_extract32.py tests/functional/builtins/codegen/test_as_wei_value.py tests/functional/builtins/codegen/test_convert.py -q (1583 passed, 2 xfailed)
  • .venv/bin/pytest tests/functional/builtins/codegen/test_slice.py tests/functional/builtins/codegen/test_extract32.py -q (81 passed)
  • .venv/bin/pytest tests/functional/builtins/codegen/test_concat.py tests/functional/builtins/codegen/test_keccak256.py tests/functional/builtins/codegen/test_sha256.py tests/functional/builtins/codegen/test_unsafe_math.py tests/functional/builtins/codegen/test_floor.py tests/functional/builtins/codegen/test_ceil.py tests/functional/builtins/codegen/test_uint2str.py tests/functional/builtins/codegen/test_length.py tests/functional/builtins/codegen/test_blobhash.py tests/functional/builtins/codegen/test_ec.py tests/functional/builtins/codegen/test_ecrecover.py -q (121 passed)
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants