-
Notifications
You must be signed in to change notification settings - Fork 17
v8: add pre-built V8 library build and distribution infrastructure #3872
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
jwendell
wants to merge
1
commit into
envoyproxy:main
Choose a base branch
from
jwendell:v8-1
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+348
−0
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| licenses(["notice"]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Script to build V8's wee8 static library and package it for distribution. | ||
| # | ||
| # This script uses the self-contained Bazel workspace in bazel/v8/build/ which | ||
| # has all V8 dependencies pinned. It does NOT depend on the Envoy repository. | ||
| # | ||
| # Usage: | ||
| # ./build_v8.sh [--arch x86_64] [--output-dir /tmp/v8-out] [--bazel-opts "..."] | ||
| # | ||
| # The resulting tarball can be uploaded to GitHub releases for consumption by | ||
| # the v8_prebuilt repository rule. | ||
|
|
||
| set -e -o pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| BUILD_DIR="$SCRIPT_DIR/build" | ||
| ARCH="x86_64" | ||
| OUTPUT_DIR="/tmp/v8-prebuilt" | ||
| BAZEL_BUILD_OPTS="" | ||
|
|
||
| usage() { | ||
| echo "Usage: $0 [options]" | ||
| echo "" | ||
| echo "Options:" | ||
| echo " --arch Target architecture (x86_64 or aarch64, default: x86_64)" | ||
| echo " --output-dir Output directory for the tarball (default: /tmp/v8-prebuilt)" | ||
| echo " --bazel-opts Additional Bazel build options" | ||
| echo "" | ||
| exit 1 | ||
| } | ||
|
|
||
| while [[ $# -gt 0 ]]; do | ||
| case $1 in | ||
| --arch) | ||
| ARCH="$2" | ||
| shift 2 | ||
| ;; | ||
| --output-dir) | ||
| OUTPUT_DIR="$2" | ||
| shift 2 | ||
| ;; | ||
| --bazel-opts) | ||
| BAZEL_BUILD_OPTS="$2" | ||
| shift 2 | ||
| ;; | ||
| *) | ||
| echo "Unknown option: $1" | ||
| usage | ||
| ;; | ||
| esac | ||
| done | ||
|
|
||
| if [[ ! -f "$BUILD_DIR/WORKSPACE" ]]; then | ||
| echo "Error: Cannot find $BUILD_DIR/WORKSPACE" | ||
| echo "This script must be run from the envoy_toolshed repository." | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Extract V8 version from the WORKSPACE file | ||
| V8_VERSION=$(grep '^V8_VERSION' "$BUILD_DIR/WORKSPACE" | sed 's/.*"\(.*\)".*/\1/') | ||
| if [[ -z "$V8_VERSION" ]]; then | ||
| echo "Error: Could not determine V8 version from $BUILD_DIR/WORKSPACE" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Building V8 wee8 library:" | ||
| echo " V8 version: $V8_VERSION" | ||
| echo " Architecture: $ARCH" | ||
| echo " Build dir: $BUILD_DIR" | ||
| echo " Output dir: $OUTPUT_DIR" | ||
| echo "" | ||
|
|
||
| WORK_DIR=$(mktemp -d) | ||
| trap 'rm -rf "$WORK_DIR"' EXIT | ||
|
|
||
| # Step 1: Build V8's wee8 target using the self-contained build workspace | ||
| echo "Step 1: Building @v8//:wee8 ..." | ||
| cd "$BUILD_DIR" | ||
| # shellcheck disable=SC2086 | ||
| bazel build -c opt @v8//:wee8 $BAZEL_BUILD_OPTS | ||
|
|
||
| # Step 2: Collect all .o files from V8 and its V8-specific dependencies | ||
| echo "" | ||
| echo "Step 2: Collecting object files ..." | ||
|
|
||
| BAZEL_BIN=$(bazel info -c opt output_path)/k8-opt/bin | ||
| BAZEL_EXTERNAL="$BAZEL_BIN/external" | ||
|
|
||
| # Bazel puts compiled objects in _objs/ subdirectories, not in .a archives. | ||
| # Collect .o files from V8 and V8-specific deps (NOT abseil-cpp or llvm_toolchain). | ||
| ALL_OBJECTS=() | ||
| for dep_dir in v8 dragonbox fast_float fp16 simdutf highway; do | ||
| if [[ -d "$BAZEL_EXTERNAL/$dep_dir" ]]; then | ||
| while IFS= read -r -d '' obj; do | ||
| ALL_OBJECTS+=("$obj") | ||
| done < <(find "$BAZEL_EXTERNAL/$dep_dir" -name '*.pic.o' -print0) | ||
| fi | ||
| done | ||
|
|
||
| if [[ ${#ALL_OBJECTS[@]} -eq 0 ]]; then | ||
| echo "Error: No object files found. Build may have failed." | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo " Found ${#ALL_OBJECTS[@]} object files" | ||
|
|
||
| # Step 3: Create a single static library from all object files | ||
| echo "" | ||
| echo "Step 3: Creating libwee8.a ..." | ||
|
|
||
| STAGING_DIR="$WORK_DIR/staging" | ||
| mkdir -p "$STAGING_DIR/lib" "$STAGING_DIR/include" | ||
|
|
||
| ar rcs "$STAGING_DIR/lib/libwee8.a" "${ALL_OBJECTS[@]}" | ||
| echo " Created libwee8.a ($(du -h "$STAGING_DIR/lib/libwee8.a" | cut -f1))" | ||
|
|
||
| # Step 4: Copy headers | ||
| echo "" | ||
| echo "Step 4: Copying headers ..." | ||
|
|
||
| V8_EXTERNAL_SRC=$(bazel info output_base)/external/v8 | ||
|
|
||
| # Copy all V8 public API headers (needed by proxy_wasm_cpp_host and other consumers) | ||
| cp -r "$V8_EXTERNAL_SRC/include" "$STAGING_DIR/" | ||
|
|
||
| # Copy wasm-api headers at both locations: | ||
| # - include/ for direct #include "wasm.h" usage | ||
| # - third_party/wasm-api/ for V8 internal #include "third_party/wasm-api/wasm.hh" | ||
| cp "$V8_EXTERNAL_SRC/third_party/wasm-api/wasm.h" "$STAGING_DIR/include/" | ||
| cp "$V8_EXTERNAL_SRC/third_party/wasm-api/wasm.hh" "$STAGING_DIR/include/" | ||
| mkdir -p "$STAGING_DIR/third_party/wasm-api" | ||
| cp "$V8_EXTERNAL_SRC/third_party/wasm-api/wasm.h" "$STAGING_DIR/third_party/wasm-api/" | ||
| cp "$V8_EXTERNAL_SRC/third_party/wasm-api/wasm.hh" "$STAGING_DIR/third_party/wasm-api/" | ||
|
|
||
| # Copy V8 internal headers required by proxy_wasm_cpp_host (src/wasm/c-api.h | ||
| # and its transitive includes). Uses a Python script to resolve the dependency | ||
| # tree and copy only what's needed. | ||
| python3 -c " | ||
| import re, os, shutil, sys | ||
| v8_root = sys.argv[1] | ||
| staging = sys.argv[2] | ||
| visited = set() | ||
| queue = ['src/wasm/c-api.h'] | ||
| while queue: | ||
| f = queue.pop(0) | ||
| if f in visited: | ||
| continue | ||
| visited.add(f) | ||
| src = os.path.join(v8_root, f) | ||
| if not os.path.exists(src): | ||
| continue | ||
| dst = os.path.join(staging, f) | ||
| os.makedirs(os.path.dirname(dst), exist_ok=True) | ||
| shutil.copy2(src, dst) | ||
| with open(src) as fh: | ||
| for line in fh: | ||
| m = re.match(r'#include\s+\"(src/[^\"]+)\"', line) | ||
| if m: | ||
| queue.append(m.group(1)) | ||
| print(' Copied %d internal V8 headers (src/)' % len(visited)) | ||
| " "$V8_EXTERNAL_SRC" "$STAGING_DIR" | ||
|
|
||
| HEADER_COUNT=$(find "$STAGING_DIR/include" "$STAGING_DIR/src" "$STAGING_DIR/third_party" -name '*.h' -o -name '*.hh' 2>/dev/null | wc -l) | ||
| echo " Total headers: $HEADER_COUNT" | ||
|
|
||
| # Step 5: Package | ||
| echo "" | ||
| echo "Step 5: Packaging ..." | ||
|
|
||
| mkdir -p "$OUTPUT_DIR" | ||
| TARBALL="$OUTPUT_DIR/v8-wee8-${V8_VERSION}-linux-${ARCH}.tar.xz" | ||
|
|
||
| cd "$STAGING_DIR" | ||
| tar -cJf "$TARBALL" lib/ include/ src/ third_party/ | ||
|
|
||
| echo " Created: $TARBALL" | ||
| echo " Size: $(du -h "$TARBALL" | cut -f1)" | ||
| echo "" | ||
| echo "SHA256: $(sha256sum "$TARBALL" | cut -d' ' -f1)" | ||
| echo "" | ||
| echo "Done!" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| """Module extension for pre-built V8 library configuration in bzlmod.""" | ||
|
|
||
| load(":v8_libs.bzl", "v8_prebuilt") | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is the bit i would hold off on in this pr - lets get the libs building then look at how to access them workspace/bzlmod |
||
|
|
||
| def _v8_libs_impl(module_ctx): | ||
| """Implementation of the v8_libs module extension.""" | ||
| setup_tag = None | ||
| for mod in module_ctx.modules: | ||
| for tag in mod.tags.setup: | ||
| if setup_tag == None: | ||
| setup_tag = tag | ||
| else: | ||
| fail("Multiple setup() calls found for v8_extension. Only one configuration is allowed since the repository name is fixed to @v8.") | ||
|
|
||
| if setup_tag: | ||
| v8_prebuilt( | ||
| name = "v8", | ||
| version = setup_tag.version, | ||
| sha256 = { | ||
| "x86_64": setup_tag.sha256_x86_64, | ||
| "aarch64": setup_tag.sha256_aarch64, | ||
| }, | ||
| ) | ||
|
|
||
| _setup = tag_class( | ||
| attrs = { | ||
| "version": attr.string( | ||
| mandatory = True, | ||
| doc = "V8 version to use", | ||
| ), | ||
| "sha256_x86_64": attr.string( | ||
| doc = "SHA256 hash for x86_64 architecture", | ||
| ), | ||
| "sha256_aarch64": attr.string( | ||
| doc = "SHA256 hash for aarch64 architecture", | ||
| ), | ||
| }, | ||
| ) | ||
|
|
||
| v8_extension = module_extension( | ||
| implementation = _v8_libs_impl, | ||
| tag_classes = { | ||
| "setup": _setup, | ||
| }, | ||
| ) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| """Repository rule for pre-built V8 (wee8) static library.""" | ||
|
|
||
| _ARCH_MAP = { | ||
| "amd64": "x86_64", | ||
| "x86_64": "x86_64", | ||
| "aarch64": "aarch64", | ||
| "arm64": "aarch64", | ||
| } | ||
|
|
||
| _BUILD_FILE_CONTENT = """ | ||
| package(default_visibility = ["//visibility:public"]) | ||
|
|
||
| cc_import( | ||
| name = "wee8_import", | ||
| static_library = "lib/libwee8.a", | ||
| alwayslink = True, | ||
| ) | ||
|
|
||
| cc_library( | ||
| name = "wee8", | ||
| hdrs = glob(["include/**/*.h", "include/**/*.hh", "src/**/*.h", "third_party/**/*.h", "third_party/**/*.hh"]), | ||
| defines = [ | ||
| "GOOGLE3", | ||
| "V8_ADVANCED_BIGINT_ALGORITHMS", | ||
| "V8_CONCURRENT_MARKING", | ||
| "V8_DEPRECATION_WARNINGS", | ||
| "V8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA", | ||
| "V8_ENABLE_EXTENSIBLE_RO_SNAPSHOT", | ||
| "V8_ENABLE_LAZY_SOURCE_POSITIONS", | ||
| "V8_ENABLE_MAGLEV", | ||
| "V8_ENABLE_SPARKPLUG", | ||
| "V8_ENABLE_TURBOFAN", | ||
| "V8_ENABLE_UNDEFINED_DOUBLE", | ||
| "V8_ENABLE_WEBASSEMBLY", | ||
| "V8_HAVE_TARGET_OS", | ||
| "V8_IMMINENT_DEPRECATION_WARNINGS", | ||
| "V8_TARGET_ARCH_X64", | ||
| "V8_TARGET_OS_LINUX", | ||
| "V8_TLS_USED_IN_LIBRARY", | ||
| "V8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64", | ||
| ], | ||
| includes = [".", "include", "third_party"], | ||
| deps = [ | ||
| ":wee8_import", | ||
| "@abseil-cpp//absl/container:btree", | ||
| "@abseil-cpp//absl/container:flat_hash_map", | ||
| "@abseil-cpp//absl/container:flat_hash_set", | ||
| "@abseil-cpp//absl/functional:overload", | ||
| "@abseil-cpp//absl/synchronization", | ||
| "@abseil-cpp//absl/time", | ||
| ], | ||
| visibility = ["//visibility:public"], | ||
| ) | ||
| """ | ||
|
|
||
| def _v8_prebuilt_impl(ctx): | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same with this |
||
| """Downloads pre-built V8 wee8 library from GitHub releases.""" | ||
|
|
||
| # Auto-detect host architecture | ||
| host_arch = ctx.os.arch | ||
| arch = _ARCH_MAP.get(host_arch) | ||
| if not arch: | ||
| fail("Unsupported host architecture for V8 pre-built: %s" % host_arch) | ||
|
|
||
| # Allow local testing via V8_PREBUILT_PATH environment variable. | ||
| # When set, it should point to a directory containing the tarball, e.g.: | ||
| # export V8_PREBUILT_PATH=/tmp/v8-prebuilt | ||
| local_path = ctx.os.environ.get("V8_PREBUILT_PATH", "") | ||
| if local_path: | ||
| tarball = "{path}/v8-wee8-{version}-linux-{arch}.tar.xz".format( | ||
| path = local_path, | ||
| version = ctx.attr.version, | ||
| arch = arch, | ||
| ) | ||
| ctx.extract(ctx.path(tarball)) | ||
| else: | ||
| sha256 = ctx.attr.sha256.get(arch, "") | ||
| if not sha256: | ||
| fail("No V8 pre-built SHA256 provided for architecture: %s" % arch) | ||
|
|
||
| ctx.download_and_extract( | ||
| url = "https://github.com/envoyproxy/toolshed/releases/download/v8-v{version}/v8-wee8-{version}-linux-{arch}.tar.xz".format( | ||
| version = ctx.attr.version, | ||
| arch = arch, | ||
| ), | ||
| sha256 = sha256, | ||
| ) | ||
| ctx.file("BUILD.bazel", _BUILD_FILE_CONTENT) | ||
|
|
||
| v8_prebuilt = repository_rule( | ||
| implementation = _v8_prebuilt_impl, | ||
| attrs = { | ||
| "version": attr.string( | ||
| mandatory = True, | ||
| doc = "V8 version (e.g., '14.6.202.10')", | ||
| ), | ||
| "sha256": attr.string_dict( | ||
| mandatory = True, | ||
| doc = "SHA256 hashes keyed by architecture (x86_64, aarch64)", | ||
| ), | ||
| }, | ||
| environ = ["V8_PREBUILT_PATH"], | ||
| doc = "Downloads pre-built V8 wee8 static library for use with proxy-wasm", | ||
| ) | ||
|
|
||
| def setup_v8_prebuilt(version, sha256): | ||
| """Setup function for WORKSPACE. | ||
|
|
||
| Creates @v8 repository with pre-built wee8 library. | ||
| The host architecture is auto-detected at fetch time. | ||
|
|
||
| Args: | ||
| version: V8 version to download (must be already published). | ||
| sha256: Dict of {arch: sha256} for each supported architecture. | ||
| """ | ||
| v8_prebuilt( | ||
| name = "v8", | ||
| version = version, | ||
| sha256 = sha256, | ||
| ) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we put the python in a python file - also you cant just call python - it would need the python toolchain to be available
also not seeing where this is actually called