Skip to content

Add native JavaScript class bindings and optimized utilities #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
scorsin-oai wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add native JavaScript class bindings and optimized utilities #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
34a83ee
Add native JavaScript class bindings and optimized utilities
scorsin-oai Jul 23, 2026
f06d306
Fix native JavaScript class constructor semantics
scorsin-oai Jul 23, 2026
fe7e824
Add web Base64 implementation for native utilities
scorsin-oai Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions docs/docs/stdlib-filesystem.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ const binary = fs.readFileSync('image.png') as ArrayBuffer;
const bytes = new Uint8Array(binary);
```

#### `writeFileSync(path: string, data: ArrayBuffer | string): void`
#### `writeFileSync(path: string, data: ArrayBuffer | Uint8Array | string): void`

Synchronously writes data to a file, creating it if it doesn't exist or overwriting if it does.

Expand All @@ -106,7 +106,7 @@ fs.writeFileSync('output.txt', 'Hello, World!');

// Write binary
const imageData = new Uint8Array([...]); // image bytes
fs.writeFileSync('output.png', imageData.buffer);
fs.writeFileSync('output.png', imageData);

// Write JSON
const config = { setting1: true, setting2: 'value' };
Expand Down Expand Up @@ -352,4 +352,3 @@ const mockFs = {
currentWorkingDirectory: jest.fn(() => '/mock/path')
};
```

87 changes: 57 additions & 30 deletions libs/utils/src/utils/encoding/Base64Utils.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
#include "utils/encoding/Base64Utils.hpp"

#include <boost/algorithm/string.hpp>
#include <openssl/base64.h>

#include <algorithm>
#include <cstring>
#include <array>
#include <string>
#include <vector>

Expand Down Expand Up @@ -39,15 +38,10 @@ std::string uint64ToBase64(uint64_t data) {
return binaryToBase64(bytes.data(), bytes.size());
}

bool base64ToBinaryInternal(const char* encodedString, size_t inSize, std::vector<uint8_t>* ret) {
// Strip out newlines since the decoder returns an error code (0) if it finds them within the encoded string.
std::string noNewlines(encodedString, inSize);
boost::algorithm::replace_all(noNewlines, "\n", "");
boost::algorithm::replace_all(noNewlines, "\r", "");

static bool decodePreparedBase64(const std::string& preparedBase64, std::vector<uint8_t>* ret) {
// Determine the max array length we'll need given the string length
size_t maxOutSize = 0;
if (EVP_DecodedLength(&maxOutSize, noNewlines.length()) != 1 || maxOutSize == 0) {
if (EVP_DecodedLength(&maxOutSize, preparedBase64.length()) != 1 || maxOutSize == 0) {
return false;
}

Expand All @@ -56,8 +50,8 @@ bool base64ToBinaryInternal(const char* encodedString, size_t inSize, std::vecto
if (EVP_DecodeBase64(reinterpret_cast<uint8_t*>(ret->data()),
&outSize,
maxOutSize,
reinterpret_cast<const uint8_t*>(noNewlines.c_str()),
noNewlines.length()) != 1) {
reinterpret_cast<const uint8_t*>(preparedBase64.c_str()),
preparedBase64.length()) != 1) {
// make it empty to indicate error
ret->resize(0);
return false;
Expand All @@ -67,6 +61,17 @@ bool base64ToBinaryInternal(const char* encodedString, size_t inSize, std::vecto
return true;
}

static bool base64ToBinaryInternal(const char* encodedString, size_t inSize, std::vector<uint8_t>* ret) {
// Strip out newlines since the decoder returns an error code (0) if it finds them within the encoded string.
std::string noNewlines;
noNewlines.reserve(inSize);
noNewlines.append(encodedString, inSize);
noNewlines.erase(std::remove(noNewlines.begin(), noNewlines.end(), '\n'), noNewlines.end());
noNewlines.erase(std::remove(noNewlines.begin(), noNewlines.end(), '\r'), noNewlines.end());

return decodePreparedBase64(noNewlines, ret);
}

std::vector<uint8_t> base64ToBinary(std::string_view base64) {
std::vector<uint8_t> decodedData;
base64ToBinaryInternal(base64.data(), base64.size(), &decodedData);
Expand All @@ -77,6 +82,20 @@ bool base64ToBinary(std::string_view base64, std::vector<uint8_t>& decodedData)
return base64ToBinaryInternal(base64.data(), base64.size(), &decodedData);
}

bool base64UrlToBinary(std::string_view base64url, std::vector<uint8_t>& decodedData) {
std::string standardBase64;
standardBase64.reserve(base64url.size() + 4);
standardBase64.append(base64url.data(), base64url.size());
standardBase64.erase(std::remove(standardBase64.begin(), standardBase64.end(), '\n'), standardBase64.end());
standardBase64.erase(std::remove(standardBase64.begin(), standardBase64.end(), '\r'), standardBase64.end());
base64UrlToBase64InPlace(standardBase64);
if (standardBase64.empty()) {
decodedData.clear();
return true;
}
return decodePreparedBase64(standardBase64, &decodedData);
}

uint64_t base64ToUInt64(const std::string& base64) {
std::vector<uint8_t> bytes = base64ToBinary(base64);
uint64_t retVal = 0;
Expand All @@ -90,44 +109,52 @@ uint64_t base64ToUInt64(const std::string& base64) {
}

std::string base64UrlToBase64(const std::string& base64url) {
std::string temp;
temp.reserve(base64url.size() + 4);
std::string temp(base64url);
base64UrlToBase64InPlace(temp);
return temp;
}

void base64UrlToBase64InPlace(std::string& base64url) {
// change Base64 alphabet from urlsafe version to standard
for (const auto& c : base64url) {
for (auto& c : base64url) {
if (c == '-') {
temp += '+';
c = '+';
} else if (c == '_') {
temp += '/';
} else {
temp += c;
c = '/';
}
}

// add padding
if ((base64url.size() % 4) != 0u) {
int toAppend = 4 - static_cast<int>(base64url.size() % 4);
for (int i = 0; i < toAppend; i++) {
temp += '=';
}
auto toAppend = 4 - static_cast<int>(base64url.size() % 4);
base64url.append(toAppend, '=');
}

return temp;
}

std::string base64ToBase64Url(const std::string& base64) {
std::string temp(base64);
base64ToBase64UrlInPlace(temp);
return temp;
}

void base64ToBase64UrlInPlace(std::string& base64) {
// remove padding
size_t found = temp.find_last_not_of('=');
if (found == std::string::npos)
return "";
size_t found = base64.find_last_not_of('=');
if (found == std::string::npos) {
base64.clear();
return;
}

// change Base64 alphabet from standard version to urlsafe
boost::algorithm::replace_all(temp, "+", "-");
boost::algorithm::replace_all(temp, "/", "_");
for (auto& c : base64) {
if (c == '+') {
c = '-';
} else if (c == '/') {
c = '_';
}
}

return temp.substr(0, found + 1);
base64.resize(found + 1);
}

} // namespace snap::utils::encoding
19 changes: 19 additions & 0 deletions libs/utils/src/utils/encoding/Base64Utils.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,13 @@ std::vector<uint8_t> base64ToBinary(std::string_view base64);
*/
bool base64ToBinary(std::string_view base64, std::vector<uint8_t>& decoded);

/**
* @brief Base64-url decode the string and store the resulting bytes in the |decoded| vector.
* Newlines are stripped from the string before decoding. Missing padding is accepted.
* Returns false if failed to decode.
*/
bool base64UrlToBinary(std::string_view base64url, std::vector<uint8_t>& decoded);

/**
* @brief Base64 decode the string and store the first 8-bytes of the result in the
* returned uint64_t. If the result is more than 8-bytes then it will be truncated.
Expand All @@ -57,10 +64,22 @@ uint64_t base64ToUInt64(const std::string& base64);
*/
std::string base64UrlToBase64(const std::string& base64url);

/**
* @brief Convert the Base64 URL string to standard Base64 in place.
* See RFC 4648 'Table 2: The "URL and Filename safe" Base 64 Alphabet'
*/
void base64UrlToBase64InPlace(std::string& base64url);

/**
* @brief Convert the Base64 string to a Base64 URL string.
* See RFC 4648 'Table 2: The "URL and Filename safe" Base 64 Alphabet'
*/
std::string base64ToBase64Url(const std::string& base64);

/**
* @brief Convert the Base64 string to a Base64 URL string in place.
* See RFC 4648 'Table 2: The "URL and Filename safe" Base 64 Alphabet'
*/
void base64ToBase64UrlInPlace(std::string& base64);

} // namespace snap::utils::encoding
1 change: 1 addition & 0 deletions src/valdi_modules/src/valdi/coreutils/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ valdi_module(
visibility = ["//visibility:public"],
web_deps = [":coreutils_web"],
web_register_native_module_id_overrides = {
"coreutils/web/Base64Native.js": "coreutils/src/Base64Native",
"coreutils/web/UnicodeNative.js": "coreutils/src/UnicodeNative, coreutils/src/unicode/UnicodeNative",
},
deps = [
Expand Down
141 changes: 11 additions & 130 deletions src/valdi_modules/src/valdi/coreutils/src/Base64.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,147 +1,28 @@
/**
* Ported from https://github.com/beatgammit/base64-js/blob/master/index.js
*/

const lookup: string[] = [];
const revLookup: { [key: string]: number } = {};

const code = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
for (let i = 0, len = code.length; i < len; ++i) {
lookup[i] = code[i];
revLookup[code.charCodeAt(i)] = i;
}

// Support decoding URL-safe base64 strings, as Node.js does.
// See: https://en.wikipedia.org/wiki/Base64#URL_applications
revLookup['-'.charCodeAt(0)] = 62;
revLookup['_'.charCodeAt(0)] = 63;

function getLens(b64: string): [number, number] {
const len = b64.length;

if (len % 4 > 0) {
throw new Error('Invalid string. Length must be a multiple of 4');
}

// Trim off extra bytes after placeholder bytes are found
// See: https://github.com/beatgammit/base64-js/issues/42
let validLen = b64.indexOf('=');
if (validLen === -1) validLen = len;

const placeHoldersLen = validLen === len ? 0 : 4 - (validLen % 4);

return [validLen, placeHoldersLen];
}

function _byteLength(b64: string, validLen: number, placeHoldersLen: number): number {
return ((validLen + placeHoldersLen) * 3) / 4 - placeHoldersLen;
}

function tripletToBase64(num: number): string {
return lookup[(num >> 18) & 0x3f] + lookup[(num >> 12) & 0x3f] + lookup[(num >> 6) & 0x3f] + lookup[num & 0x3f];
}

function encodeChunk(uint8: Uint8Array, start: number, end: number): string {
let tmp: number;
const output: string[] = [];
for (let i = start; i < end; i += 3) {
tmp = ((uint8[i] << 16) & 0xff0000) + ((uint8[i + 1] << 8) & 0xff00) + (uint8[i + 2] & 0xff);
output.push(tripletToBase64(tmp));
}
return output.join('');
}
import { decodeFromBase64, encodeToBase64 } from './Base64Native';

@clholgat clholgat Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This swaps the pure-JS Base64 implementation for the C++-backed Base64Native module, but there's no web polyglot fallback for it — unlike the Unicode migration in this same PR.

coreutils/web/ only contains UnicodeNative.ts, and web_register_native_module_id_overrides in coreutils/BUILD.bazel only maps UnicodeNative.js; there's no Base64Native entry. On web, ./Base64Native won't resolve at runtime, so encodeToBase64/decodeFromBase64 will be undefined and Base64.fromByteArray / toByteArray / byteLength will throw.

Could you add a coreutils/web/Base64Native.ts mirroring UnicodeNative.ts, plus a matching entry in the override map?

@scorsin-oai scorsin-oai Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I was hoping that would all be part of my web PR which has this change, but I understand in the mean time this is breaking things on web.

@clholgat clholgat Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Realistically, it'll be at least a week before the web stuff lands at the earliest so we need a stop gap until then.


export namespace Base64 {
interface Base64Options {
urlSafe?: boolean;
}

export function fromByteArray(uint8: Uint8Array, { urlSafe }: Base64Options = {}): string {
let tmp: number;
const len = uint8.length;
const extraBytes = len % 3; // if we have 1 byte left, pad 2 bytes
const parts: string[] = [];
const maxChunkLength = 16383; // must be multiple of 3

// go through the array every three bytes, we'll deal with trailing stuff later
for (let i = 0, len2 = len - extraBytes; i < len2; i += maxChunkLength) {
parts.push(encodeChunk(uint8, i, i + maxChunkLength > len2 ? len2 : i + maxChunkLength));
}

// pad the end with zeros, but make sure to not forget the extra bytes
if (extraBytes === 1) {
tmp = uint8[len - 1];
parts.push(lookup[tmp >> 2] + lookup[(tmp << 4) & 0x3f] + '==');
} else if (extraBytes === 2) {
tmp = (uint8[len - 2] << 8) + uint8[len - 1];
parts.push(lookup[tmp >> 10] + lookup[(tmp >> 4) & 0x3f] + lookup[(tmp << 2) & 0x3f] + '=');
}

const base64String = parts.join('');

if (urlSafe) {
// Convert to URL-safe base64
return base64String.replace(/\+/g, '-').replace(/\//g, '_').replace(/=/g, '');
} else {
return base64String;
}
return encodeToBase64(uint8, urlSafe === true);
}

export function toByteArray(b64: string): Uint8Array {
let tmp = 0;

// Add padding to make length multiple of 4, if necessary
const originalLen = b64.length;
if (originalLen % 4 > 0) {
const paddingCount = 4 - (originalLen % 4);
b64 += '='.repeat(paddingCount);
}

const lens = getLens(b64);
const validLen = lens[0];
const placeHoldersLen = lens[1];

const arr = new Uint8Array(_byteLength(b64, validLen, placeHoldersLen));

let curByte = 0;

// if there are placeholders, only get up to the last complete 4 chars
const len = placeHoldersLen > 0 ? validLen - 4 : validLen;

let i = 0;
for (i = 0; i < len; i += 4) {
tmp =
(revLookup[b64.charCodeAt(i)] << 18) |
(revLookup[b64.charCodeAt(i + 1)] << 12) |
(revLookup[b64.charCodeAt(i + 2)] << 6) |
revLookup[b64.charCodeAt(i + 3)];
arr[curByte++] = (tmp >> 16) & 0xff;
arr[curByte++] = (tmp >> 8) & 0xff;
arr[curByte++] = tmp & 0xff;
}

if (placeHoldersLen === 2) {
tmp = (revLookup[b64.charCodeAt(i)] << 2) | (revLookup[b64.charCodeAt(i + 1)] >> 4);
arr[curByte++] = tmp & 0xff;
}

if (placeHoldersLen === 1) {
tmp =
(revLookup[b64.charCodeAt(i)] << 10) |
(revLookup[b64.charCodeAt(i + 1)] << 4) |
(revLookup[b64.charCodeAt(i + 2)] >> 2);
arr[curByte++] = (tmp >> 8) & 0xff;
arr[curByte++] = tmp & 0xff;
}

return arr;
return decodeFromBase64(b64);
}

// base64 is 4/3 + up to two characters of the original data
export function byteLength(b64: string): number {
const lens = getLens(b64);
const validLen = lens[0];
const placeHoldersLen = lens[1];
let validLen = b64.indexOf('=');
if (validLen === -1) validLen = b64.length;

if (validLen % 4 === 1) {
throw new Error('Invalid string. Length must be a valid Base64 length');
}

const placeHoldersLen = validLen === b64.length ? (4 - (validLen % 4)) % 4 : 4 - (validLen % 4);
return ((validLen + placeHoldersLen) * 3) / 4 - placeHoldersLen;
Comment on lines +21 to 26

@clholgat clholgat Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

byteLength no longer rejects malformed strings that contain = at a non-multiple-of-4 boundary, which the previous implementation threw on. The has-= branch uses 4 - (validLen % 4) without the % 4 wrap the no-padding branch has, and the only guard is validLen % 4 === 1, so:

  • byteLength('=') → -1 (validLen=0 → placeHoldersLen=4 → (0+4)*3/4 - 4 = -1)
  • byteLength('Zm9v=') → 2 (validLen=4 → placeHoldersLen=4 → 6 - 4 = 2)

Both threw before this change. A decoded-size helper returning a negative/incorrect value is risky for any caller that uses it to size an allocation. Suggest rejecting = at a non-%4 boundary and adding tests for these cases — the suite currently only covers byteLength('Z').

@scorsin-oai scorsin-oai Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't remember the exact specifics but I know I did that on purpose, byteLength() was more strict that our decoding code for no valid reason

@scorsin-oai scorsin-oai Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prior to this change: Base64 decode was lenient, it added missing padding before decoding, byteLength() was stricter, it did not accept some inputs that decode would accept. Now they accept unpadded inputs

@clholgat clholgat Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed the unpadded leniency itself is good — 'Zg' → 1 and 'Zm9v' → 3 now is right, and matching decode there makes sense.

The thing I was flagging is narrower: for a few malformed inputs byteLength now returns a value that can't be a real byte length —

  • byteLength('=') → -1
  • byteLength('==') → -1
  • byteLength('Zm9v=') → 2 (would expect 3, or a reject)

The -1 is the one I'd want to avoid: a caller using byteLength to size a new Uint8Array(n) / buffer gets a negative length. Root cause is the =-present branch using 4 - (validLen % 4) without the % 4 wrap the no-= branch has, so a leading/misplaced = (where validLen % 4 === 0) gives placeHoldersLen = 4 and the formula goes negative.

Not a blocker and fine to defer — but might be worth clamping to 0, or rejecting = at a non-multiple-of-4 offset. Your call.

}
}
3 changes: 3 additions & 0 deletions src/valdi_modules/src/valdi/coreutils/src/Base64Native.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export function encodeToBase64(uint8: Uint8Array, urlSafe: boolean): string;

export function decodeFromBase64(base64: string): Uint8Array;
Loading
Loading