Normalization & Filtering

This page describes the normalization and filtering stages in CmpStr. Both steps are applied to all input data before any metric or phonetic processing takes place and are essential for robust, language-agnostic, and application-specific string comparison.

Normalization

Normalization transforms input text into a canonical form based on a set of normalization flags. This ensures that superficial differences — such as case, whitespace, or diacritics — do not influence comparison results. Normalization is always performed before filtering and can also be used independently of the rest of the CmpStr API.

In CmpStr, normalization is handled by the Normalizer class. It supports both single strings and arrays of strings. The normalization behavior is defined by a flag string, where each character represents a specific transformation step.

When using CmpStr, normalization is applied automatically whenever flags are provided via options or through setFlags(). The normalization stage always precedes filtering and phonetic processing.

Flag Canonicalization and Caching

Normalization flags in CmpStr are canonicalized internally. This means that flags are always reordered into a deterministic, predefined sequence before the normalization pipeline is compiled. As a result:

• Flag strings such as itw, wti, or twi are treated as equivalent.
• Each logical flag combination corresponds to exactly one compiled normalization pipeline.
• Pipeline compilation and normalization results can be cached efficiently and reused safely.

The compiled pipeline is cached per canonical flag set, and normalized values are cached in an internal hashtable for optimal performance when processing large datasets or repeated comparisons.

Supported Flags

• d Normalize to NFD (Normalization Form Decomposed)
• i Case insensitive (convert to lowercase)
• k Remove non-letter characters
• n Remove non-number characters
• r Remove double characters (e.g., "ll" → "l")
• s Remove punctuation and special characters
• t Trim leading and trailing whitespace
• u Normalize to NFC (Normalization Form Composed)
• w Collapse whitespace (replace multiple spaces with a single space)
• x Normalize to NFKC (Normalization Form Compatibility Composed)

Example Usage

import { Normalizer } from 'cmpstr';

const input = '  Café Müller!! ';
const flags = 'ditws';

const normalized = Normalizer.normalize( input, flags );
// result: 'cafe muller'

Asynchronous Normalization

For large datasets or non-blocking workflows, normalization can also be performed asynchronously:

const result = await Normalizer.normalizeAsync( [ 'Hallo', '  hELLo', 'hey ', ' HolA ' ], 'iwt' );
// result: [ 'hallo', 'hello', 'hey', 'hola' ]

Try yourself at OneCompiler

Filtering

Filtering provides a flexible and extensible mechanism for additional preprocessing after normalization and before comparison. Filters can be used to implement custom cleaning, tokenization, masking, or domain-specific transformations.

Filters are managed by the Filter class. Each filter is associated with:

• a hook (a named processing stage, e.g. input)
• a unique ID
• a filter function
• optional configuration such as priority and activation state

Multiple filters can be attached to the same hook and are executed in order of ascending priority.

Like normalization, filter functions are pre-compiled as a pipeline for each hook and executed directly during computing, eliminating multiple sorting, calling, and batch processing across multiple filter functions.

A filter function receives a string and returns either a string (synchronous) or a Promise< string > (asynchronous):

type FilterFn = ( input: string ) => string;

Managing Filters

Filters can be added, paused, resumed, removed, or listed at runtime, allowing full control over the preprocessing pipeline.

import { CmpStr } from 'cmpstr';

// Replace all digits from input strings by an asterix (`*`)
CmpStr.filter.add( 'input', 'digits2Asterix', s => s.replace( /\d+/g, '*' ), { priority: 5 } );

// Pauses, resumes or removes the filter
CmpStr.filter.pause( 'input', 'digits2Asterix' );
CmpStr.filter.resume( 'input', 'digits2Asterix' );
CmpStr.filter.remove( 'input', 'digits2Asterix' );

Try yourself at OneCompiler

Listing Filters

Filters registered for a specific hook can be inspected using:

const activeFilters = CmpStr.filter.list( 'input', true ); // list only active filters
const allFilters = CmpStr.filter.list( 'input' ); // list all the filters

Clearing Filters

All filters for a specific hook — or all hooks — can be removed:

CmpStr.filter.clear( 'input' ); // remove all filters for the 'input' hook
CmpStr.filter.clear(); // remove all filters globally

Filter Options

• priority Execution order (lower number = higher priority, default: 10).
• active Whether the filter is currently active (default: true).
• overrideable Whether an existing filter with the same ID can be replaced (default true).

Normalization & Filtering Pipeline

When using CmpStr, all input data is processed through the following pipeline before any metric computation:

1. Normalization (according to canonicalized flags)
2. Filtering (all active filters for the relevant hook)
3. Phonetic processing (if configured)

This guarantees that all comparisons operate on consistently prepared input data, regardless of the chosen metric or comparison mode.