diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 8a2d409706..238c465fa2 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -217,10 +217,7 @@ jobs: run: npm run build - name: Run Unit Tests - run: echo "npm run test:unit" - - - name: Run Integration Tests - run: echo "npm run test:integration" + run: npm run test:unit test-e2e: runs-on: ubuntu-latest diff --git a/docs/custom-fields.md b/docs/custom-fields.md new file mode 100644 index 0000000000..22e21e0e13 --- /dev/null +++ b/docs/custom-fields.md @@ -0,0 +1,199 @@ +# Custom Fields Architecture + +Custom fields let organizations explicitly select event data properties to index for use in filters and search. Indexing is forward-only: creating a definition affects new events and never mutates or reindexes historical events. This document covers the full lifecycle, slot system internals, deletion policy, and operator support. + +## Overview + +When an event is processed, the pipeline handler inspects the event's `Data` dictionary and writes typed values into the `Idx` sub-document using a **pooled slot** model. Rather than creating a unique Elasticsearch field per organization per field name (which would cause mapping explosion in a multi-tenant index), all organizations share a small pool of physical ES fields like `idx.keyword-1`, `idx.double-2`, etc. Each organization gets its own independent slot namespace: "department" for Org A and "department" for Org B both map to `idx.keyword-1` but are isolated by tenant-scoped queries. + +### Foundatio Integration + +The custom fields system is built on [Foundatio.Repositories.Elasticsearch custom fields](https://repositories.foundatio.dev/guide/custom-fields). Key components: + +- `IHaveVirtualCustomFields` — implemented by `PersistentEvent` to control how field values are read/written +- `ICustomFieldDefinitionRepository` — stores field definitions with slot assignments per `(EntityType, TenantKey, IndexType)` +- `EventCustomFieldService` — wires the document-changing pipeline hook and handles system field provisioning +- `EventIndex` — registers the 8 standard custom field types via `AddStandardCustomFieldTypes()` + +### Supported Field Types + +| Type | ES Mapping | Physical Slot Pattern | Filter Operators | +|------|-----------|----------------------|-----------------| +| `keyword` | Keyword (exact match) | `idx.keyword-{n}` | equals, not-equals, exists, missing | +| `string` | Text + `.keyword` sub-field | `idx.string-{n}` | contains/search, exists, missing | +| `int` | Integer | `idx.int-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `long` | Long | `idx.long-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `double` | Double | `idx.double-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `float` | Float | `idx.float-{n}` | equals, gt, gte, lt, lte, range, exists, missing | +| `bool` | Boolean | `idx.bool-{n}` | true, false, exists, missing | +| `date` | Date | `idx.date-{n}` | equals, range, gt, gte, lt, lte, exists, missing | + +> **Note on `string` cost**: Each `string` slot creates **two** Elasticsearch field mappers (the `text` field and its `.keyword` sub-field), making it twice as expensive as other types toward Elasticsearch's `index.mapping.total_fields.limit` (Exceptionless default 1,500). + +### Choosing a Field Type + +The configured type controls only the indexed representation in `Idx`; the original value in the event's `Data` dictionary is not changed. Use `keyword` for identifiers and versions whose exact text matters, even when they look numeric. Use `double` only when numeric range comparisons are intended. + +For example, a `DatabaseVersion` value of `"4.90"` is indexed as the exact string `"4.90"` when configured as `keyword`, but as the number `4.9` when configured as `double`. A development value such as `"4.90 build 1234 30-Aug-2024"` is valid as a `keyword` and is skipped as a `double` because it cannot be converted. In every case, the original `Data["DatabaseVersion"]` value remains unchanged. + +## Slot System + +### How Slots Are Assigned + +Slots are assigned **sequentially** per `(EntityType, TenantKey, IndexType)` scope: + +``` +Org A: "department" → keyword slot 1 → idx.keyword-1 +Org A: "region" → keyword slot 2 → idx.keyword-2 +Org B: "department" → keyword slot 1 → idx.keyword-1 ← same physical field, different tenant +Org B: "priority" → int slot 1 → idx.int-1 +``` + +Slot assignment is protected by a **distributed lock** per scope to prevent duplicate allocation under concurrent writes. + +### System Fields + +Three system fields are provisioned automatically per organization and are **protected from deletion**: + +| Field Name | Type | Slot | Purpose | +|-----------|------|------|---------| +| `sessionend` | `date` | `date-1` | Session end timestamp (session tracking) | +| `haserror` | `bool` | `bool-1` | Whether the session has an associated error | +| `@ref:session` | `keyword` | `keyword-1` | Session reference identifier | + +`EnsureSystemFieldsAsync` provisions these definitions before user-defined fields and verifies that each reserved name has the expected type and slot. It fails with a deterministic conflict if a reserved definition is duplicated, soft-deleted, assigned to the wrong slot or type, or if another definition already occupies a reserved slot. This prevents silently writing data to one slot while query resolution reads another. + +Legacy event documents can still contain the pre-pooled fields `idx.session-r`, `idx.sessionend-d`, and `idx.haserror-b`. Session filters expand across both the current pooled slot and the corresponding legacy field. Positive, range, and exists expressions use `OR`; missing expressions use `AND`; and negation wraps the combined expression. + +### Slot Exhaustion and Elasticsearch Field Limits + +Exceptionless configures `index.mapping.total_fields.limit` from `Elasticsearch:FieldsLimit`, which defaults to **1,500 field mappers** (Elasticsearch itself defaults to 1,000). Physical slot fields are only created in the index mapping the first time a document with that slot is indexed. The maximum Elasticsearch fields from custom fields is bounded by the highest slot number ever used, multiplied by number of types, multiplied by 2 (for `string` types). + +Two limits bound allocation per organization. `MaxFieldsPerOrganization` limits active user definitions, and `MaxLifetimeFieldsPerOrganization` limits all user definitions ever allocated, including soft-deleted definitions. Both default to 20; system fields are excluded. Existing organizations already above a reduced limit remain readable and indexable, but cannot allocate another slot. + +Retention-aware hard deletion and slot reclamation are not implemented. Deleted definitions continue to reserve their slots, and the lifetime ceiling prevents create/delete churn from growing the slot high-water mark without bound. + +## Field Lifecycle + +### Creating a Field + +1. User calls `POST /organizations/{id}/event-custom-fields` +2. API validates: premium plan check, reserved name check, active quota check, duplicate name check +3. `EnsureSystemFieldsAsync` provisions `sessionend`, `haserror`, and `@ref:session` if they are not yet present +4. `AddFieldAsync` assigns the next available slot and persists the definition +5. **From this point on, new events with matching data keys are indexed into the slot** +6. **Existing events are NOT backfilled** — they retain their original `Idx` content unchanged + +> **Search semantics on creation**: Custom field indexing applies only to events processed **after** the field definition is created. Historical events are not re-indexed. Both `data.fieldname:value` and `idx.fieldname:value` resolve to the new pooled slot, so neither expression searches a retained legacy named index. V1 has no built-in historical backfill. Elasticsearch reindexing alone is insufficient because it does not run the Exceptionless custom-field transform that populates pooled slots. Replaying original payloads can create duplicate events and requires an operator-owned deduplication plan. + +### Upgrade Cutover from Automatic Extended-Data Indexing + +Before this custom-field model, paid organizations automatically copied primitive extended-data values into named `Idx` fields and `data.fieldname:value` queries resolved to those legacy fields. This release intentionally replaces that behavior with explicit definitions: + +- Unregistered extended-data keys are no longer indexed. +- Existing named `Idx` values remain physically unchanged but are not queried through a new custom-field definition. +- Creating a definition indexes only events processed after creation; there is no automatic backfill. +- The Exceptionless-owned session fields (`@ref:session`, `sessionend`, and `haserror`) are the narrow compatibility exception and continue to read both legacy and pooled storage during the retention window. + +Before upgrading a self-hosted installation, inventory saved views and integrations that filter arbitrary `data.*` fields. After upgrading, create definitions for the fields that still matter before resuming ingestion when uninterrupted forward indexing is required. There is no supported general backfill in v1. + +### Updating a Field + +Only `Description` and `DisplayOrder` are mutable. `Name`, `IndexType`, and `IndexSlot` are immutable once created (enforced by Foundatio's repository at save time). + +### Deleting a Field + +Deletion is a synchronous soft-delete designed to prevent **slot reuse corruption** — where a recycled slot causes historical events for a deleted field to appear in queries for a new field with the same slot. + +1. API checks for usage in saved view filters — returns 409 Conflict if found +2. API marks `IsDeleted = true` and calls `SaveAsync` +3. The field name is freed from the slot system (a new field can use the same name) +4. The slot number is **not** freed — it remains occupied +5. New events no longer index data into this slot +6. API returns 204 No Content; the field disappears from the management UI + +> **Slot Reuse Safety**: If a slot is freed and immediately recycled for a new field, historical events within the retention window that had data for the old field will appear in queries for the new field. For example: delete "customer_id" (keyword-3), create "project_id" (gets keyword-3), then searching `project_id:acme` returns historical events where `customer_id` was `acme`. This is a data integrity violation. + +Hard-delete (slot freeing) is deferred until retention-aware cleanup is implemented. Until then, the slot number grows monotonically and is never reused for a different field. Deleting the owning organization is the exception: organization teardown permanently removes all of its custom-field definitions because no tenant data remains eligible for future queries. + +> **Search semantics on deletion**: After Phase 1, no new events write to the deleted slot and the field name no longer resolves in queries. Existing slot values remain in historical event documents until those events age out, but direct raw-slot queries are blocked by the repository query resolver. + +### Slot Recycling + +Slot recycling (reusing a freed slot number for a new field) is **currently deferred** to prevent data contamination within the retention window. See "Deleting a Field" above. Deleted definitions continue reserving their slots until retention-aware cleanup is implemented. + +**Name reuse is always safe**: After soft-deletion, the same field *name* can be immediately reused. The new definition gets the *next available* slot number (monotonically increasing), not the old slot. This means: + +``` +Field A created → keyword slot 1 (active) +Field B created → keyword slot 2 (active) +Field A deleted → soft-delete (slot 1 still occupied, name freed) +Field C created with same name → keyword slot 3 (new slot, no contamination) +Query for "Field C" → only returns events since Field C was created +``` + + + +The active field limit (`MaxFieldsPerOrganization`, default 20) counts only fields that are: +- Not soft-deleted (`IsDeleted = false`) +- Not system fields (`sessionend`, `haserror`, `@ref:session`) + +Soft-deleted fields do **not** count toward the active quota, but they do count toward `MaxLifetimeFieldsPerOrganization` (default 20). A deleted field therefore does not guarantee that another slot can be allocated. + +### Deletion Blocked by Saved Views + +If a custom field is referenced in any saved view filter for the organization, deletion is blocked with HTTP 409 Conflict. References are taken from the canonical parsed query tree, so quoted values and escaped text do not create false positives. Users must remove the logical `idx.{fieldName}` or `data.{fieldName}` reference from all saved views before deletion proceeds. + +## Plan Restrictions + +Custom fields require a paid plan. Organizations on the free plan receive HTTP 426 Upgrade Required when attempting to create a custom field. Existing fields are unaffected if an organization downgrades — they remain indexed but the management UI is read-only. + +## Security Model + +- All custom field API endpoints require authentication and verify organization ownership before any operation +- Field names are validated against a strict allowlist (`[a-zA-Z0-9_.\-]`, max 100 chars, no `@` prefix) +- Names starting with `@` are reserved for Exceptionless internal data keys (`@error`, `@request`, etc.) +- Users cannot access or modify custom fields belonging to other organizations (tenant isolation is enforced by the API handler's organization-access checks) +- System fields (`@ref:session`, `sessionend`, `haserror`) cannot be created, modified, or deleted via the API + +## Elasticsearch Mapping Considerations + +- Custom field slot templates are registered via `AddStandardCustomFieldTypes()` in `EventIndex` +- Startup applies and validates all eight typed templates and the bounded pooled-slot mappings on every current, aliased, non-expired event index before readiness; legacy suffix templates remain during mixed-version rollout +- An incompatible existing typed-slot mapping is a rollout blocker that requires an explicit migration; Elasticsearch cannot safely change an existing field type in place +- Templates use the pattern `idx.{type}-*` (e.g., `idx.keyword-*`, `idx.double-*`) +- Event indices predeclare every allocatable pooled slot through `MaxLifetimeFieldsPerOrganization + 1` (the extra slot accounts for reserved system fields). This keeps exact string, facet, and sort resolution stable when the newest daily partition has no value for a sparse slot. +- Monitor total field count relative to `index.mapping.total_fields.limit` (`Elasticsearch:FieldsLimit`, Exceptionless default 1,500) in high-volume deployments +- Each predeclared slot number reserves nine field mappers across the eight types: the `string` type creates a text mapper plus its `.keyword` mapper, and all other types create one. Startup rejects configurations where this pool would consume more than 80% of `Elasticsearch:FieldsLimit`, preserving headroom for the event schema. + +## Common Questions + +**Can I reuse a field name after deleting it?** +Yes, immediately. After soft-deletion, the field name is freed and can be used for a new field. The new field gets a **new** slot number (not the old one), which prevents historical events for the deleted field from appearing in queries for the new field. Slot numbers grow monotonically and are not recycled while retention-aware cleanup remains unimplemented. + +**Does the 20-field quota include soft-deleted fields?** +The active quota does not. The lifetime quota does. With both defaults set to 20, a soft-deleted field frees active capacity but not lifetime slot capacity. + +**Will deleting a field break existing queries?** +Saved view filters that reference the field are blocked at deletion time. Custom code cannot query `idx.keyword-N:value` directly because raw slot access is blocked. The Exceptionless query builder translates active field names to slot paths automatically. + +**Is there a per-type field limit?** +No. The active quota (`MaxFieldsPerOrganization = 20`) is a total across all types. There is no separate limit per type. + +**What happens if I downgrade my plan?** +Existing field definitions and indexed data are preserved. The custom fields management UI becomes read-only. New field creation requires re-upgrading. + +**Can I have more than 20 fields?** +Both limits default to 20 per organization. Self-hosted deployments can raise `MaxFieldsPerOrganization` and `MaxLifetimeFieldsPerOrganization`; the lifetime limit must be greater than or equal to the active limit. When only the active key is specified, the lifetime limit defaults to that value. Because all possible pooled slots are declared up front, raising the lifetime limit may also require raising `Elasticsearch:FieldsLimit`; invalid combinations fail during startup rather than later during event ingestion. + +**Can slot numbers grow unboundedly from field churn?** +No for a single organization under the default lifetime ceiling. Slot reclamation is still unavailable, so operators should monitor Elasticsearch total-field headroom across all organizations and retained daily indices. + +## Production Rollout and Recovery + +1. Deploy ingestion and API instances first. Every instance gates readiness on the shared schema-and-day marker; one distributed-lock holder installs and validates retained-index mappings, and waiting instances proceed only after that succeeds. Drain older writers before exposing typed writes. +2. Seed required definitions only after the backend fleet is current. Expose the management UI after exact, range, and facet canaries succeed. +3. Monitor the low-cardinality custom-field diagnostics for mapping/provision failures, conversion skips, lifetime-limit rejection, and Elasticsearch field-count headroom. Alert on any mapping or provisioning failure. +4. On failure, stop further rollout and definition mutations, preserve definitions and raw event data, fix forward, and repeat the canary. Never hard-delete definitions or reclaim slots during incident response. + +The two-phase indexing transform builds replacement managed slots away from the document. New-event mapping failures preserve raw `Data` while stripping untrusted managed slots; saved-event infrastructure failures abort the write rather than persisting a de-indexed event. diff --git a/docs/docs/filtering-and-searching.md b/docs/docs/filtering-and-searching.md index d0ba2f1a50..f3de5cc7e2 100644 --- a/docs/docs/filtering-and-searching.md +++ b/docs/docs/filtering-and-searching.md @@ -120,9 +120,11 @@ Specify a `date` or `numeric` range as part of the term. ## Custom Extended Data -All simple data types (`string`, `boolean`, `date`, `number`) that are stored in extended data will be indexed. _NOTE_: Field names will be lowercased and escaped. Any field name that is not a valid identifier (containing only letter and digits) or is longer than 25 characters will be ignored. +Extended-data properties are not indexed automatically. An organization administrator must first create a custom event field with the same name and choose its index type. The field name may contain ASCII letters, digits, underscores, dots, and dashes and may be up to 100 characters long. -**Example:** Lets assume that our events extended data contains a property called `Age` with a value of `18`. To search for this value our query would be `data.age:18`. +Custom-field indexing is forward-only. Only events processed after the definition is created are searchable through it; existing events are not backfilled. Both `data.age:18` and `idx.age:18` resolve to the organization's active custom-field definition. + +**Example:** If an administrator creates an integer custom field named `age`, events received afterward with an `age` value of `18` can be found with `data.age:18`. *** diff --git a/docs/docs/self-hosting/upgrading-self-hosted-instance.md b/docs/docs/self-hosting/upgrading-self-hosted-instance.md index dd0974989f..59c018b206 100644 --- a/docs/docs/self-hosting/upgrading-self-hosted-instance.md +++ b/docs/docs/self-hosting/upgrading-self-hosted-instance.md @@ -8,6 +8,18 @@ title: "Upgrading" **If you are upgrading from v1 or [v2](https://github.com/exceptionless/Exceptionless/releases/tag/v2.0.0) you will need to upgrade to [v3.0](https://github.com/exceptionless/Exceptionless/releases/tag/v3.0.0) before upgrading to the latest release.** +## Custom event field indexing cutover + +The custom event field release replaces automatic indexing of every primitive extended-data property with explicit, organization-scoped definitions. Before upgrading, inventory saved views and integrations that rely on arbitrary `data.*` filters. + +After upgrading: + +1. Create definitions for the extended-data fields that must remain searchable. +2. If uninterrupted forward indexing matters, create those definitions before resuming event ingestion. +3. Plan for forward-only indexing. V1 has no built-in historical backfill, and Elasticsearch reindexing alone does not populate pooled slots. Replaying original payloads is operator-owned and requires a deduplication strategy because it can create duplicate events. + +Existing legacy index values remain in Elasticsearch until their events age out, but new custom-field queries use pooled slots and do not search those legacy values. Exceptionless-owned session fields retain dual-read compatibility during this transition. + ## Upgrading from v7.1 to v8 We simplified the self hosting process by integrating the UI into the existing app images. As such `exceptionless/ui` docker images are deprecated and we recommend using `exceptionless/app`. diff --git a/src/Exceptionless.Core/Bootstrapper.cs b/src/Exceptionless.Core/Bootstrapper.cs index cfdd12902e..7cf7c06896 100644 --- a/src/Exceptionless.Core/Bootstrapper.cs +++ b/src/Exceptionless.Core/Bootstrapper.cs @@ -39,6 +39,7 @@ using Foundatio.Queues; using Foundatio.Repositories.Elasticsearch; using Foundatio.Repositories.Elasticsearch.Configuration; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Elasticsearch.Jobs; using Foundatio.Repositories.Migrations; using Foundatio.Resilience; @@ -78,6 +79,7 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddSingleton(s => s.GetRequiredService().Client); services.AddSingleton(s => s.GetRequiredService()); + services.AddSingleton(s => s.GetRequiredService().CustomFieldDefinitionRepository!); services.AddStartupAction(); services.AddSingleton(); @@ -178,6 +180,9 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddStartupAction(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/Exceptionless.Core/Configuration/AppOptions.cs b/src/Exceptionless.Core/Configuration/AppOptions.cs index 99da0ce9d3..5d85b0d21a 100644 --- a/src/Exceptionless.Core/Configuration/AppOptions.cs +++ b/src/Exceptionless.Core/Configuration/AppOptions.cs @@ -70,6 +70,7 @@ public class AppOptions public int BulkBatchSize { get; internal set; } public CacheOptions CacheOptions { get; internal set; } = null!; + public CustomFieldOptions CustomFieldOptions { get; internal set; } = null!; public MessageBusOptions MessageBusOptions { get; internal set; } = null!; public QueueOptions QueueOptions { get; internal set; } = null!; public StorageOptions StorageOptions { get; internal set; } = null!; @@ -124,6 +125,7 @@ public static AppOptions ReadFromConfiguration(IConfiguration config) catch { } options.CacheOptions = CacheOptions.ReadFromConfiguration(config, options); + options.CustomFieldOptions = CustomFieldOptions.ReadFromConfiguration(config, options); options.MessageBusOptions = MessageBusOptions.ReadFromConfiguration(config, options); options.QueueOptions = QueueOptions.ReadFromConfiguration(config, options); options.StorageOptions = StorageOptions.ReadFromConfiguration(config, options); diff --git a/src/Exceptionless.Core/Configuration/CustomFieldOptions.cs b/src/Exceptionless.Core/Configuration/CustomFieldOptions.cs new file mode 100644 index 0000000000..2908636b57 --- /dev/null +++ b/src/Exceptionless.Core/Configuration/CustomFieldOptions.cs @@ -0,0 +1,25 @@ +using Microsoft.Extensions.Configuration; + +namespace Exceptionless.Core.Configuration; + +public class CustomFieldOptions +{ + public int MaxFieldsPerOrganization { get; internal set; } + public int MaxLifetimeFieldsPerOrganization { get; internal set; } + + public static CustomFieldOptions ReadFromConfiguration(IConfiguration config, AppOptions appOptions) + { + int activeLimit = config.GetValue(nameof(MaxFieldsPerOrganization), 20); + int lifetimeLimit = config.GetValue(nameof(MaxLifetimeFieldsPerOrganization), activeLimit); + if (activeLimit <= 0) + throw new ArgumentOutOfRangeException(nameof(MaxFieldsPerOrganization), "Custom field active limit must be greater than zero."); + if (lifetimeLimit < activeLimit) + throw new ArgumentOutOfRangeException(nameof(MaxLifetimeFieldsPerOrganization), "Custom field lifetime limit must be greater than or equal to the active limit."); + + return new CustomFieldOptions + { + MaxFieldsPerOrganization = activeLimit, + MaxLifetimeFieldsPerOrganization = lifetimeLimit + }; + } +} diff --git a/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs b/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs index 6a869d0425..9fa183befc 100644 --- a/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs +++ b/src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs @@ -10,77 +10,6 @@ public static class PersistentEventExtensions { private static readonly char[] _commaSeparator = [',']; - public static void CopyDataToIndex(this PersistentEvent ev, string[]? keysToCopy = null) - { - if (ev.Data is null) - return; - - ev.Idx ??= new DataDictionary(); - - keysToCopy = keysToCopy?.Length > 0 ? keysToCopy : ev.Data.Keys.ToArray(); - - foreach (string key in keysToCopy.Where(k => !String.IsNullOrEmpty(k) && ev.Data.ContainsKey(k))) - { - string field = key.Trim().ToLowerInvariant(); - - if (field.StartsWith("@ref:")) - { - field = field.Substring(5); - if (!field.IsValidFieldName()) - continue; - - ev.Idx[field + "-r"] = ev.Data[key]?.ToString(); - continue; - } - - if (field.StartsWith('@') || ev.Data[key] is null) - continue; - - if (!field.IsValidFieldName()) - continue; - - var dataType = ev.Data[key]?.GetType(); - if (dataType is null) - continue; - - if (dataType == typeof(bool)) - { - ev.Idx[field + "-b"] = ev.Data[key]; - } - else if (dataType.IsNumeric()) - { - ev.Idx[field + "-n"] = ev.Data[key]; - } - else if (dataType == typeof(DateTime) || dataType == typeof(DateTimeOffset)) - { - ev.Idx[field + "-d"] = ev.Data[key]; - } - else if (dataType == typeof(string)) - { - string? input = ev.Data[key]?.ToString(); - if (String.IsNullOrEmpty(input) || input.Length >= 1000) - continue; - - if (input.GetJsonType() != JsonType.None) - continue; - - if (input[0] == '"') - input = input.TrimStart('"').TrimEnd('"'); - - if (Boolean.TryParse(input, out bool value)) - ev.Idx[field + "-b"] = value; - else if (DateTimeOffset.TryParse(input, out var dtoValue)) - ev.Idx[field + "-d"] = dtoValue; - else if (Decimal.TryParse(input, out decimal decValue)) - ev.Idx[field + "-n"] = decValue; - else if (Double.TryParse(input, out double dblValue) && !Double.IsNaN(dblValue) && !Double.IsInfinity(dblValue)) - ev.Idx[field + "-n"] = dblValue; - else - ev.Idx[field + "-s"] = input; - } - } - } - public static string? GetEventReference(this PersistentEvent ev, string name) { if (String.IsNullOrEmpty(name) || ev.Data is null) @@ -147,7 +76,7 @@ public static bool HasSessionEndTime(this PersistentEvent ev) return null; } - public static bool UpdateSessionStart(this PersistentEvent ev, DateTime lastActivityUtc, bool isSessionEnd = false) + public static bool UpdateSessionStart(this PersistentEvent ev, DateTime lastActivityUtc, bool isSessionEnd = false, bool hasError = false) { if (!ev.IsSessionStart()) return false; @@ -168,18 +97,19 @@ public static bool UpdateSessionStart(this PersistentEvent ev, DateTime lastActi if (isSessionEnd) { ev.Data[Event.KnownDataKeys.SessionEnd] = lastActivityUtc; - ev.CopyDataToIndex([Event.KnownDataKeys.SessionEnd]); } else { ev.Data.Remove(Event.KnownDataKeys.SessionEnd); - ev.Idx?.Remove(Event.KnownDataKeys.SessionEnd + "-d"); } + if (hasError) + ev.Data[Event.KnownDataKeys.SessionHasError] = true; + return true; } - public static PersistentEvent ToSessionStartEvent(this PersistentEvent source, ITextSerializer serializer, ILogger logger, DateTime? lastActivityUtc = null, bool? isSessionEnd = null, bool hasPremiumFeatures = true, bool includePrivateInformation = true) + public static PersistentEvent ToSessionStartEvent(this PersistentEvent source, ITextSerializer serializer, ILogger logger, DateTime? lastActivityUtc = null, bool? isSessionEnd = null, bool includePrivateInformation = true, bool hasError = false) { var startEvent = new PersistentEvent { @@ -238,10 +168,7 @@ public static PersistentEvent ToSessionStartEvent(this PersistentEvent source, I } if (lastActivityUtc.HasValue) - startEvent.UpdateSessionStart(lastActivityUtc.Value, isSessionEnd.GetValueOrDefault()); - - if (hasPremiumFeatures) - startEvent.CopyDataToIndex([]); + startEvent.UpdateSessionStart(lastActivityUtc.Value, isSessionEnd.GetValueOrDefault(), hasError); return startEvent; } diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs index 8765f9b19a..447f9ab2c4 100644 --- a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationMaintenanceWorkItemHandler.cs @@ -2,6 +2,7 @@ using Exceptionless.Core.Models; using Exceptionless.Core.Models.WorkItems; using Exceptionless.Core.Repositories; +using Exceptionless.Core.Services; using Foundatio.Jobs; using Foundatio.Lock; using Foundatio.Repositories; @@ -13,13 +14,15 @@ public class OrganizationMaintenanceWorkItemHandler : WorkItemHandlerBase { private readonly IOrganizationRepository _organizationRepository; private readonly BillingManager _billingManager; + private readonly EventCustomFieldService _eventCustomFieldService; private readonly TimeProvider _timeProvider; private readonly ILockProvider _lockProvider; - public OrganizationMaintenanceWorkItemHandler(IOrganizationRepository organizationRepository, ILockProvider lockProvider, BillingManager billingManager, TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(loggerFactory) + public OrganizationMaintenanceWorkItemHandler(IOrganizationRepository organizationRepository, ILockProvider lockProvider, BillingManager billingManager, EventCustomFieldService eventCustomFieldService, TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(loggerFactory) { _organizationRepository = organizationRepository; _billingManager = billingManager; + _eventCustomFieldService = eventCustomFieldService; _timeProvider = timeProvider; _lockProvider = lockProvider; } @@ -34,7 +37,8 @@ public override async Task HandleItemAsync(WorkItemContext context) const int LIMIT = 100; var wi = context.GetData()!; - Log.LogInformation("Received upgrade organizations work item. Upgrade Plans: {UpgradePlans}", wi.UpgradePlans); + Log.LogInformation("Received organization maintenance work item. UpgradePlans: {UpgradePlans} RemoveOldUsageStats: {RemoveOldUsageStats} EnsureSystemCustomFields: {EnsureSystemCustomFields}", + wi.UpgradePlans, wi.RemoveOldUsageStats, wi.EnsureSystemCustomFields); var results = await _organizationRepository.GetAllAsync(o => o.PageLimit(LIMIT)); while (results.Documents.Count > 0 && !context.CancellationToken.IsCancellationRequested) @@ -53,6 +57,18 @@ public override async Task HandleItemAsync(WorkItemContext context) foreach (var usage in organization.Usage.Where(u => u.Date < utcNow.Subtract(TimeSpan.FromDays(366))).ToList()) organization.Usage.Remove(usage); } + + if (wi.EnsureSystemCustomFields) + { + try + { + await _eventCustomFieldService.EnsureSystemFieldsAsync(organization.Id); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + Log.LogError(ex, "Error ensuring system custom fields for organization {OrganizationId}", organization.Id); + } + } } if (wi.UpgradePlans || wi.RemoveOldUsageStats) diff --git a/src/Exceptionless.Core/Models/PersistentEvent.cs b/src/Exceptionless.Core/Models/PersistentEvent.cs index 968a4acd37..a1e3426bff 100644 --- a/src/Exceptionless.Core/Models/PersistentEvent.cs +++ b/src/Exceptionless.Core/Models/PersistentEvent.cs @@ -2,12 +2,13 @@ using System.Diagnostics; using Exceptionless.Core.Attributes; using Exceptionless.Core.Extensions; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Models; namespace Exceptionless.Core.Models; [DebuggerDisplay("Id: {Id}, Type: {Type}, Date: {Date}, Message: {Message}, Value: {Value}, Count: {Count}")] -public class PersistentEvent : Event, IOwnedByOrganizationAndProjectAndStackWithIdentity, IHaveCreatedDate, IValidatableObject +public class PersistentEvent : Event, IOwnedByOrganizationAndProjectAndStackWithIdentity, IHaveCreatedDate, IValidatableObject, IHaveVirtualCustomFields { /// /// Unique id that identifies an event. @@ -52,6 +53,28 @@ public class PersistentEvent : Event, IOwnedByOrganizationAndProjectAndStackWith [MiniValidation.SkipRecursion] public DataDictionary? Idx { get; set; } + // IHaveVirtualCustomFields explicit implementation + IDictionary IHaveVirtualCustomFields.Idx => (IDictionary)(Idx ??= new DataDictionary()); + + public string GetTenantKey() => OrganizationId; + + public IDictionary GetCustomFields() + { + if (Data is null) return new DataDictionary(); + var result = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var kvp in Data.Where(kvp => !String.IsNullOrEmpty(kvp.Key) + && (!kvp.Key.StartsWith('@') || kvp.Key.StartsWith("@ref:", StringComparison.OrdinalIgnoreCase)) + && kvp.Value is string or bool or int or long or float or double or decimal or DateTime or DateTimeOffset)) + { + result[kvp.Key] = kvp.Value; + } + return result; + } + + public object GetCustomField(string name) => Data is not null && Data.TryGetValue(name, out var v) && v is not null ? v : null!; + public void SetCustomField(string name, object value) { Data ??= new DataDictionary(); Data[name] = value; } + public void RemoveCustomField(string name) => Data?.Remove(name); + public IEnumerable Validate(ValidationContext validationContext) { if (Date == DateTimeOffset.MinValue) diff --git a/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs index d619d1a3dd..2bd1d04ca6 100644 --- a/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs +++ b/src/Exceptionless.Core/Models/WorkItems/OrganizationMaintenanceWorkItem.cs @@ -4,4 +4,5 @@ public class OrganizationMaintenanceWorkItem { public bool UpgradePlans { get; set; } public bool RemoveOldUsageStats { get; set; } + public bool EnsureSystemCustomFields { get; set; } } diff --git a/src/Exceptionless.Core/Pipeline/035_CopySimpleDataToIdxAction.cs b/src/Exceptionless.Core/Pipeline/035_CopySimpleDataToIdxAction.cs deleted file mode 100644 index b85723efa2..0000000000 --- a/src/Exceptionless.Core/Pipeline/035_CopySimpleDataToIdxAction.cs +++ /dev/null @@ -1,29 +0,0 @@ -using Exceptionless.Core.Plugins.EventProcessor; -using Microsoft.Extensions.Logging; - -namespace Exceptionless.Core.Pipeline; - -[Priority(40)] -public class CopySimpleDataToIdxAction : EventPipelineActionBase -{ - public CopySimpleDataToIdxAction(AppOptions options, ILoggerFactory loggerFactory) : base(options, loggerFactory) { } - - public override Task ProcessAsync(EventContext ctx) - { - if (!ctx.Organization.HasPremiumFeatures) - return Task.CompletedTask; - - // TODO: Do we need a pipeline action to trim keys and remove null values that may be sent by other native clients. - ctx.Event.CopyDataToIndex([]); - int fieldCount = ctx.Event.Idx?.Count ?? 0; - AppDiagnostics.EventsFieldCount.Record(fieldCount); - if (fieldCount > 20 && _logger.IsEnabled(LogLevel.Warning)) - { - var ev = ctx.Event; - using (_logger.BeginScope(new ExceptionlessState().Organization(ctx.Organization.Id).Property("Event", new { ev.Date, ev.StackId, ev.Type, ev.Source, ev.Message, ev.Value, ev.Geo, ev.ReferenceId, ev.Tags, ev.Idx }))) - _logger.LogWarning("Event has {FieldCount} indexed fields", fieldCount); - } - - return Task.CompletedTask; - } -} diff --git a/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs b/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs index 580831e116..3ae40afa97 100644 --- a/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs +++ b/src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs @@ -89,7 +89,8 @@ private async Task ProcessManualSessionsAsync(ICollection contexts }); // try to update an existing session - string? sessionStartEventId = await UpdateSessionStartEventAsync(projectId, session.Key, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null); + bool sessionHasError = session.Any(ctx => String.Equals(ctx.Event.Type, Event.KnownTypes.Error, StringComparison.Ordinal)); + string? sessionStartEventId = await UpdateSessionStartEventAsync(projectId, session.Key, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null, sessionHasError); // do we already have a session start for this session id? if (!String.IsNullOrEmpty(sessionStartEventId) && sessionStartEvent is not null) @@ -100,7 +101,7 @@ private async Task ProcessManualSessionsAsync(ICollection contexts else if (String.IsNullOrEmpty(sessionStartEventId) && sessionStartEvent is not null) { // no existing session, session start is in the batch - sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null); + sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null, sessionHasError); sessionStartEvent.SetProperty("SetSessionStartEventId", true); } else if (String.IsNullOrEmpty(sessionStartEventId)) @@ -116,7 +117,7 @@ private async Task ProcessManualSessionsAsync(ICollection contexts } // create a new session start event - await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null); + await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, sessionEndEvent is not null, sessionHasError); } } } @@ -179,16 +180,17 @@ private async Task ProcessAutoSessionsAsync(ICollection contexts) session.ForEach(s => s.Event.SetSessionId(sessionId)); + bool identitySessionHasError = session.Any(ctx => String.Equals(ctx.Event.Type, Event.KnownTypes.Error, StringComparison.Ordinal)); if (isNewSession) { if (sessionStartEvent is not null) { - sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd()); + sessionStartEvent.Event.UpdateSessionStart(lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd(), identitySessionHasError); sessionStartEvent.SetProperty("SetSessionStartEventId", true); } else { - await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd()); + await CreateSessionStartEventAsync(firstSessionEvent, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd(), identitySessionHasError); } if (!lastSessionEvent.Event.IsSessionEnd()) @@ -203,7 +205,7 @@ private async Task ProcessAutoSessionsAsync(ICollection contexts) sessionStartEvent.IsCancelled = true; } - await UpdateSessionStartEventAsync(projectId, sessionId, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd()); + await UpdateSessionStartEventAsync(projectId, sessionId, lastSessionEvent.Event.Date.UtcDateTime, lastSessionEvent.Event.IsSessionEnd(), identitySessionHasError); } } } @@ -284,9 +286,9 @@ private Task SetIdentitySessionIdAsync(string projectId, string identity, return _cache.SetAsync(GetIdentitySessionIdCacheKey(projectId, identity), sessionId, _sessionTimeout); } - private async Task CreateSessionStartEventAsync(EventContext startContext, DateTime? lastActivityUtc, bool? isSessionEnd) + private async Task CreateSessionStartEventAsync(EventContext startContext, DateTime? lastActivityUtc, bool? isSessionEnd, bool hasError = false) { - var startEvent = startContext.Event.ToSessionStartEvent(_serializer, _logger, lastActivityUtc, isSessionEnd, startContext.Organization.HasPremiumFeatures, startContext.IncludePrivateInformation); + var startEvent = startContext.Event.ToSessionStartEvent(_serializer, _logger, lastActivityUtc, isSessionEnd, startContext.IncludePrivateInformation, hasError); var startEventContexts = new List { new(startEvent, startContext.Organization, startContext.Project) }; diff --git a/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs b/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs index 7fd742b21b..ec637ff15e 100644 --- a/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs +++ b/src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs @@ -8,10 +8,12 @@ using Foundatio.Caching; using Foundatio.Extensions.Hosting.Startup; using Foundatio.Jobs; +using Foundatio.Lock; using Foundatio.Messaging; using Foundatio.Queues; using Foundatio.Repositories.Elasticsearch; using Foundatio.Repositories.Elasticsearch.Configuration; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Elasticsearch.Queries.Builders; using Foundatio.Repositories.Serialization; using Foundatio.Resilience; @@ -23,6 +25,8 @@ namespace Exceptionless.Core.Repositories.Configuration; public sealed class ExceptionlessElasticConfiguration : ElasticConfiguration, IStartupAction { private readonly AppOptions _appOptions; + private readonly ICacheClient _cacheClient; + private readonly TimeProvider _timeProvider; public ExceptionlessElasticConfiguration( AppOptions appOptions, @@ -37,10 +41,13 @@ ILoggerFactory loggerFactory ) : base(workItemQueue, cacheClient, messageBus, serializer, timeProvider, resiliencePolicyProvider, loggerFactory) { _appOptions = appOptions; + _cacheClient = cacheClient; + _timeProvider = timeProvider; _logger.LogInformation("All new indexes will be created with {ElasticsearchNumberOfShards} Shards and {ElasticsearchNumberOfReplicas} Replicas", _appOptions.ElasticsearchOptions.NumberOfShards, _appOptions.ElasticsearchOptions.NumberOfReplicas); AddIndex(Stacks = new StackIndex(this)); AddIndex(Events = new EventIndex(this, serviceProvider, appOptions)); + AddCustomFieldIndex(_appOptions.ElasticsearchOptions.ScopePrefix + "customfields", appOptions.ElasticsearchOptions.NumberOfReplicas); AddIndex(Migrations = new MigrationIndex(this, _appOptions.ElasticsearchOptions.ScopePrefix + "migrations", appOptions.ElasticsearchOptions.NumberOfReplicas)); AddIndex(Organizations = new OrganizationIndex(this)); AddIndex(OAuthApplications = new OAuthApplicationIndex(this)); @@ -52,12 +59,33 @@ ILoggerFactory loggerFactory AddIndex(WebHooks = new WebHookIndex(this)); } - public Task RunAsync(CancellationToken shutdownToken = default) + public async Task RunAsync(CancellationToken shutdownToken = default) { if (_appOptions.ElasticsearchOptions.DisableIndexConfiguration) - return Task.CompletedTask; - - return ConfigureIndexesAsync(); + return; + + await ConfigureIndexesAsync(); + + // This schema gate is deliberately independent of Foundatio's numeric index-version + // configuration marker. Typed pooled slots do not require a v2 reindex, but every + // retained v1 daily partition must receive the templates before the host is ready. + // The date is part of the marker so the first host started each UTC day verifies the + // newly-created daily partition. A new schema version must use a new marker version. + string mappingMarkerKey = $"event-custom-field-mappings:v2:{_timeProvider.GetUtcNow():yyyyMMdd}"; + if ((await _cacheClient.GetAsync(mappingMarkerKey)).HasValue) + return; + + await using var mappingLock = await _lockProvider.AcquireAsync( + $"{mappingMarkerKey}:lock", + TimeSpan.FromMinutes(15), + true, + shutdownToken); + + if ((await _cacheClient.GetAsync(mappingMarkerKey)).HasValue) + return; + + await Events.EnsureCustomFieldMappingsAsync(() => mappingLock.RenewAsync(TimeSpan.FromMinutes(15))); + await _cacheClient.SetAsync(mappingMarkerKey, true, TimeSpan.FromDays(2)); } public override void ConfigureGlobalQueryBuilders(ElasticQueryBuilder builder) diff --git a/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs b/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs index 9c7b77b4cd..b8fce1fbf2 100644 --- a/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs +++ b/src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs @@ -2,6 +2,8 @@ using System.Reflection; using System.Text.Json; using System.Text.Json.Serialization.Metadata; +using Exceptionless; +using Elastic.Clients.Elasticsearch.Fluent; using Elastic.Clients.Elasticsearch.IndexManagement; using Elastic.Clients.Elasticsearch.Mapping; using Exceptionless.Core.Configuration; @@ -9,6 +11,7 @@ using Exceptionless.Core.Models; using Exceptionless.Core.Models.Data; using Exceptionless.Core.Repositories.Queries; +using Exceptionless.Core.Repositories.Queries.Visitors; using Exceptionless.Core.Serialization; using Foundatio.Caching; using Foundatio.Parsers.ElasticQueries; @@ -23,13 +26,44 @@ namespace Exceptionless.Core.Repositories.Configuration; public sealed class EventIndex : DailyIndex { + private const int CustomFieldMappersPerSlot = 9; + private const double MaximumCustomFieldMappingBudgetRatio = 0.8; + + private static readonly IReadOnlyDictionary _customFieldElasticsearchTypes = new Dictionary(StringComparer.Ordinal) + { + ["bool"] = "boolean", + ["date"] = "date", + ["double"] = "double", + ["float"] = "float", + ["int"] = "integer", + ["keyword"] = "keyword", + ["long"] = "long", + ["string"] = "text" + }; + private readonly ExceptionlessElasticConfiguration _configuration; private readonly IServiceProvider _serviceProvider; + private readonly int _maxCustomFieldSlot; public EventIndex(ExceptionlessElasticConfiguration configuration, IServiceProvider serviceProvider, AppOptions appOptions) : base(configuration, configuration.Options.ScopePrefix + "events", 1, doc => ((PersistentEvent)doc).Date.UtcDateTime) { + AddStandardCustomFieldTypes(); + _configuration = configuration; _serviceProvider = serviceProvider; + // System definitions reserve slot 1 for several types. A tenant can then consume the + // configured lifetime budget in that same type, so predeclare one additional slot. + // Keeping these mappings in the code model makes exact string resolution independent of + // which daily partition most recently received a value for a sparse pooled slot. + _maxCustomFieldSlot = appOptions.CustomFieldOptions.MaxLifetimeFieldsPerOrganization + 1; + int customFieldMapperBudget = checked(_maxCustomFieldSlot * CustomFieldMappersPerSlot); + int maximumCustomFieldMapperBudget = (int)Math.Floor(configuration.Options.FieldsLimit * MaximumCustomFieldMappingBudgetRatio); + if (customFieldMapperBudget > maximumCustomFieldMapperBudget) + { + throw new ArgumentOutOfRangeException( + nameof(appOptions.CustomFieldOptions.MaxLifetimeFieldsPerOrganization), + $"The custom field lifetime limit requires {customFieldMapperBudget} pooled field mappers, which exceeds the reserved 80% mapping budget of {maximumCustomFieldMapperBudget} for Elasticsearch FieldsLimit {configuration.Options.FieldsLimit}. Reduce MaxLifetimeFieldsPerOrganization or increase FieldsLimit."); + } if (appOptions.MaximumRetentionDays > 0) MaxIndexAge = TimeSpan.FromDays(appOptions.MaximumRetentionDays); @@ -53,12 +87,7 @@ public override void ConfigureIndexMapping(TypeMappingDescriptor dt - .Add("idx_bool", t => t.Match("*-b").Mapping(m => m.Boolean(s => { }))) - .Add("idx_date", t => t.Match("*-d").Mapping(m => m.Date(s => { }))) - .Add("idx_number", t => t.Match("*-n").Mapping(m => m.DoubleNumber(s => { }))) - .Add("idx_reference", t => t.Match("*-r").Mapping(m => m.Keyword(s => s.IgnoreAbove(256)))) - .Add("idx_string", t => t.Match("*-s").Mapping(m => m.Keyword(s => s.IgnoreAbove(1024))))) + .DynamicTemplates(templates => ConfigureCustomFieldDynamicTemplates(templates)) .Properties(p => p .SetupDefaults() .Keyword(e => e.OrganizationId) @@ -80,7 +109,9 @@ public override void ConfigureIndexMapping(TypeMappingDescriptor e.Count) .Boolean(e => e.IsFirstOccurrence) .FieldAlias(Alias.IsFirstOccurrence, a => a.Path(f => f.IsFirstOccurrence)) - .Object(e => e.Idx, o => o.Dynamic(DynamicMapping.True)) + .Object(e => e.Idx, o => o + .Dynamic(DynamicMapping.True) + .Properties(ConfigureCustomFieldSlotProperties)) .Object(e => e.Data, o => o.Properties(p2 => p2 .AddVersionMapping() .AddLevelMapping() @@ -133,6 +164,119 @@ public override async Task ConfigureAsync() await base.ConfigureAsync(); } + public async Task EnsureCustomFieldMappingsAsync(Func? renewLock = null) + { + var logger = Configuration.LoggerFactory.CreateLogger(); + DateTime utcNow = Configuration.TimeProvider.GetUtcNow().UtcDateTime; + var indexes = (await GetIndexesAsync(Version)) + .Where(index => index.CurrentVersion == Version) + .Where(index => !MaxIndexAge.HasValue || utcNow <= GetIndexExpirationDate(index.DateUtc)) + .ToList(); + + foreach (var index in indexes) + { + if (renewLock is not null) + await renewLock(); + + await ValidateExistingCustomFieldMappingsAsync(index.Index, logger); + + var response = await Configuration.Client.Indices.PutMappingAsync(mapping => + { + mapping.Indices(index.Index); + mapping.DynamicTemplates(templates => ConfigureCustomFieldDynamicTemplates(templates)); + mapping.Properties(properties => properties.Object(e => e.Idx, o => o + .Dynamic(DynamicMapping.True) + .Properties(ConfigureCustomFieldSlotProperties))); + }); + + logger.LogRequest(response); + if (response.IsValidResponse) + continue; + + AppDiagnostics.CustomFieldMappingFailures.Add(1); + string errorMessage = response.DebugInformation; + logger.LogError(response.ApiCallDetails.OriginalException, "Error updating typed custom field mappings on event index {Index}: {Message}", index.Index, errorMessage); + throw new ApplicationException($"Error updating typed custom field mappings on event index {index.Index}: {errorMessage}", response.ApiCallDetails.OriginalException); + } + } + + private async Task ValidateExistingCustomFieldMappingsAsync(string indexName, ILogger logger) + { + var response = await Configuration.Client.FieldCapsAsync(indexName, request => request + .Fields("*") + .IncludeEmptyFields()); + logger.LogRequest(response); + if (!response.IsValidResponse) + { + AppDiagnostics.CustomFieldMappingFailures.Add(1); + throw new ApplicationException($"Error reading custom field mappings from event index {indexName}: {response.DebugInformation}", response.ApiCallDetails.OriginalException); + } + + AppDiagnostics.CustomFieldMappedFieldCount.Record(response.Fields.Count); + if (response.Fields.Count >= _configuration.Options.FieldsLimit * 0.8) + logger.LogWarning("Event index {Index} has {FieldCount} mapped fields against a total field limit of {FieldLimit}", indexName, response.Fields.Count, _configuration.Options.FieldsLimit); + + foreach (var (fieldName, capabilities) in response.Fields) + { + if (!fieldName.StartsWith("idx.", StringComparison.Ordinal)) + continue; + + string slotName = fieldName["idx.".Length..]; + int separatorIndex = slotName.LastIndexOf('-'); + if (separatorIndex <= 0 || !Int32.TryParse(slotName.AsSpan(separatorIndex + 1), out _)) + continue; + + string slotType = slotName[..separatorIndex]; + if (!_customFieldElasticsearchTypes.TryGetValue(slotType, out string? expectedType)) + continue; + + if (capabilities.ContainsKey(expectedType)) + continue; + + AppDiagnostics.CustomFieldMappingFailures.Add(1); + string actualTypes = String.Join(", ", capabilities.Keys.Order(StringComparer.Ordinal)); + throw new InvalidOperationException( + $"Event index '{indexName}' has incompatible mapping for '{fieldName}'. Expected '{expectedType}', found '{actualTypes}'. An explicit migration is required before rollout."); + } + } + + private void ConfigureCustomFieldDynamicTemplates(FluentCollectionOfKeyValuePairOfStringDynamicTemplate templates) + { + // Retain legacy suffix templates while older ingestion nodes may still write them. + templates + .Add("idx_legacy_bool", template => template.PathMatch("idx.*").Match("*-b").Mapping(mapping => mapping.Boolean())) + .Add("idx_legacy_date", template => template.PathMatch("idx.*").Match("*-d").Mapping(mapping => mapping.Date())) + .Add("idx_legacy_number", template => template.PathMatch("idx.*").Match("*-n").Mapping(mapping => mapping.DoubleNumber())) + .Add("idx_legacy_reference", template => template.PathMatch("idx.*").Match("*-r").Mapping(mapping => mapping.Keyword(keyword => keyword.IgnoreAbove(256)))) + .Add("idx_legacy_string", template => template.PathMatch("idx.*").Match("*-s").Mapping(mapping => mapping.Keyword(keyword => keyword.IgnoreAbove(1024)))); + + foreach (var customFieldType in CustomFieldTypes.Values) + { + templates.Add( + $"idx_{customFieldType.Type}", + template => template + .PathMatch("idx.*") + .Match($"{customFieldType.Type}-*") + .Mapping(customFieldType.ConfigureMapping())); + } + } + + private void ConfigureCustomFieldSlotProperties(PropertiesDescriptor properties) + { + for (int slot = 1; slot <= _maxCustomFieldSlot; slot++) + { + properties + .Boolean($"bool-{slot}") + .Date($"date-{slot}") + .DoubleNumber($"double-{slot}") + .FloatNumber($"float-{slot}") + .IntegerNumber($"int-{slot}") + .Keyword($"keyword-{slot}") + .LongNumber($"long-{slot}") + .Text($"string-{slot}", text => text.AddKeywordField()); + } + } + protected override void ConfigureQueryParser(ElasticQueryParserConfiguration config) { config @@ -153,7 +297,7 @@ protected override void ConfigureQueryParser(ElasticQueryParserConfiguration con EventIndexExtensions.DataPath(Event.KnownDataKeys.UserInfo, u => u.Identity), EventIndexExtensions.DataPath(Event.KnownDataKeys.UserInfo, u => u.Name) ]) - .AddQueryVisitor(new EventFieldsQueryVisitor()) + .AddQueryVisitor(new EventSystemFieldCompatibilityQueryVisitor()) .UseFieldMap(new Dictionary { { Alias.BrowserVersion, EventIndexExtensions.DataDictionaryPath(Event.KnownDataKeys.RequestInfo, r => r.Data, RequestInfo.KnownDataKeys.BrowserVersion) }, { Alias.BrowserMajorVersion, EventIndexExtensions.DataDictionaryPath(Event.KnownDataKeys.RequestInfo, r => r.Data, RequestInfo.KnownDataKeys.BrowserMajorVersion) }, diff --git a/src/Exceptionless.Core/Repositories/EventRepository.cs b/src/Exceptionless.Core/Repositories/EventRepository.cs index 3fba37a69d..3c84537674 100644 --- a/src/Exceptionless.Core/Repositories/EventRepository.cs +++ b/src/Exceptionless.Core/Repositories/EventRepository.cs @@ -1,22 +1,32 @@ using Elastic.Clients.Elasticsearch.QueryDsl; using Exceptionless.Core.Models; using Exceptionless.Core.Repositories.Configuration; +using Exceptionless.Core.Repositories.Options; using Exceptionless.Core.Repositories.Queries; +using Exceptionless.Core.Services; using Exceptionless.Core.Validation; using Exceptionless.DateTimeExtensions; +using Foundatio.Parsers.LuceneQueries.Visitors; using Foundatio.Repositories; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Foundatio.Repositories.Elasticsearch.Extensions; using Foundatio.Repositories.Models; +using Foundatio.Repositories.Options; namespace Exceptionless.Core.Repositories; public class EventRepository : RepositoryOwnedByOrganizationAndProject, IEventRepository { private readonly TimeProvider _timeProvider; + private readonly IProjectRepository _projectRepository; + private readonly IStackRepository _stackRepository; - public EventRepository(ExceptionlessElasticConfiguration configuration, AppOptions options, MiniValidationValidator validator) + public EventRepository(ExceptionlessElasticConfiguration configuration, AppOptions options, MiniValidationValidator validator, IProjectRepository projectRepository, IStackRepository stackRepository) : base(configuration.Events, validator, options) { _timeProvider = configuration.TimeProvider; + _projectRepository = projectRepository; + _stackRepository = stackRepository; DisableCache(); // NOTE: If cache is ever enabled, then fast paths for patching/deleting with scripts will be super slow! BatchNotifications = true; @@ -35,7 +45,14 @@ public Task> GetOpenSessionsAsync(DateTime createdB { var query = new RepositoryQuery() .FieldEquals(e => e.Type, Event.KnownTypes.Session) - .ElasticFilter(new BoolQuery { MustNot = [new ExistsQuery { Field = $"idx.{Event.KnownDataKeys.SessionEnd}-d" }] }); + .ElasticFilter(new BoolQuery + { + MustNot = + [ + new ExistsQuery { Field = $"idx.{EventCustomFieldService.SessionEndIdxField}" }, + new ExistsQuery { Field = $"idx.{EventCustomFieldService.SessionEndField.LegacyIdxField}" } + ] + }); if (createdBeforeUtc.Ticks > 0) query = query.DateRange(null, createdBeforeUtc, (PersistentEvent e) => e.Date); // No lower bound, upper bound is exclusive @@ -52,7 +69,7 @@ public async Task UpdateSessionStartLastActivityAsync(string id, DateTime if (ev is null) return false; - if (!ev.UpdateSessionStart(lastActivityUtc, isSessionEnd)) + if (!ev.UpdateSessionStart(lastActivityUtc, isSessionEnd, hasError)) return false; await SaveAsync(ev, o => o.Notifications(sendNotifications)); @@ -65,7 +82,7 @@ public Task RemoveAllAsync(string organizationId, string? clientIpAddress, var query = new RepositoryQuery().Organization(organizationId); if (utcStart.HasValue && utcEnd.HasValue) - query = query.DateRange(utcStart, utcEnd, InferField(e => e.Date)).Index(utcStart, utcEnd); + query = query.DateRange(utcStart, utcEnd, InferField(e => e.Date)); else if (utcEnd.HasValue) query = query.DateRange(null, utcEnd, (PersistentEvent e) => e.Date); else if (utcStart.HasValue) @@ -196,4 +213,108 @@ public Task RemoveAllByStackIdsAsync(string[] stackIds) return RemoveAllAsync(q => q.Stack(stackIds)); } + + /// + /// Override to prevent the base from clearing idx (which would destroy slot values populated by EventCustomFieldService). + /// Custom field indexing is handled externally by EventCustomFieldService. + /// + protected override Task OnCustomFieldsDocumentsChanging(object sender, DocumentsChangeEventArgs args) + => Task.CompletedTask; + + /// + /// Resolve the tenant key from the query's organization filter. + /// + protected override string? GetTenantKey(IRepositoryQuery query) + { + var appFilter = query.GetAppFilter(); + if (appFilter?.Organizations.Count == 1) + return appFilter.Organizations.Single().Id; + + var organizationIds = query.GetOrganizations(); + return organizationIds.Count == 1 ? organizationIds.First() : null; + } + + /// + /// Custom field query resolution: resolves idx.fieldName and data.fieldName to idx.{type}-{slot}. + /// Blocks raw slot access (e.g., idx.keyword-7) to prevent querying deleted or other tenants' fields. + /// Returns null for non-idx/data fields so the global resolver (field aliases) still works. + /// + protected override async Task OnCustomFieldsBeforeQuery(object sender, BeforeQueryEventArgs args) + { + var tenantKey = await ResolveTenantKeyAsync(args.Query); + + var definitionRepo = ElasticIndex.Configuration.CustomFieldDefinitionRepository; + + // Lazy-load field mapping only when a query actually references idx.* or data.* fields. + // Most queries (count, date histograms, simple filters) never hit custom fields, + // so deferring this avoids a cache/ES lookup on every hot-path query. + Dictionary? mapping = null; + + args.Options.QueryFieldResolver(async (field, _) => + { + string? fieldName = null; + if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + fieldName = field.Substring(4); + else if (field.StartsWith("data.", StringComparison.OrdinalIgnoreCase)) + fieldName = field.Substring(5); + else if (field.StartsWith("ref.", StringComparison.OrdinalIgnoreCase)) + fieldName = $"@ref:{field.Substring(4)}"; + + if (fieldName is null) + return null; + + // System fields have deterministic slots that don't require tenant resolution. + if (EventCustomFieldService.TryGetSystemField(fieldName, out var systemField)) + return $"idx.{systemField.IdxField}"; + + if (EventCustomFieldService.SystemFields.Any(systemField => + String.Equals(systemField.LegacyIdxField, fieldName, StringComparison.OrdinalIgnoreCase))) + { + return $"idx.{fieldName}"; + } + + // Non-system fields require a tenant key to look up their slot assignment. + if (String.IsNullOrEmpty(tenantKey) || definitionRepo is null) + { + // Without tenant context, block raw idx access; data.* fields fall through. + return field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase) ? "idx.__blocked__" : null; + } + + mapping ??= (await definitionRepo.GetFieldMappingAsync(EntityTypeName, tenantKey)) + .ToDictionary(kvp => kvp.Key, kvp => kvp.Value.GetIdxName(), StringComparer.OrdinalIgnoreCase); + + if (mapping.TryGetValue(fieldName, out var idxName)) + return $"idx.{idxName}"; + + // Block raw slot access (e.g., idx.keyword-7) and unknown idx fields. + // Redirect to a non-existent field so the clause matches nothing. + if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + return "idx.__blocked__"; + + // For data.* and ref.* fields that don't map to a custom field, return null to let + // other resolvers handle legitimate data paths (e.g., data.@version). + return null; + }); + } + + private async Task ResolveTenantKeyAsync(IRepositoryQuery query) + { + var appFilter = query.GetAppFilter(); + if (appFilter?.Organizations.Count == 1) + return appFilter.Organizations.Single().Id; + + var organizationIds = query.GetOrganizations(); + if (organizationIds.Count == 1) + return organizationIds.Single(); + + var projectIds = query.GetProjects(); + if (projectIds.Count == 1) + return (await _projectRepository.GetByIdAsync(projectIds.Single()))?.OrganizationId; + + var stackIds = query.GetStacks(); + if (stackIds.Count == 1) + return (await _stackRepository.GetByIdAsync(stackIds.Single()))?.OrganizationId; + + return null; + } } diff --git a/src/Exceptionless.Core/Repositories/Queries/Validation/AppQueryValidator.cs b/src/Exceptionless.Core/Repositories/Queries/Validation/AppQueryValidator.cs index 00940d2762..01b7a96996 100644 --- a/src/Exceptionless.Core/Repositories/Queries/Validation/AppQueryValidator.cs +++ b/src/Exceptionless.Core/Repositories/Queries/Validation/AppQueryValidator.cs @@ -58,7 +58,9 @@ public async Task ValidateQueryAsync(string? query) public async Task ValidateQueryAsync(IQueryNode query) { var info = await ValidationVisitor.RunAsync(query); - return ApplyQueryRules(info); + var result = ApplyQueryRules(info); + result.ReferencedFields = info.ReferencedFields.ToArray(); + return result; } protected virtual QueryProcessResult ApplyQueryRules(QueryValidationResult result) @@ -95,7 +97,9 @@ public async Task ValidateAggregationsAsync(string? aggs) public async Task ValidateAggregationsAsync(IQueryNode query) { var info = await ValidationVisitor.RunAsync(query, new QueryVisitorContext()); - return ApplyAggregationRules(info); + var result = ApplyAggregationRules(info); + result.ReferencedFields = info.ReferencedFields.ToArray(); + return result; } protected virtual QueryProcessResult ApplyAggregationRules(QueryValidationResult result) @@ -108,5 +112,6 @@ public record QueryProcessResult public bool IsValid { get; init; } public string? Message { get; set; } public bool UsesPremiumFeatures { get; set; } + public IReadOnlyCollection ReferencedFields { get; set; } = []; } } diff --git a/src/Exceptionless.Core/Repositories/Queries/Validation/PersistentEventQueryValidator.cs b/src/Exceptionless.Core/Repositories/Queries/Validation/PersistentEventQueryValidator.cs index 5cd11109f5..0bc9d23bfc 100644 --- a/src/Exceptionless.Core/Repositories/Queries/Validation/PersistentEventQueryValidator.cs +++ b/src/Exceptionless.Core/Repositories/Queries/Validation/PersistentEventQueryValidator.cs @@ -117,7 +117,7 @@ protected override QueryProcessResult ApplyAggregationRules(QueryValidationResul return new QueryProcessResult { Message = "Aggregation count exceeded" }; // Only allow fields that are numeric or have high commonality. - if (!result.ReferencedFields.All(_allowedAggregationFields.Contains)) + if (!result.ReferencedFields.All(IsAllowedAggregationField)) return new QueryProcessResult { Message = "One or more aggregation fields are not allowed" }; // Distinct queries are expensive. @@ -135,4 +135,9 @@ protected override QueryProcessResult ApplyAggregationRules(QueryValidationResul UsesPremiumFeatures = usesPremiumFeatures }; } + + private static bool IsAllowedAggregationField(string field) + => _allowedAggregationFields.Contains(field) + || field.StartsWith("data.", StringComparison.OrdinalIgnoreCase) + || field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase); } diff --git a/src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs b/src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs deleted file mode 100644 index 43065cc189..0000000000 --- a/src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs +++ /dev/null @@ -1,135 +0,0 @@ -using Exceptionless.Core.Extensions; -using Exceptionless.Core.Models; -using Foundatio.Parsers.LuceneQueries.Nodes; -using Foundatio.Parsers.LuceneQueries.Visitors; - -namespace Exceptionless.Core.Repositories.Queries; - -public class EventFieldsQueryVisitor : ChainableQueryVisitor -{ - public override async Task VisitAsync(GroupNode node, IQueryVisitorContext context) - { - var childTerms = new List(); - if (node.Left is TermNode { Field: null, Term: not null } leftTermNode) - childTerms.Add(leftTermNode.Term); - - if (node.Left is TermRangeNode { Field: null } leftTermRangeNode) - { - if (leftTermRangeNode.Min is not null) - childTerms.Add(leftTermRangeNode.Min); - if (leftTermRangeNode.Max is not null) - childTerms.Add(leftTermRangeNode.Max); - } - - if (node.Right is TermNode { Field: null, Term: not null } rightTermNode) - childTerms.Add(rightTermNode.Term); - - if (node.Right is TermRangeNode { Field: null } rightTermRangeNode) - { - if (rightTermRangeNode.Min is not null) - childTerms.Add(rightTermRangeNode.Min); - if (rightTermRangeNode.Max is not null) - childTerms.Add(rightTermRangeNode.Max); - } - - node.Field = GetCustomFieldName(node.Field, childTerms.ToArray()) ?? node.Field; - foreach (var child in node.Children) - await child.AcceptAsync(this, context); - } - - public override Task VisitAsync(TermNode node, IQueryVisitorContext context) - { - // using all fields search - if (String.IsNullOrEmpty(node.Field)) - { - return Task.CompletedTask; - } - - node.Field = GetCustomFieldName(node.Field, [node.Term]); - return Task.CompletedTask; - } - - public override Task VisitAsync(TermRangeNode node, IQueryVisitorContext context) - { - node.Field = GetCustomFieldName(node.Field, [node.Min, node.Max]); - return Task.CompletedTask; - } - - public override Task VisitAsync(ExistsNode node, IQueryVisitorContext context) - { - node.Field = GetCustomFieldName(node.Field, []); - return Task.CompletedTask; - } - - public override Task VisitAsync(MissingNode node, IQueryVisitorContext context) - { - node.Field = GetCustomFieldName(node.Field, []); - return Task.CompletedTask; - } - - private string? GetCustomFieldName(string? field, string?[] terms) - { - if (String.IsNullOrEmpty(field)) - return null; - - string[] parts = field.Split('.'); - if (parts.Length != 2 || (parts.Length == 2 && parts[1].StartsWith("@"))) - return field; - - if (String.Equals(parts[0], "data", StringComparison.OrdinalIgnoreCase)) - { - string termType; - if (String.Equals(parts[1], Event.KnownDataKeys.SessionEnd, StringComparison.OrdinalIgnoreCase)) - termType = "d"; - else if (String.Equals(parts[1], Event.KnownDataKeys.SessionHasError, StringComparison.OrdinalIgnoreCase)) - termType = "b"; - else - termType = GetTermType(terms); - - field = $"idx.{parts[1].ToLowerInvariant()}-{termType}"; - } - else if (String.Equals(parts[0], "ref", StringComparison.OrdinalIgnoreCase)) - { - field = $"idx.{parts[1].ToLowerInvariant()}-r"; - } - - return field; - } - - private static string GetTermType(string?[] terms) - { - string termType = "s"; - - var trimmedTerms = terms.OfType().Distinct().ToList(); - foreach (string term in trimmedTerms) - { - if (term.StartsWith('*')) - continue; - - if (Boolean.TryParse(term, out bool _)) - termType = "b"; - else if (term.IsNumeric()) - termType = "n"; - else if (DateTime.TryParse(term, out DateTime _)) - termType = "d"; - - break; - } - - // Some terms can be a string date range: [now TO now/d+1d} - if (String.Equals(termType, "s") && trimmedTerms.Count > 0 && trimmedTerms.All(t => String.Equals(t, "now", StringComparison.OrdinalIgnoreCase) || t.StartsWith("now/", StringComparison.OrdinalIgnoreCase))) - termType = "d"; - - return termType; - } - - public static Task RunAsync(IQueryNode node, IQueryVisitorContext? context = null) - { - return new EventFieldsQueryVisitor().AcceptAsync(node, context ?? new QueryVisitorContext()); - } - - public static IQueryNode? Run(IQueryNode node, IQueryVisitorContext? context = null) - { - return RunAsync(node, context).GetAwaiter().GetResult(); - } -} diff --git a/src/Exceptionless.Core/Repositories/Queries/Visitors/EventSystemFieldCompatibilityQueryVisitor.cs b/src/Exceptionless.Core/Repositories/Queries/Visitors/EventSystemFieldCompatibilityQueryVisitor.cs new file mode 100644 index 0000000000..aee17c29f0 --- /dev/null +++ b/src/Exceptionless.Core/Repositories/Queries/Visitors/EventSystemFieldCompatibilityQueryVisitor.cs @@ -0,0 +1,80 @@ +using Exceptionless.Core.Services; +using Foundatio.Parsers.LuceneQueries.Extensions; +using Foundatio.Parsers.LuceneQueries.Nodes; +using Foundatio.Parsers.LuceneQueries.Visitors; + +namespace Exceptionless.Core.Repositories.Queries.Visitors; + +/// +/// Expands session system-field filters across the legacy named index field and the current pooled slot. +/// +public sealed class EventSystemFieldCompatibilityQueryVisitor : ChainableMutatingQueryVisitor +{ + public override Task VisitAsync(GroupNode node, IQueryVisitorContext context) + { + if (TryGetLegacyField(node.Field, out _)) + return Task.FromResult(Expand(node, GroupOperator.Or)); + + return base.VisitAsync(node, context); + } + + public override IQueryNode Visit(TermNode node, IQueryVisitorContext context) + => Expand(node, GroupOperator.Or); + + public override IQueryNode Visit(TermRangeNode node, IQueryVisitorContext context) + => Expand(node, GroupOperator.Or); + + public override IQueryNode Visit(ExistsNode node, IQueryVisitorContext context) + => Expand(node, GroupOperator.Or); + + public override IQueryNode Visit(MissingNode node, IQueryVisitorContext context) + => Expand(node, GroupOperator.And); + + private static IQueryNode Expand(IQueryNode node, GroupOperator compatibilityOperator) + { + if (node is not IFieldQueryNode fieldNode || !TryGetLegacyField(fieldNode.Field, out string legacyField)) + return node; + + var currentNode = node.Clone(); + var currentFieldNode = (IFieldQueryNode)currentNode; + currentFieldNode.IsNegated = null; + currentFieldNode.Prefix = null; + + var legacyNode = node.Clone(); + var legacyFieldNode = (IFieldQueryNode)legacyNode; + legacyFieldNode.Field = legacyField; + legacyFieldNode.IsNegated = null; + legacyFieldNode.Prefix = null; + + return node.ReplaceSelf(new GroupNode + { + HasParens = true, + IsNegated = fieldNode.IsNegated, + Prefix = fieldNode.Prefix, + Operator = compatibilityOperator, + Left = currentNode, + Right = legacyNode + }); + } + + private static bool TryGetLegacyField(string? field, out string legacyField) + { + legacyField = String.Empty; + if (String.IsNullOrWhiteSpace(field)) + return false; + + string? systemFieldName = null; + if (field.StartsWith("data.", StringComparison.OrdinalIgnoreCase)) + systemFieldName = field[5..]; + else if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + systemFieldName = field[4..]; + else if (field.StartsWith("ref.", StringComparison.OrdinalIgnoreCase)) + systemFieldName = $"@ref:{field[4..]}"; + + if (systemFieldName is null || !EventCustomFieldService.TryGetSystemField(systemFieldName, out var descriptor)) + return false; + + legacyField = $"idx.{descriptor.LegacyIdxField}"; + return true; + } +} diff --git a/src/Exceptionless.Core/Services/EventCustomFieldQueryPolicy.cs b/src/Exceptionless.Core/Services/EventCustomFieldQueryPolicy.cs new file mode 100644 index 0000000000..779d49cd1b --- /dev/null +++ b/src/Exceptionless.Core/Services/EventCustomFieldQueryPolicy.cs @@ -0,0 +1,109 @@ +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories.Queries; +using Foundatio.Repositories.Elasticsearch.CustomFields; + +namespace Exceptionless.Core.Services; + +/// +/// Validates logical event custom-field references before repository field resolution. +/// This prevents unknown fields, raw pooled slots, and ambiguous tenant scopes from +/// degrading into queries that silently return no results. +/// +public sealed class EventCustomFieldQueryPolicy(ICustomFieldDefinitionRepository customFieldDefinitionRepository) +{ + public const string UnknownFilterField = "unknown_filter_field"; + public const string CustomFieldScopeRequired = "custom_field_scope_required"; + + public async Task ValidateAsync( + IEnumerable referencedFields, + AppFilter? appFilter, + CancellationToken cancellationToken = default) + { + var customFields = referencedFields + .Select(TryGetLogicalCustomField) + .Where(field => field is not null) + .Select(field => field!) + .Distinct(StringComparer.OrdinalIgnoreCase) + .ToArray(); + + if (customFields.Length == 0) + return ValidationResult.Valid; + + if (appFilter?.Organizations.Count != 1) + { + return ValidationResult.Invalid( + CustomFieldScopeRequired, + "Custom-field searches must be scoped to exactly one organization."); + } + + string organizationId = appFilter.Organizations.Single().Id; + var mapping = await customFieldDefinitionRepository.GetFieldMappingAsync(nameof(PersistentEvent), organizationId); + var activeFieldNames = mapping.Keys.ToHashSet(StringComparer.OrdinalIgnoreCase); + + foreach (string field in customFields) + { + string logicalName = field[(field.IndexOf('.') + 1)..]; + // Definitions created before storage-slot names became reserved remain usable. + // Unmatched physical slots are still rejected below. + if (activeFieldNames.Contains(logicalName)) + continue; + + if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase) + && EventCustomFieldService.IsManagedCustomFieldSlotKey(logicalName)) + { + return ValidationResult.Invalid( + UnknownFilterField, + $"Raw custom-field slot '{field}' cannot be queried. Use the configured logical field name instead.", + field); + } + + if (!EventCustomFieldService.IsValidFieldName(logicalName) || !activeFieldNames.Contains(logicalName)) + { + return ValidationResult.Invalid( + UnknownFilterField, + $"Filter field '{field}' is not an active configured custom field for this organization.", + field); + } + } + + return ValidationResult.Valid; + } + + private static string? TryGetLogicalCustomField(string field) + { + if (String.IsNullOrWhiteSpace(field)) + return null; + + if (field.StartsWith("data.", StringComparison.OrdinalIgnoreCase)) + { + string logicalName = field["data.".Length..]; + if (logicalName.StartsWith('@') || EventCustomFieldService.IsSystemField(logicalName)) + return null; + + return field; + } + + if (field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + { + string logicalName = field["idx.".Length..]; + if (EventCustomFieldService.IsSystemField(logicalName) + || EventCustomFieldService.SystemFields.Any(systemField => + String.Equals(systemField.LegacyIdxField, logicalName, StringComparison.OrdinalIgnoreCase))) + { + return null; + } + + return field; + } + + return null; + } + + public sealed record ValidationResult(bool IsValid, string? ErrorCode = null, string? Message = null, string? Field = null) + { + public static ValidationResult Valid { get; } = new(true); + + public static ValidationResult Invalid(string errorCode, string message, string? field = null) + => new(false, errorCode, message, field); + } +} diff --git a/src/Exceptionless.Core/Services/EventCustomFieldService.cs b/src/Exceptionless.Core/Services/EventCustomFieldService.cs new file mode 100644 index 0000000000..dcac8212f3 --- /dev/null +++ b/src/Exceptionless.Core/Services/EventCustomFieldService.cs @@ -0,0 +1,579 @@ +using System.Globalization; +using Exceptionless; +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Lock; +using Foundatio.Repositories; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Foundatio.Repositories.Models; +using Foundatio.Repositories.Options; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Services; + +public class EventCustomFieldService : IStartupAction +{ + private readonly IEventRepository _eventRepository; + private readonly ICustomFieldDefinitionRepository _customFieldDefinitionRepository; + private readonly ILockProvider _lockProvider; + private readonly ILogger _logger; + + private const int MaxKeywordLength = 256; + + public const string SessionReferenceIdxField = "keyword-1"; + public const string SessionEndIdxField = "date-1"; + public const string SessionHasErrorIdxField = "bool-1"; + + /// + /// Canonical session field definitions shared by provisioning, indexing, and query compatibility. + /// + public static readonly SystemFieldDescriptor SessionReferenceField = + new("@ref:session", "keyword", SessionReferenceIdxField, "session-r"); + public static readonly SystemFieldDescriptor SessionEndField = + new(Event.KnownDataKeys.SessionEnd, "date", SessionEndIdxField, "sessionend-d"); + public static readonly SystemFieldDescriptor SessionHasErrorField = + new(Event.KnownDataKeys.SessionHasError, "bool", SessionHasErrorIdxField, "haserror-b"); + + public static readonly IReadOnlyList SystemFields = + [ + SessionReferenceField, + SessionEndField, + SessionHasErrorField + ]; + + public static string GetSavedViewConsistencyLockName(string organizationId) + => $"custom-field-saved-views:{organizationId}"; + + /// + /// The set of index types registered by AddStandardCustomFieldTypes() in EventIndex. + /// Only these types are supported for custom field definitions; any other type string would result + /// in an un-indexed, unqueryable field. + /// + public static readonly IReadOnlySet SupportedIndexTypes = new HashSet(StringComparer.OrdinalIgnoreCase) + { + "bool", "date", "double", "float", "int", "keyword", "long", "string" + }; + + public EventCustomFieldService( + IEventRepository eventRepository, + ICustomFieldDefinitionRepository customFieldDefinitionRepository, + ILockProvider lockProvider, + ILoggerFactory loggerFactory) + { + _eventRepository = eventRepository; + _customFieldDefinitionRepository = customFieldDefinitionRepository; + _lockProvider = lockProvider; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + _eventRepository.DocumentsChanging.AddHandler(OnDocumentsChangingAsync); + return Task.CompletedTask; + } + + /// + /// Ensures system fields exist for the given organization and occupy their reserved slots. + /// Invalid persisted state is rejected rather than silently creating definitions that queries cannot read. + /// + public Task EnsureSystemFieldsAsync(string organizationId) + => EnsureSystemFieldsAsync(organizationId, CancellationToken.None); + + private async Task EnsureSystemFieldsAsync(string organizationId, CancellationToken cancellationToken) + { + try + { + await EnsureSystemFieldsCoreAsync(organizationId, cancellationToken); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + AppDiagnostics.CustomFieldProvisioningFailures.Add(1); + throw; + } + } + + private async Task EnsureSystemFieldsCoreAsync(string organizationId, CancellationToken cancellationToken) + { + await using var provisioningLock = await _lockProvider.TryAcquireAsync( + $"custom-field-system:{organizationId}", + TimeSpan.FromSeconds(30), + TimeSpan.FromSeconds(5)); + if (provisioningLock is null) + throw new TimeoutException("System custom field provisioning is already in progress for this organization. Please try again."); + + var results = await _customFieldDefinitionRepository.FindAsync( + q => q + .FieldEquals(field => field.EntityType, nameof(PersistentEvent)) + .FieldEquals(field => field.TenantKey, organizationId), + o => o.IncludeSoftDeletes().SearchAfterPaging().PageLimit(1000)); + + var definitions = new List(); + do + { + definitions.AddRange(results.Documents); + } while (await results.NextPageAsync()); + + foreach (var systemField in SystemFields) + { + var namedDefinitions = definitions + .Where(definition => String.Equals(definition.Name, systemField.Name, StringComparison.OrdinalIgnoreCase)) + .ToList(); + + if (namedDefinitions.Count > 1) + throw CreateSystemFieldConflict(systemField, $"found {namedDefinitions.Count} definitions with the reserved name"); + + if (namedDefinitions.Count == 1) + { + ValidateSystemFieldDefinition(systemField, namedDefinitions[0], organizationId); + continue; + } + + var slotOccupant = definitions.FirstOrDefault(definition => + String.Equals(definition.IndexType, systemField.IndexType, StringComparison.Ordinal) + && String.Equals(definition.GetIdxName(), systemField.IdxField, StringComparison.Ordinal)); + if (slotOccupant is not null) + throw CreateSystemFieldConflict(systemField, $"reserved slot is occupied by '{slotOccupant.Name}'"); + + var definition = await _customFieldDefinitionRepository.AddFieldAsync( + nameof(PersistentEvent), organizationId, systemField.Name, systemField.IndexType, + description: $"System field: {systemField.Name}"); + ValidateSystemFieldDefinition(systemField, definition, organizationId); + definitions.Add(definition); + } + } + + /// + /// Returns true if the given field name is a system/reserved field that cannot be deleted. + /// + public static bool IsSystemField(string fieldName) + { + return TryGetSystemField(fieldName, out _); + } + + public static bool TryGetSystemField(string fieldName, out SystemFieldDescriptor descriptor) + { + descriptor = SystemFields.FirstOrDefault(field => String.Equals(field.Name, fieldName, StringComparison.OrdinalIgnoreCase))!; + return descriptor is not null; + } + + /// + /// Creates a new custom field definition under a distributed lock so concurrent requests + /// from the same organization cannot race past the quota check. + /// Returns a typed outcome so callers can distinguish duplicate names from capacity limits. + /// + public async Task CreateFieldAsync( + string organizationId, + string name, + string indexType, + int maxFieldsPerOrganization, + int maxLifetimeFieldsPerOrganization, + string? description = null, + int? displayOrder = null, + CancellationToken cancellationToken = default) + { + // Ensure system fields are provisioned before user-defined fields so they occupy slot 1 of their type. + await EnsureSystemFieldsAsync(organizationId, cancellationToken); + + await using var fieldLock = await _lockProvider.TryAcquireAsync( + $"custom-field-create:{organizationId}", TimeSpan.FromSeconds(30), TimeSpan.FromSeconds(5)); + if (fieldLock is null) + { + _logger.LogWarning("Could not acquire custom field creation lock for organization {OrganizationId}", organizationId); + throw new TimeoutException("Custom field creation is already in progress for this organization. Please try again."); + } + + // Re-read active and soft-deleted definitions inside the lock. Soft-deleted definitions + // still own their physical slot and therefore count against the lifetime mapping budget. + var existingPage = await _customFieldDefinitionRepository.FindAsync( + q => q + .FieldEquals(field => field.EntityType, nameof(PersistentEvent)) + .FieldEquals(field => field.TenantKey, organizationId), + o => o.IncludeSoftDeletes().SearchAfterPaging().PageLimit(1000)); + var allDefinitions = new List(); + do + { + allDefinitions.AddRange(existingPage.Documents); + } while (await existingPage.NextPageAsync()); + + var activeDefinitions = allDefinitions.Where(field => !field.IsDeleted).ToList(); + + if (activeDefinitions.Any(field => String.Equals(field.Name, name, StringComparison.OrdinalIgnoreCase))) + return new CreateFieldResult(CreateFieldStatus.Duplicate); + + // System fields are not counted against the user quota. + var userDefinedActiveCount = activeDefinitions.Count(field => !IsSystemField(field.Name)); + if (userDefinedActiveCount >= maxFieldsPerOrganization) + return new CreateFieldResult(CreateFieldStatus.ActiveLimitReached); + + var userDefinedLifetimeCount = allDefinitions.Count(field => !IsSystemField(field.Name)); + if (userDefinedLifetimeCount >= maxLifetimeFieldsPerOrganization) + { + AppDiagnostics.CustomFieldLifetimeLimitReached.Add(1); + return new CreateFieldResult(CreateFieldStatus.LifetimeLimitReached); + } + + var definition = await _customFieldDefinitionRepository.AddFieldAsync( + nameof(PersistentEvent), organizationId, name, indexType, description, displayOrder ?? 0); + return new CreateFieldResult(CreateFieldStatus.Created, definition); + } + + private async Task OnDocumentsChangingAsync(object sender, DocumentsChangeEventArgs args) + { + if (args.ChangeType == ChangeType.Removed) + return; + + if (args.Documents is null || args.Documents.Count == 0) + return; + + var documentsByOrganization = args.Documents + .Where(document => document.Value is not null) + .GroupBy(document => document.Value.OrganizationId) + .Where(g => !String.IsNullOrEmpty(g.Key)); + + foreach (var organizationGroup in documentsByOrganization) + { + IDictionary? fieldMapping = null; + try + { + fieldMapping = await _customFieldDefinitionRepository.GetFieldMappingAsync(nameof(PersistentEvent), organizationGroup.Key); + + // Lazily ensure all system fields are provisioned for this organization. + // Check each system field individually to handle partial-provisioning failures. + if (SystemFields.Any(field => !fieldMapping.ContainsKey(field.Name))) + { + await EnsureSystemFieldsAsync(organizationGroup.Key); + fieldMapping = await _customFieldDefinitionRepository.GetFieldMappingAsync(nameof(PersistentEvent), organizationGroup.Key); + } + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + AppDiagnostics.CustomFieldMappingFailures.Add(1); + _logger.LogError(ex, "Error loading custom field definitions for organization {OrganizationId}", organizationGroup.Key); + + if (args.ChangeType == ChangeType.Added) + { + foreach (var document in organizationGroup) + ClearCustomFieldSlots(document.Value); + + continue; + } + + // Never persist a saved event after stripping or partially rebuilding its slots. + // The caller can retry once the definition repository is available again. + throw; + } + + foreach (var document in organizationGroup) + { + try + { + document.Value.Idx = BuildCustomFieldSlots( + document.Value, + fieldMapping, + preserveUnmanagedSlots: args.ChangeType != ChangeType.Added); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + AppDiagnostics.CustomFieldProcessingFailures.Add(1); + _logger.LogError(ex, "Error processing custom fields for event {EventId}", document.Value.Id); + + if (args.ChangeType == ChangeType.Added) + { + ClearCustomFieldSlots(document.Value); + continue; + } + + throw; + } + } + } + } + + private DataDictionary? BuildCustomFieldSlots( + PersistentEvent ev, + IDictionary fieldMapping, + bool preserveUnmanagedSlots) + { + var idx = !preserveUnmanagedSlots || ev.Idx is null + ? new DataDictionary() + : new DataDictionary(ev.Idx.Where(field => !IsManagedCustomFieldSlotKey(field.Key))); + + if (fieldMapping.Count == 0 || ev.Data is null || ev.Data.Count == 0) + return idx.Count == 0 ? null : idx; + + // Iterate the field mapping (max ~20 entries) rather than all of ev.Data + // to avoid allocating an intermediate dictionary for events with large payloads. + // DataDictionary uses OrdinalIgnoreCase so the lookup is case-insensitive. + foreach (var (fieldName, definition) in fieldMapping) + { + if (definition.IsDeleted) + continue; + + if (!ev.Data.TryGetValue(fieldName, out var rawValue) || rawValue is null) + continue; + + // Only primitive types are indexable (mirrors GetCustomFields filtering). + if (rawValue is not (string or bool or int or long or float or double or decimal or DateTime or DateTimeOffset)) + continue; + + try + { + var value = ConvertValue(rawValue, definition.IndexType); + if (value is not null) + idx[definition.GetIdxName()] = value; + else + AppDiagnostics.CustomFieldConversionSkips.Add(1, new KeyValuePair("index_type", definition.IndexType)); + } + catch (Exception ex) when (ex is FormatException or InvalidCastException or OverflowException) + { + AppDiagnostics.CustomFieldConversionSkips.Add(1, new KeyValuePair("index_type", definition.IndexType)); + _logger.LogDebug(ex, "Skipping custom field {FieldName}: type mismatch for index type {IndexType}", fieldName, definition.IndexType); + } + } + + return idx.Count == 0 ? null : idx; + } + + private static void ClearCustomFieldSlots(PersistentEvent ev) + { + // Idx is server-managed. New events must never persist client-supplied pooled or + // legacy compatibility slots; saved legacy events preserve unmanaged slots above. + ev.Idx = null; + } + + public static bool IsManagedCustomFieldSlotKey(string idxKey) + { + if (String.IsNullOrWhiteSpace(idxKey)) + return false; + + int separatorIndex = idxKey.LastIndexOf('-'); + if (separatorIndex <= 0 || separatorIndex == idxKey.Length - 1) + return false; + + return SupportedIndexTypes.Contains(idxKey[..separatorIndex]) + && Int32.TryParse(idxKey.AsSpan(separatorIndex + 1), out _); + } + + /// + /// Strictly converts a value to the target index type. Returns null if conversion + /// is not possible (value is skipped rather than failing event ingestion). + /// + public static object? ConvertValue(object? value, string indexType) + { + if (value is null) + return null; + + return indexType switch + { + "keyword" => ConvertToKeyword(value), + "string" => ConvertToString(value), + "bool" => ConvertToBool(value), + "int" => ConvertToInt(value), + "long" => ConvertToLong(value), + "float" => ConvertToFloat(value), + "double" => ConvertToDouble(value), + "date" => ConvertToDate(value), + _ => null + }; + } + + private static object? ConvertToKeyword(object value) + { + string? str = FormatInvariant(value); + if (str is null || str.Length > MaxKeywordLength) + return null; + return str; + } + + private static object? ConvertToString(object value) + { + string? str = FormatInvariant(value); + if (str is null || str.Length > 8192) + return null; + return str; + } + + /// + /// Formats a primitive value to a culture-invariant string suitable for keyword/string ES fields. + /// Using without a format provider would produce locale-dependent + /// output for float/double/decimal (e.g., "1,5" on German servers) and non-ISO DateTime strings. + /// + private static string? FormatInvariant(object value) + { + return value switch + { + string s => s, + bool b => b.ToString(), // "True"/"False" + int i => i.ToString(CultureInfo.InvariantCulture), + long l => l.ToString(CultureInfo.InvariantCulture), + float f => f.ToString(CultureInfo.InvariantCulture), + double d => d.ToString(CultureInfo.InvariantCulture), + decimal m => m.ToString(CultureInfo.InvariantCulture), + DateTime dt when dt.Kind != DateTimeKind.Unspecified => dt.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture), + DateTime dt => DateTime.SpecifyKind(dt, DateTimeKind.Utc).ToString("O", CultureInfo.InvariantCulture), + DateTimeOffset dto => dto.UtcDateTime.ToString("O", CultureInfo.InvariantCulture), + _ => null + }; + } + + private static object? ConvertToBool(object value) + { + return value switch + { + bool b => b, + int i => i != 0, + long l => l != 0, + string s when s.Equals("true", StringComparison.OrdinalIgnoreCase) || s == "1" => (object)true, + string s when s.Equals("false", StringComparison.OrdinalIgnoreCase) || s == "0" => (object)false, + _ => null + }; + } + + private static object? ConvertToInt(object value) + { + return value switch + { + int i => i, + short s => (int)s, + byte b => (int)b, + sbyte sb => (int)sb, + long l when l is >= Int32.MinValue and <= Int32.MaxValue => (int)l, + double d when Double.IsFinite(d) && d is >= Int32.MinValue and <= Int32.MaxValue && Math.Truncate(d) == d => (int)d, + float f when Single.IsFinite(f) && f is >= Int32.MinValue and <= Int32.MaxValue && MathF.Truncate(f) == f => (int)f, + decimal m when m is >= Int32.MinValue and <= Int32.MaxValue && Decimal.Truncate(m) == m => (int)m, + string s when Int32.TryParse(s, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToLong(object value) + { + return value switch + { + long l => l, + int i => (long)i, + short s => (long)s, + byte b => (long)b, + sbyte sb => (long)sb, + double d when Double.IsFinite(d) && d >= Int64.MinValue && d < Int64.MaxValue && Math.Truncate(d) == d => (long)d, + float f when Single.IsFinite(f) && f >= Int64.MinValue && f < Int64.MaxValue && MathF.Truncate(f) == f => (long)f, + decimal m when m is >= Int64.MinValue and <= Int64.MaxValue && Decimal.Truncate(m) == m => (long)m, + string s when Int64.TryParse(s, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToFloat(object value) + { + return value switch + { + float f when Single.IsFinite(f) => f, + int i => (float)i, + long l => (float)l, + double d when Double.IsFinite(d) && d is >= Single.MinValue and <= Single.MaxValue => (float)d, + decimal m => (float)m, + string s when Single.TryParse(s, NumberStyles.Float, CultureInfo.InvariantCulture, out var parsed) && Single.IsFinite(parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToDouble(object value) + { + return value switch + { + double d when Double.IsFinite(d) => d, + float f when Single.IsFinite(f) => (double)f, + int i => (double)i, + long l => (double)l, + decimal m => (double)m, + string s when Double.TryParse(s, NumberStyles.Float, CultureInfo.InvariantCulture, out var parsed) && Double.IsFinite(parsed) => parsed, + _ => null + }; + } + + private static object? ConvertToDate(object value) + { + return value switch + { + DateTime dt when dt.Kind != DateTimeKind.Unspecified => dt.ToUniversalTime(), + DateTime dt => DateTime.SpecifyKind(dt, DateTimeKind.Utc), + DateTimeOffset dto => dto.UtcDateTime, + // AssumeUniversal treats strings without explicit timezone info as UTC, avoiding + // silent server-local-time interpretation. Strings with explicit offsets use those offsets. + string s when DateTimeOffset.TryParse(s, CultureInfo.InvariantCulture, + DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed) => parsed.UtcDateTime, + _ => null + }; + } + + /// + /// Validates that a custom field name meets requirements: + /// - Not empty, max 100 chars + /// - Any name starting with '@' is reserved + /// - Only ASCII letters, digits, underscore, dot, dash allowed (no Unicode) + /// + public static bool IsValidFieldName(string name) + { + if (String.IsNullOrWhiteSpace(name)) + return false; + + if (name.Length > 100) + return false; + + // Any @-prefixed name is reserved + if (name.StartsWith('@')) + return false; + + // Physical pooled slots and legacy compatibility slots are implementation details. + // Allowing a logical definition with one of these names would make idx. + // ambiguous or unqueryable. + if (IsManagedCustomFieldSlotKey(name) + || SystemFields.Any(field => String.Equals(field.LegacyIdxField, name, StringComparison.OrdinalIgnoreCase))) + { + return false; + } + + // Only ASCII alphanumeric, underscore, dot, and dash — no Unicode identifiers + return name.All(c => Char.IsAsciiLetterOrDigit(c) || c == '_' || c == '.' || c == '-'); + } + + private static void ValidateSystemFieldDefinition(SystemFieldDescriptor systemField, CustomFieldDefinition definition, string organizationId) + { + if (!String.Equals(definition.Name, systemField.Name, StringComparison.Ordinal)) + throw CreateSystemFieldConflict(systemField, $"name is '{definition.Name}'"); + + if (!String.Equals(definition.EntityType, nameof(PersistentEvent), StringComparison.Ordinal)) + throw CreateSystemFieldConflict(systemField, $"entity type is '{definition.EntityType}'"); + + if (!String.Equals(definition.TenantKey, organizationId, StringComparison.Ordinal)) + throw CreateSystemFieldConflict(systemField, $"tenant is '{definition.TenantKey}'"); + + if (!String.Equals(definition.IndexType, systemField.IndexType, StringComparison.Ordinal)) + throw CreateSystemFieldConflict(systemField, $"index type is '{definition.IndexType}'"); + + if (!String.Equals(definition.GetIdxName(), systemField.IdxField, StringComparison.Ordinal)) + throw CreateSystemFieldConflict(systemField, $"slot is '{definition.GetIdxName()}'"); + + if (definition.IsDeleted) + throw CreateSystemFieldConflict(systemField, "definition is soft-deleted"); + } + + private static InvalidOperationException CreateSystemFieldConflict(SystemFieldDescriptor systemField, string reason) + { + return new InvalidOperationException( + $"System custom field '{systemField.Name}' must be an active {systemField.IndexType} field at idx.{systemField.IdxField}, but {reason}."); + } + + public sealed record SystemFieldDescriptor(string Name, string IndexType, string IdxField, string LegacyIdxField); + + public enum CreateFieldStatus + { + Created, + Duplicate, + ActiveLimitReached, + LifetimeLimitReached + } + + public sealed record CreateFieldResult(CreateFieldStatus Status, CustomFieldDefinition? Definition = null); +} diff --git a/src/Exceptionless.Core/Services/OrganizationService.cs b/src/Exceptionless.Core/Services/OrganizationService.cs index 5413a5cfe7..f563777a18 100644 --- a/src/Exceptionless.Core/Services/OrganizationService.cs +++ b/src/Exceptionless.Core/Services/OrganizationService.cs @@ -3,7 +3,9 @@ using Exceptionless.Core.Repositories; using Foundatio.Extensions.Hosting.Startup; using Foundatio.Repositories; +using Foundatio.Repositories.Elasticsearch.CustomFields; using Foundatio.Repositories.Models; +using Foundatio.Repositories.Options; using Microsoft.Extensions.Logging; using Stripe; @@ -14,6 +16,7 @@ public class OrganizationService : IStartupAction private const int BATCH_SIZE = 50; private readonly IOrganizationRepository _organizationRepository; private readonly IProjectRepository _projectRepository; + private readonly ICustomFieldDefinitionRepository _customFieldDefinitionRepository; private readonly ISavedViewRepository _savedViewRepository; private readonly ITokenRepository _tokenRepository; private readonly IUserRepository _userRepository; @@ -22,10 +25,11 @@ public class OrganizationService : IStartupAction private readonly UsageService _usageService; private readonly ILogger _logger; - public OrganizationService(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, ISavedViewRepository savedViewRepository, ITokenRepository tokenRepository, IUserRepository userRepository, IWebHookRepository webHookRepository, IStripeBillingClient stripeBillingClient, UsageService usageService, ILoggerFactory loggerFactory) + public OrganizationService(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, ICustomFieldDefinitionRepository customFieldDefinitionRepository, ISavedViewRepository savedViewRepository, ITokenRepository tokenRepository, IUserRepository userRepository, IWebHookRepository webHookRepository, IStripeBillingClient stripeBillingClient, UsageService usageService, ILoggerFactory loggerFactory) { _organizationRepository = organizationRepository; _projectRepository = projectRepository; + _customFieldDefinitionRepository = customFieldDefinitionRepository; _savedViewRepository = savedViewRepository; _tokenRepository = tokenRepository; _userRepository = userRepository; @@ -187,6 +191,15 @@ public Task RemoveSavedViewsAsync(Organization organization) return _savedViewRepository.RemoveAllByOrganizationIdAsync(organization.Id); } + public Task RemoveCustomFieldDefinitionsAsync(Organization organization) + { + _logger.LogDebug("Removing custom field definitions for {OrganizationName} ({OrganizationId})", organization.Name, organization.Id); + return _customFieldDefinitionRepository.RemoveAllAsync(q => q + .FieldEquals(field => field.EntityType, nameof(PersistentEvent)) + .FieldEquals(field => field.TenantKey, organization.Id), + o => o.SoftDeleteMode(SoftDeleteQueryMode.All)); + } + public Task RemoveUserSavedViewsAsync(string organizationId, string userId) { _logger.LogDebug("Removing private saved views for user {UserId} in organization {OrganizationId}", userId, organizationId); @@ -201,6 +214,7 @@ public async Task SoftDeleteOrganizationAsync(Organization organization, string await RemoveTokensAsync(organization); await RemoveWebHooksAsync(organization); await RemoveSavedViewsAsync(organization); + await RemoveCustomFieldDefinitionsAsync(organization); await CancelSubscriptionsAsync(organization); await RemoveUsersAsync(organization, currentUserId); await CleanupProjectNotificationSettingsAsync(organization, []); diff --git a/src/Exceptionless.Core/Utility/AppDiagnostics.cs b/src/Exceptionless.Core/Utility/AppDiagnostics.cs index c081925b6c..2da22d5fc2 100644 --- a/src/Exceptionless.Core/Utility/AppDiagnostics.cs +++ b/src/Exceptionless.Core/Utility/AppDiagnostics.cs @@ -100,6 +100,13 @@ public GaugeInfo(Meter meter, string name) internal static readonly Counter EventsDeleted = Meter.CreateCounter("ex.events.deleted", description: "Events that were deleted"); internal static readonly Counter EventsRetryCount = Meter.CreateCounter("ex.events.retry.count", description: "Events where processing was retried"); internal static readonly Counter EventsRetryErrors = Meter.CreateCounter("ex.events.retry.errors", description: "Events where retry processing got an error"); + + internal static readonly Counter CustomFieldMappingFailures = Meter.CreateCounter("ex.custom_fields.mapping.failures", description: "Custom field definition mapping load failures"); + internal static readonly Counter CustomFieldProvisioningFailures = Meter.CreateCounter("ex.custom_fields.provisioning.failures", description: "Custom field system-definition provisioning failures"); + internal static readonly Counter CustomFieldProcessingFailures = Meter.CreateCounter("ex.custom_fields.processing.failures", description: "Unexpected custom field processing failures"); + internal static readonly Counter CustomFieldConversionSkips = Meter.CreateCounter("ex.custom_fields.conversion.skips", description: "Custom field values skipped because they do not match the configured index type"); + internal static readonly Counter CustomFieldLifetimeLimitReached = Meter.CreateCounter("ex.custom_fields.lifetime_limit.reached", description: "Custom field creation attempts rejected by the lifetime slot limit"); + internal static readonly Histogram CustomFieldMappedFieldCount = Meter.CreateHistogram("ex.custom_fields.mapping.field_count", description: "Total mapped fields observed while validating a retained event index"); internal static readonly Histogram EventsFieldCount = Meter.CreateHistogram("ex.events.field.count", description: "Number of fields per event"); internal static readonly Counter PostsParsed = Meter.CreateCounter("ex.posts.parsed", description: "Post batch submission parsed"); diff --git a/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs b/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs index 5cd02efbe3..3f7943b749 100644 --- a/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs +++ b/src/Exceptionless.Web/Api/Endpoints/OrganizationEndpoints.cs @@ -85,6 +85,48 @@ public static IEndpointRouteBuilder MapOrganizationEndpoints(this IEndpointRoute } }); + group.MapGet("organizations/{id:objectid}/event-custom-fields", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper) + => (await mediator.InvokeAsync>>(new OrganizationMessages.GetEventCustomFields(id, httpContext))).ToHttpResult(resultMapper)) + .Produces>() + .ProducesProblem(StatusCodes.Status404NotFound) + .WithSummary("Get event custom fields"); + + group.MapPost("organizations/{id:objectid}/event-custom-fields", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] NewCustomFieldDefinition field) + => (await mediator.InvokeAsync>(new OrganizationMessages.CreateEventCustomField(id, field, httpContext))).ToHttpResult(resultMapper)) + .Accepts("application/json", "application/*+json") + .Produces(StatusCodes.Status201Created) + .ProducesProblem(StatusCodes.Status400BadRequest) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status409Conflict) + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) + .ProducesProblem(StatusCodes.Status426UpgradeRequired) + .WithSummary("Create event custom field"); + + group.MapPatch("organizations/{id:objectid}/event-custom-fields/{fieldId:objectid}", async (string id, string fieldId, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => + { + if (changes is null) + return ApiValidation.MissingRequestBody(); + + return (await mediator.InvokeAsync>(new OrganizationMessages.UpdateEventCustomField(id, fieldId, changes, httpContext))).ToHttpResult(resultMapper); + }) + .Accepts>("application/json", "application/*+json") + .Produces() + .ProducesProblem(StatusCodes.Status400BadRequest) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status409Conflict) + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) + .ProducesProblem(StatusCodes.Status426UpgradeRequired) + .WithSummary("Update event custom field"); + + group.MapDelete("organizations/{id:objectid}/event-custom-fields/{fieldId:objectid}", async (string id, string fieldId, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper) + => (await mediator.InvokeAsync(new OrganizationMessages.DeleteEventCustomField(id, fieldId, httpContext))).ToHttpResult(resultMapper)) + .Produces(StatusCodes.Status204NoContent) + .ProducesProblem(StatusCodes.Status400BadRequest) + .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status409Conflict) + .ProducesProblem(StatusCodes.Status426UpgradeRequired) + .WithSummary("Delete event custom field"); + group.MapPatch("organizations/{id:objectid}", async (string id, HttpContext httpContext, IMediator mediator, IMediatorResultMapper resultMapper, [FromBody] Delta? changes) => { if (changes is null) diff --git a/src/Exceptionless.Web/Api/Endpoints/SavedViewEndpoints.cs b/src/Exceptionless.Web/Api/Endpoints/SavedViewEndpoints.cs index a16f644e5a..39eedcaa8f 100644 --- a/src/Exceptionless.Web/Api/Endpoints/SavedViewEndpoints.cs +++ b/src/Exceptionless.Web/Api/Endpoints/SavedViewEndpoints.cs @@ -141,11 +141,13 @@ public static IEndpointRouteBuilder MapSavedViewEndpoints(this IEndpointRouteBui .Accepts>("application/json", "application/*+json", "application/octet-stream", "text/json", "text/plain") .RequireAuthorization(AuthorizationRoles.GlobalAdminPolicy) .Produces>() + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) .WithSummary("Replace all predefined saved views with the provided definitions") .WithMetadata(new EndpointDocumentation { RequestBodyDescription = "The full set of predefined saved view definitions.", ResponseDescriptions = new() { ["200"] = "The predefined saved views were replaced.", + ["422"] = "A definition is invalid or references an organization-specific custom field.", } }); @@ -167,6 +169,7 @@ public static IEndpointRouteBuilder MapSavedViewEndpoints(this IEndpointRouteBui .RequireAuthorization(AuthorizationRoles.GlobalAdminPolicy) .Produces() .ProducesProblem(StatusCodes.Status404NotFound) + .ProducesProblem(StatusCodes.Status422UnprocessableEntity) .WithSummary("Save a saved view as a global predefined saved view") .WithMetadata(new EndpointDocumentation { ParameterDescriptions = new() { @@ -175,6 +178,7 @@ public static IEndpointRouteBuilder MapSavedViewEndpoints(this IEndpointRouteBui ResponseDescriptions = new() { ["200"] = "The predefined saved view was created or updated.", ["404"] = "The saved view could not be found.", + ["422"] = "The saved view references an organization-specific custom field.", } }); diff --git a/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs b/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs index bb05f1f940..ef7b09641e 100644 --- a/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/AdminHandler.cs @@ -184,6 +184,9 @@ public async Task Handle(AdminRunMaintenance message) { switch (message.Name.ToLowerInvariant()) { + case "ensure-system-custom-fields": + await workItemQueue.EnqueueAsync(new OrganizationMaintenanceWorkItem { EnsureSystemCustomFields = true }); + break; case "fix-stack-stats": var effectiveUtcStart = message.UtcStart ?? timeProvider.GetUtcNow().UtcDateTime.AddDays(-90); diff --git a/src/Exceptionless.Web/Api/Handlers/EventCustomFieldHandler.cs b/src/Exceptionless.Web/Api/Handlers/EventCustomFieldHandler.cs new file mode 100644 index 0000000000..db2d2eeb4a --- /dev/null +++ b/src/Exceptionless.Web/Api/Handlers/EventCustomFieldHandler.cs @@ -0,0 +1,221 @@ +using System.ComponentModel.DataAnnotations; +using Exceptionless.Core; +using Exceptionless.Core.Extensions; +using Exceptionless.Core.Models; +using Exceptionless.Core.Repositories; +using Exceptionless.Core.Queries.Validation; +using Exceptionless.Core.Services; +using Exceptionless.Web.Api.Messages; +using Exceptionless.Web.Api.Results; +using Exceptionless.Web.Extensions; +using Exceptionless.Web.Models; +using Foundatio.Lock; +using Foundatio.Mediator; +using Foundatio.Repositories; +using Foundatio.Repositories.Elasticsearch.CustomFields; +using Foundatio.Repositories.Exceptions; +using Foundatio.Repositories.Models; + +namespace Exceptionless.Web.Api.Handlers; + +public sealed class EventCustomFieldHandler( + EventCustomFieldService eventCustomFieldService, + IOrganizationRepository organizationRepository, + ICustomFieldDefinitionRepository customFieldDefinitionRepository, + ISavedViewRepository savedViewRepository, + ILockProvider lockProvider, + PersistentEventQueryValidator eventQueryValidator, + EventStackQueryValidator eventStackQueryValidator, + AppOptions options) +{ + public async Task>> Handle(GetEventCustomFields message) + { + var organization = await GetOrganizationAsync(message.Id, message.Context); + if (organization is null) + return Result.NotFound("Organization not found."); + + var results = await customFieldDefinitionRepository.FindByTenantAsync(nameof(PersistentEvent), message.Id); + var fields = new List(); + do + { + fields.AddRange(results.Documents + .Where(field => !EventCustomFieldService.IsSystemField(field.Name)) + .Select(CustomFieldDefinitionResponse.FromDefinition)); + } while (await results.NextPageAsync()); + + return fields + .OrderBy(field => field.DisplayOrder) + .ThenBy(field => field.Name, StringComparer.OrdinalIgnoreCase) + .ToArray(); + } + + public async Task> Handle(CreateEventCustomField message) + { + var organization = await GetOrganizationAsync(message.Id, message.Context); + if (organization is null) + return Result.NotFound("Organization not found."); + + if (!organization.HasPremiumFeatures) + return Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, "Custom fields require a paid plan. Please upgrade to add custom fields.")); + + if (EventCustomFieldService.IsSystemField(message.Field.Name)) + return Result.BadRequest($"'{message.Field.Name}' is a reserved system field and cannot be created manually."); + + EventCustomFieldService.CreateFieldResult createResult; + try + { + createResult = await eventCustomFieldService.CreateFieldAsync( + message.Id, + message.Field.Name, + message.Field.IndexType.ToLowerInvariant(), + options.CustomFieldOptions.MaxFieldsPerOrganization, + options.CustomFieldOptions.MaxLifetimeFieldsPerOrganization, + message.Field.Description, + message.Field.DisplayOrder, + message.Context.RequestAborted); + } + catch (TimeoutException ex) + { + return Result.Conflict(ex.Message); + } + catch (Exception ex) when (ex is ArgumentException or ValidationException or InvalidOperationException or DocumentValidationException) + { + return Result.Invalid(ValidationError.Create("general", ex.Message)); + } + + if (createResult.Status == EventCustomFieldService.CreateFieldStatus.Duplicate) + return Result.Conflict($"A custom field named '{message.Field.Name}' already exists for this organization."); + + if (createResult.Status == EventCustomFieldService.CreateFieldStatus.ActiveLimitReached) + { + return Result.Invalid(ValidationError.Create( + ApiValidationErrorIdentifiers.CustomFieldActiveLimit, + $"Maximum of {options.CustomFieldOptions.MaxFieldsPerOrganization} active custom fields per organization has been reached.")); + } + + if (createResult.Status == EventCustomFieldService.CreateFieldStatus.LifetimeLimitReached) + { + return Result.Invalid(ValidationError.Create( + ApiValidationErrorIdentifiers.CustomFieldLifetimeLimit, + $"Maximum lifetime allocation of {options.CustomFieldOptions.MaxLifetimeFieldsPerOrganization} custom field slots per organization has been reached.")); + } + + var response = CustomFieldDefinitionResponse.FromDefinition(createResult.Definition!); + return Result.Created(response, $"/api/v2/organizations/{message.Id}/event-custom-fields"); + } + + public async Task> Handle(UpdateEventCustomField message) + { + var organization = await GetOrganizationAsync(message.Id, message.Context); + if (organization is null) + return Result.NotFound("Organization not found."); + + if (!organization.HasPremiumFeatures) + return Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, "Custom fields require a paid plan. Please upgrade to manage custom fields.")); + + await using var consistencyLock = await TryAcquireConsistencyLockAsync(message.Id); + if (consistencyLock is null) + return Result.Conflict("Custom field or saved-view changes are already in progress for this organization. Please try again."); + + var definition = await GetDefinitionAsync(message.Id, message.FieldId); + if (definition is null) + return Result.NotFound("Custom field not found."); + + if (EventCustomFieldService.IsSystemField(definition.Name)) + return Result.BadRequest($"'{definition.Name}' is a reserved system field and cannot be modified."); + + var changes = message.Changes.GetEntity(); + if (message.Changes.ContainsChangedProperty(field => field.Description!) && changes.Description?.Length > UpdateCustomFieldDefinition.MaxDescriptionLength) + return Result.Invalid(ValidationError.Create("description", $"Description cannot exceed {UpdateCustomFieldDefinition.MaxDescriptionLength} characters.")); + + bool changed = false; + if (message.Changes.ContainsChangedProperty(field => field.Description!)) + { + definition.Description = String.IsNullOrEmpty(changes.Description) ? null : changes.Description; + changed = true; + } + + if (message.Changes.ContainsChangedProperty(field => field.DisplayOrder!) && changes.DisplayOrder.HasValue) + { + definition.DisplayOrder = changes.DisplayOrder.Value; + changed = true; + } + + if (changed) + await customFieldDefinitionRepository.SaveAsync(definition); + + return CustomFieldDefinitionResponse.FromDefinition(definition); + } + + public async Task Handle(DeleteEventCustomField message) + { + var organization = await GetOrganizationAsync(message.Id, message.Context); + if (organization is null) + return Result.NotFound("Organization not found."); + + if (!organization.HasPremiumFeatures) + return Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, "Custom fields require a paid plan. Please upgrade to manage custom fields.")); + + await using var consistencyLock = await TryAcquireConsistencyLockAsync(message.Id); + if (consistencyLock is null) + return Result.Conflict("Custom field or saved-view changes are already in progress for this organization. Please try again."); + + var definition = await GetDefinitionAsync(message.Id, message.FieldId); + if (definition is null) + return Result.NotFound("Custom field not found."); + + if (EventCustomFieldService.IsSystemField(definition.Name)) + return Result.BadRequest($"'{definition.Name}' is a reserved system field and cannot be deleted."); + + var savedViews = await savedViewRepository.GetByOrganizationIdAsync(message.Id, o => o.SearchAfterPaging().PageLimit(1000)); + do + { + foreach (var savedView in savedViews.Documents) + { + var queryValidator = String.Equals(savedView.ViewType, "stacks", StringComparison.OrdinalIgnoreCase) + ? (AppQueryValidator)eventStackQueryValidator + : eventQueryValidator; + var validation = await queryValidator.ValidateQueryAsync(savedView.Filter); + if (!validation.IsValid) + continue; + + bool isReferenced = validation.ReferencedFields.Any(field => + String.Equals(field, $"data.{definition.Name}", StringComparison.OrdinalIgnoreCase) + || String.Equals(field, $"idx.{definition.Name}", StringComparison.OrdinalIgnoreCase)); + if (isReferenced) + return Result.Conflict($"Custom field '{definition.Name}' is used in one or more saved filters and cannot be deleted. Remove it from all filters first."); + } + } while (await savedViews.NextPageAsync()); + + definition.IsDeleted = true; + await customFieldDefinitionRepository.SaveAsync(definition); + + return Result.NoContent(); + } + + private async Task GetOrganizationAsync(string organizationId, HttpContext httpContext) + { + if (String.IsNullOrEmpty(organizationId) || !httpContext.Request.CanAccessOrganization(organizationId)) + return null; + + return await organizationRepository.GetByIdAsync(organizationId, o => o.Cache(false)); + } + + private async Task GetDefinitionAsync(string organizationId, string fieldId) + { + var definition = await customFieldDefinitionRepository.GetByIdAsync(fieldId); + return definition is null + || definition.IsDeleted + || !String.Equals(definition.TenantKey, organizationId, StringComparison.Ordinal) + || !String.Equals(definition.EntityType, nameof(PersistentEvent), StringComparison.Ordinal) + ? null + : definition; + } + + private Task TryAcquireConsistencyLockAsync(string organizationId) + => lockProvider.TryAcquireAsync( + EventCustomFieldService.GetSavedViewConsistencyLockName(organizationId), + TimeSpan.FromMinutes(5), + TimeSpan.FromSeconds(5)); + +} diff --git a/src/Exceptionless.Web/Api/Handlers/EventHandler.cs b/src/Exceptionless.Web/Api/Handlers/EventHandler.cs index c1bdb6d5c1..dd2f760f13 100644 --- a/src/Exceptionless.Web/Api/Handlers/EventHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/EventHandler.cs @@ -46,6 +46,7 @@ public class EventHandler( ITextSerializer serializer, PersistentEventQueryValidator validator, EventStackQueryValidator stackModeValidator, + EventCustomFieldQueryPolicy eventCustomFieldQueryPolicy, AppOptions appOptions, UsageService usageService, TimeProvider timeProvider, @@ -57,6 +58,8 @@ public class EventHandler( private static readonly ICollection _allowedDateFields = new List { EventIndex.Alias.Date }; private const string DefaultDateField = EventIndex.Alias.Date; private static Result PlanLimitResult(string message) => Result.Invalid(ValidationError.Create(ApiValidationErrorIdentifiers.PlanLimit, message)); + private static Result CustomFieldValidationResult(EventCustomFieldQueryPolicy.ValidationResult validation) + => Result.Invalid(ValidationError.Create(validation.ErrorCode!, validation.Message!)); private static bool ShouldIncludeTotal(string? include) => ShouldInclude(include, "total"); private static bool ShouldInclude(string? include, string value) @@ -698,6 +701,11 @@ private async Task> CountInternalAsync(AppFilter sf, TimeInf if (!far.IsValid) return Result.BadRequest(far.Message ?? "Invalid aggregations."); + var customFieldValidation = await eventCustomFieldQueryPolicy.ValidateAsync( + pr.ReferencedFields.Concat(far.ReferencedFields), sf, httpContext.RequestAborted); + if (!customFieldValidation.IsValid) + return CustomFieldValidationResult(customFieldValidation); + sf.UsesPremiumFeatures = pr.UsesPremiumFeatures || far.UsesPremiumFeatures; AppFilter? systemFilter = ApiFilterPolicy.ShouldApplySystemFilter(sf, filter, httpContext.Request) ? sf : null; if (systemFilter is not null && ApiFilterPolicy.IsPremiumFeatureQueryBlocked(systemFilter)) @@ -759,6 +767,10 @@ private async Task>> GetInternalAsync(AppFilter sf, T if (!pr.IsValid) return Result.BadRequest(pr.Message ?? "Invalid filter."); + var customFieldValidation = await eventCustomFieldQueryPolicy.ValidateAsync(pr.ReferencedFields, sf, httpContext.RequestAborted); + if (!customFieldValidation.IsValid) + return CustomFieldValidationResult>(customFieldValidation); + sf.UsesPremiumFeatures = pr.UsesPremiumFeatures || premiumFeatureUpgradeMessage is not null; AppFilter? appliedAppFilter = ApiFilterPolicy.ShouldApplySystemFilter(sf, filter, httpContext.Request) ? sf : null; if (appliedAppFilter is not null && ApiFilterPolicy.IsPremiumFeatureQueryBlocked(appliedAppFilter)) diff --git a/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs b/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs index a64badd407..d9480c0fd1 100644 --- a/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/OrganizationHandler.cs @@ -30,6 +30,7 @@ namespace Exceptionless.Web.Api.Handlers; public class OrganizationHandler( OrganizationService organizationService, + EventCustomFieldService eventCustomFieldService, IOrganizationRepository repository, ICacheClient cacheClient, IEventRepository eventRepository, @@ -835,6 +836,17 @@ private async Task AddModelAsync(Organization value, HttpContext h var organization = await repository.AddAsync(value, o => o.Cache()); + try + { + await eventCustomFieldService.EnsureSystemFieldsAsync(organization.Id); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + // Custom fields are also provisioned lazily during event processing and before user-field creation. + // Do not leave a newly persisted organization inaccessible to its creator if this best-effort step fails. + _logger.LogError(ex, "Error provisioning system custom fields for new organization {OrganizationId}", organization.Id); + } + user.OrganizationIds.Add(organization.Id); await userRepository.SaveAsync(user, o => o.Cache()); await messagePublisher.PublishAsync(new UserMembershipChanged @@ -1035,4 +1047,5 @@ private static Result PermissionToResult(PermissionResult permission) private static User GetCurrentUser(HttpContext httpContext) => httpContext.Request.GetUser(); private static bool IsStatsMode(string? mode) => !String.IsNullOrEmpty(mode) && String.Equals(mode, "stats", StringComparison.OrdinalIgnoreCase); private static bool messageIsGlobalAdmin(HttpContext httpContext) => httpContext.Request.IsGlobalAdmin(); + } diff --git a/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs b/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs index 3d2cf82ae3..69c240eadf 100644 --- a/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs +++ b/src/Exceptionless.Web/Api/Handlers/SavedViewHandler.cs @@ -6,7 +6,10 @@ using Exceptionless.Core.Models; using Exceptionless.Core.Models.WorkItems; using Exceptionless.Core.Repositories; +using Exceptionless.Core.Repositories.Queries; +using Exceptionless.Core.Queries.Validation; using Exceptionless.Core.Seed; +using Exceptionless.Core.Services; using Exceptionless.Web.Api.Infrastructure; using Exceptionless.Web.Api.Messages; using Exceptionless.Web.Api.Results; @@ -28,6 +31,9 @@ public partial class SavedViewHandler( IOrganizationRepository organizationRepository, ILockProvider lockProvider, IQueue workItemQueue, + PersistentEventQueryValidator eventQueryValidator, + EventStackQueryValidator eventStackQueryValidator, + EventCustomFieldQueryPolicy eventCustomFieldQueryPolicy, ApiMapper mapper, LinkGenerator linkGenerator, IHttpContextAccessor httpContextAccessor) @@ -137,6 +143,10 @@ public async Task>> Ha var validationError = NewSavedView.ValidateColumns(definition.ViewType, definition.Columns).FirstOrDefault(); if (validationError is not null) return Result.Invalid(ValidationError.Create("definitions", validationError.ErrorMessage ?? "Invalid column configuration.")); + + string? portabilityError = await ValidatePredefinedFilterPortabilityAsync(definition.ViewType, definition.Filter, definition.FilterDefinitions); + if (portabilityError is not null) + return Result.Invalid(ValidationError.Create("definitions", portabilityError)); } var savedViews = message.Definitions.Select(definition => new SavedView @@ -193,6 +203,10 @@ public async Task> Handle(PromoteToPredefinedSavedView mes if (source is null) return Result.NotFound("Saved view not found."); + string? portabilityError = await ValidatePredefinedFilterPortabilityAsync(source.ViewType, source.Filter, ParseFilterDefinitions(source.FilterDefinitions)); + if (portabilityError is not null) + return Result.Invalid(ValidationError.Create("filter", portabilityError)); + var savedView = await UpsertSystemPredefinedSavedViewAsync(source); return MapToViewModel(savedView); } @@ -231,6 +245,25 @@ public async Task> Handle(UpdateSavedViewMessage message) original.UpdatedByUserId = GetCurrentUserId(); + if (changedNames.Contains(nameof(UpdateSavedView.Filter))) + { + await using var consistencyLock = await lockProvider.TryAcquireAsync( + EventCustomFieldService.GetSavedViewConsistencyLockName(original.OrganizationId), + TimeSpan.FromMinutes(5), + TimeSpan.FromSeconds(5)); + if (consistencyLock is null) + return Result.Conflict("Custom field or saved-view changes are already in progress for this organization. Please try again."); + + var validation = await ValidateFilterAsync(original.ViewType, original.Filter, original.OrganizationId); + if (validation.Error is not null) + return validation.Identifier is null + ? Result.BadRequest(validation.Error) + : Result.Invalid(ValidationError.Create(validation.Identifier, validation.Error)); + + await repository.SaveAsync(original, o => o.Cache().ImmediateConsistency()); + return MapToViewModel(original); + } + await repository.SaveAsync(original, o => o.Cache()); return MapToViewModel(original); } @@ -285,13 +318,51 @@ private async Task> PostImplAsync(NewSavedView value) mapped.CreatedByUserId = GetCurrentUserId(); mapped.Version = 1; - var model = await repository.AddAsync(mapped, o => o.Cache()); + await using var consistencyLock = await lockProvider.TryAcquireAsync( + EventCustomFieldService.GetSavedViewConsistencyLockName(mapped.OrganizationId), + TimeSpan.FromMinutes(5), + TimeSpan.FromSeconds(5)); + if (consistencyLock is null) + return Result.Conflict("Custom field or saved-view changes are already in progress for this organization. Please try again."); + + var validation = await ValidateFilterAsync(mapped.ViewType, mapped.Filter, mapped.OrganizationId); + if (validation.Error is not null) + return validation.Identifier is null + ? Result.BadRequest(validation.Error) + : Result.Invalid(ValidationError.Create(validation.Identifier, validation.Error)); + + var model = await repository.AddAsync(mapped, o => o.Cache().ImmediateConsistency()); var viewModel = MapToViewModel(model); string location = linkGenerator.GetUriByName(HttpContext, "GetSavedViewById", new { id = model.Id }) ?? throw new InvalidOperationException("Unable to generate saved view location."); return Result.Created(viewModel, location); } + private async Task ValidateFilterAsync(string viewType, string? filter, string organizationId) + { + var queryValidator = String.Equals(viewType, "stacks", StringComparison.OrdinalIgnoreCase) + ? (AppQueryValidator)eventStackQueryValidator + : eventQueryValidator; + var queryValidation = await queryValidator.ValidateQueryAsync(filter); + if (!queryValidation.IsValid) + return new FilterValidationResult(queryValidation.Message ?? "Invalid filter."); + + var organization = await organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + if (organization is null) + return new FilterValidationResult("Organization not found."); + + var customFieldValidation = await eventCustomFieldQueryPolicy.ValidateAsync( + queryValidation.ReferencedFields, new AppFilter(organization), HttpContext.RequestAborted); + return customFieldValidation.IsValid + ? FilterValidationResult.Valid + : new FilterValidationResult(customFieldValidation.Message, customFieldValidation.ErrorCode); + } + + private sealed record FilterValidationResult(string? Error = null, string? Identifier = null) + { + public static FilterValidationResult Valid { get; } = new(); + } + private async Task?> CanAddAsync(SavedView value) { if (String.IsNullOrEmpty(value.OrganizationId) || !HttpContext.Request.IsInOrganization(value.OrganizationId)) @@ -731,6 +802,96 @@ private async Task UpsertSystemPredefinedSavedViewAsync(SavedView sou return existing; } + private async Task ValidatePredefinedFilterPortabilityAsync(string viewType, string? filter, JsonElement? filterDefinitions) + { + var queryValidator = String.Equals(viewType, "stacks", StringComparison.OrdinalIgnoreCase) + ? (AppQueryValidator)eventStackQueryValidator + : eventQueryValidator; + var validation = await queryValidator.ValidateQueryAsync(filter); + if (!validation.IsValid) + return validation.Message ?? "Invalid filter."; + + if (validation.ReferencedFields.Any(IsOrganizationCustomFieldReference)) + return "Predefined saved views cannot reference organization custom fields because they are applied to every organization."; + + if (filterDefinitions.HasValue) + { + var structuredTerms = new List(); + var keywordFilters = new List(); + CollectStructuredFilterReferences(filterDefinitions.Value, structuredTerms, keywordFilters); + if (structuredTerms.Any(IsOrganizationCustomFieldReference)) + return "Predefined saved views cannot reference organization custom fields because they are applied to every organization."; + + foreach (string keywordFilter in keywordFilters) + { + var keywordValidation = await queryValidator.ValidateQueryAsync(keywordFilter); + if (!keywordValidation.IsValid) + return keywordValidation.Message ?? "Invalid structured keyword filter."; + if (keywordValidation.ReferencedFields.Any(IsOrganizationCustomFieldReference)) + return "Predefined saved views cannot reference organization custom fields because they are applied to every organization."; + } + } + + return null; + } + + private static void CollectStructuredFilterReferences(JsonElement element, ICollection terms, ICollection keywordFilters) + { + if (element.ValueKind == JsonValueKind.Array) + { + foreach (var item in element.EnumerateArray()) + CollectStructuredFilterReferences(item, terms, keywordFilters); + return; + } + + if (element.ValueKind != JsonValueKind.Object) + return; + + string? type = null; + string? value = null; + foreach (var property in element.EnumerateObject()) + { + if (property.Value.ValueKind == JsonValueKind.String + && property.Name.Equals("term", StringComparison.OrdinalIgnoreCase)) + { + terms.Add(property.Value.GetString()!); + } + else if (property.Value.ValueKind == JsonValueKind.String + && property.Name.Equals("type", StringComparison.OrdinalIgnoreCase)) + { + type = property.Value.GetString(); + } + else if (property.Value.ValueKind == JsonValueKind.String + && property.Name.Equals("value", StringComparison.OrdinalIgnoreCase)) + { + value = property.Value.GetString(); + } + + if (property.Value.ValueKind is JsonValueKind.Array or JsonValueKind.Object) + CollectStructuredFilterReferences(property.Value, terms, keywordFilters); + } + + if (String.Equals(type, "keyword", StringComparison.OrdinalIgnoreCase) && !String.IsNullOrWhiteSpace(value)) + keywordFilters.Add(value); + } + + private static bool IsOrganizationCustomFieldReference(string field) + { + if (field.StartsWith("data.", StringComparison.OrdinalIgnoreCase)) + { + string dataLogicalName = field["data.".Length..]; + return !dataLogicalName.StartsWith('@') && !EventCustomFieldService.IsSystemField(dataLogicalName); + } + + if (!field.StartsWith("idx.", StringComparison.OrdinalIgnoreCase)) + return false; + + string idxLogicalName = field["idx.".Length..]; + return !EventCustomFieldService.IsSystemField(idxLogicalName) + && !EventCustomFieldService.SystemFields.Any(systemField => + String.Equals(systemField.LegacyIdxField, idxLogicalName, StringComparison.OrdinalIgnoreCase)); + } + private SavedView CreateSystemPredefinedSavedView(SavedView source, string key, string slug) { var savedView = new SavedView diff --git a/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs b/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs index fd344fde48..3c25161b2d 100644 --- a/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs +++ b/src/Exceptionless.Web/Api/Messages/OrganizationMessages.cs @@ -19,6 +19,10 @@ public record GetInvoice(string Id, HttpContext Context); public record GetInvoices(string Id, string? Before, string? After, int Limit, HttpContext Context); public record GetPlans(string Id, HttpContext Context); public record ChangeOrganizationPlan(string Id, ChangePlanRequest? Model, string? PlanId, string? StripeToken, string? Last4, string? CouponId, HttpContext Context); +public record GetEventCustomFields(string Id, HttpContext Context); +public record CreateEventCustomField(string Id, NewCustomFieldDefinition Field, HttpContext Context); +public record UpdateEventCustomField(string Id, string FieldId, Delta Changes, HttpContext Context); +public record DeleteEventCustomField(string Id, string FieldId, HttpContext Context); public record AddOrganizationUser(string Id, string Email, HttpContext Context); public record RemoveOrganizationUser(string Id, string Email, HttpContext Context); public record SuspendOrganization(string Id, SuspensionCode Code, string? Notes, HttpContext Context); diff --git a/src/Exceptionless.Web/Api/Results/ApiResultMapper.cs b/src/Exceptionless.Web/Api/Results/ApiResultMapper.cs index 0a0ac78919..fd8ffe2463 100644 --- a/src/Exceptionless.Web/Api/Results/ApiResultMapper.cs +++ b/src/Exceptionless.Web/Api/Results/ApiResultMapper.cs @@ -12,6 +12,10 @@ public static class ApiValidationErrorIdentifiers public const string NotImplemented = "not_implemented"; public const string RateLimit = "rate_limit"; public const string RequestEntityTooLarge = "request_entity_too_large"; + public const string CustomFieldActiveLimit = "custom_field_active_limit"; + public const string CustomFieldLifetimeLimit = "custom_field_lifetime_limit"; + public const string UnknownFilterField = "unknown_filter_field"; + public const string CustomFieldScopeRequired = "custom_field_scope_required"; } /// @@ -125,7 +129,14 @@ public static IResult MapValidation(Foundatio.Mediator.IResult result) group => group.Select(error => error.ErrorMessage).Distinct(StringComparer.OrdinalIgnoreCase).ToArray(), StringComparer.Ordinal); - return HttpResults.ValidationProblem(errorDict, title: title, statusCode: StatusCodes.Status422UnprocessableEntity); + bool isCustomFieldSearchError = errors.Any(error => + String.Equals(error.Identifier, ApiValidationErrorIdentifiers.UnknownFilterField, StringComparison.OrdinalIgnoreCase) + || String.Equals(error.Identifier, ApiValidationErrorIdentifiers.CustomFieldScopeRequired, StringComparison.OrdinalIgnoreCase)); + int statusCode = isCustomFieldSearchError + ? StatusCodes.Status400BadRequest + : StatusCodes.Status422UnprocessableEntity; + + return HttpResults.ValidationProblem(errorDict, title: title, statusCode: statusCode); } private static IResult MapSuccess(Foundatio.Mediator.IResult result) diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/events-overview.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/events-overview.svelte index edfddb5b6e..a2e64d8d1b 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/events-overview.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/events-overview.svelte @@ -15,6 +15,7 @@ import * as EventsFacetedFilter from '$features/events/components/filters'; import { getExtendedDataItems, hasErrorOrSimpleError } from '$features/events/persistent-event'; import { getOrganizationQuery } from '$features/organizations/api.svelte'; + import { getCustomFieldsQuery } from '$features/organizations/custom-fields'; import { getProjectQuery, updateProject } from '$features/projects/api.svelte'; import StackCard from '$features/stacks/components/stack-card.svelte'; import Braces from '@lucide/svelte/icons/braces'; @@ -135,6 +136,15 @@ } }); + const customFieldsQuery = getCustomFieldsQuery({ + route: { + get organizationId() { + return event?.organization_id; + } + } + }); + const customFields = $derived(customFieldsQuery.isSuccess ? customFieldsQuery.data : undefined); + const hasPremiumFeatures = $derived(!organizationQuery.isSuccess || !!organizationQuery.data?.has_premium_features); type TabType = 'Environment' | 'Exception' | 'Extended Data' | 'Overview' | 'Request' | 'Trace Log' | string; @@ -422,9 +432,9 @@ {:else if tab === 'Session Events'} {:else if tab === 'Extended Data'} - + {:else} - + {/if} {/each} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte index 4cdb778894..bad6951ec7 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/extended-data-item.svelte @@ -3,11 +3,14 @@ import { Code, CodeBlock, H4 } from '$comp/typography'; import { Button } from '$comp/ui/button'; import * as DropdownMenu from '$comp/ui/dropdown-menu'; + import { type CustomFieldDefinition, CustomFieldNameSchema } from '$features/organizations/custom-fields'; + import IndexAsCustomFieldAction from '$features/organizations/custom-fields/components/index-as-custom-field-action.svelte'; import { isJSONString, isObject, isString, isXmlString } from '$features/shared/typing'; import { UseClipboard } from '$lib/hooks/use-clipboard.svelte'; import ArrowDown from '@lucide/svelte/icons/arrow-down'; import ArrowUp from '@lucide/svelte/icons/arrow-up'; import Copy from '@lucide/svelte/icons/copy'; + import Database from '@lucide/svelte/icons/database'; import MoreVertical from '@lucide/svelte/icons/more-vertical'; import ToggleLeft from '@lucide/svelte/icons/toggle-left'; import { toast } from 'svelte-sonner'; @@ -15,10 +18,12 @@ interface Props { canPromote?: boolean; class?: string; + customFields?: CustomFieldDefinition[]; data: unknown; demote?: (title: string) => Promise; excludedKeys?: string[]; isPromoted?: boolean; + organizationId?: string; promote?: (title: string) => Promise; showTitle?: boolean; title: string; @@ -27,15 +32,36 @@ let { canPromote = true, class: className, + customFields, data, demote = async () => {}, excludedKeys = [], isPromoted = false, + organizationId, promote = async () => {}, showTitle = true, title }: Props = $props(); + function isPrimitiveData(value: unknown): boolean { + if (value === null || value === undefined) { + return false; + } + + const type = typeof value; + return type === 'string' || type === 'number' || type === 'boolean'; + } + + const canIndex = $derived( + !!organizationId && + !!customFields && + isPrimitiveData(data) && + CustomFieldNameSchema.safeParse(title).success && + !['haserror', 'sessionend'].includes(title.toLowerCase()) && + !customFields.some((field) => field.name.toLowerCase() === title.toLowerCase()) + ); + let showIndexDialog = $state(false); + function transformData(data: unknown): unknown { if (isJSONString(data)) { try { @@ -153,9 +179,18 @@ {/if} {/if} + {#if canIndex} + (showIndexDialog = true)} title="Index this field for filtering"> + + Index as Custom Field + + {/if} + {#if canIndex} + + {/if}
diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/boolean-faceted-filter-builder.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/boolean-faceted-filter-builder.svelte index b18dc3ec7e..9360f9d6a0 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/boolean-faceted-filter-builder.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/boolean-faceted-filter-builder.svelte @@ -24,6 +24,13 @@ }; $effect(() => { - builderContext.set(`boolean-${term}`, builder as unknown as FacetFilterBuilder); + const key = `boolean-${term}`; + const registeredBuilder = builder as unknown as FacetFilterBuilder; + builderContext.set(key, registeredBuilder); + return () => { + if (builderContext.get(key) === registeredBuilder) { + builderContext.delete(key); + } + }; }); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/date-faceted-filter-builder.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/date-faceted-filter-builder.svelte index 0b5ec67c4a..737ae66b4b 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/date-faceted-filter-builder.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/date-faceted-filter-builder.svelte @@ -24,6 +24,13 @@ }; $effect(() => { - builderContext.set(`date-${term}`, builder as unknown as FacetFilterBuilder); + const key = `date-${term}`; + const registeredBuilder = builder as unknown as FacetFilterBuilder; + builderContext.set(key, registeredBuilder); + return () => { + if (builderContext.get(key) === registeredBuilder) { + builderContext.delete(key); + } + }; }); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.test.ts index 7d63ed90aa..323cbb1b4c 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.test.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.test.ts @@ -203,7 +203,15 @@ describe('serializeFilters', () => { const filters = [new NumberFilter('value', 42)]; const result = JSON.parse(serializeFilters(filters)); - expect(result[0]).toEqual({ term: 'value', type: 'number', value: 42 }); + expect(result[0]).toEqual({ term: 'value', type: 'number', value: '42' }); + }); + + it('preserves a 64-bit integer literal exactly', () => { + const filters = [new NumberFilter('idx.order_id', '9223372036854775807')]; + const result = deserializeFilters(serializeFilters(filters)); + + expect((result[0] as NumberFilter).value).toBe('9223372036854775807'); + expect(result[0]?.toFilter()).toBe('idx.order_id:9223372036854775807'); }); it('serializes a ProjectFilter with multiple values', () => { @@ -357,7 +365,7 @@ describe('deserializeFilters', () => { expect(filters).toHaveLength(1); expect(filters[0]).toBeInstanceOf(NumberFilter); expect((filters[0] as NumberFilter).term).toBe('value'); - expect((filters[0] as NumberFilter).value).toBe(42); + expect((filters[0] as NumberFilter).value).toBe('42'); }); it('deserializes a ProjectFilter', () => { @@ -497,7 +505,7 @@ describe('round-trip serialization', () => { expect(result).toHaveLength(1); expect((result[0] as NumberFilter).term).toBe('count'); - expect((result[0] as NumberFilter).value).toBe(99); + expect((result[0] as NumberFilter).value).toBe('99'); }); it('round-trips a ProjectFilter', () => { diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.ts index 8a4554c7c7..9bb81a4f2b 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/helpers.svelte.ts @@ -301,7 +301,7 @@ function reconstructFilter(data: SerializedFilter): IFilter | null { filter = new LevelFilter(data.value as LogLevel[] | undefined); break; case 'number': - filter = new NumberFilter(data.term, data.value as number | undefined); + filter = new NumberFilter(data.term, data.value as number | string | undefined); break; case 'project': filter = new ProjectFilter(data.value as string[] | undefined); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/models.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/models.svelte.ts index ce9d78ac2b..e06aeb4c7c 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/models.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/models.svelte.ts @@ -152,15 +152,15 @@ export class NumberFilter implements IFilter { public term = $state(); public type: string = 'number'; - public value = $state(); + public value = $state(); public get key(): string { return `${this.type}-${this.term}`; } - constructor(term?: string, value?: number) { + constructor(term?: string, value?: number | string) { this.term = term; - this.value = value; + this.value = value === undefined ? undefined : String(value); } public clone(): IFilter { diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-builder.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-builder.svelte index c9ca1399ae..9305303c51 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-builder.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-builder.svelte @@ -24,6 +24,13 @@ }; $effect(() => { - builderContext.set(`number-${term}`, builder as unknown as FacetFilterBuilder); + const key = `number-${term}`; + const registeredBuilder = builder as unknown as FacetFilterBuilder; + builderContext.set(key, registeredBuilder); + return () => { + if (builderContext.get(key) === registeredBuilder) { + builderContext.delete(key); + } + }; }); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-trigger.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-trigger.svelte index 8ae36d167a..d292608747 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-trigger.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/number-faceted-filter-trigger.svelte @@ -7,7 +7,7 @@ type Props = Omit & { changed: (filter: NumberFilter) => void; term: string; - value?: number; + value?: number | string; }; let { changed, children, class: className, term, value, ...props }: Props = $props(); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte index c4da6c8e32..568858e0bd 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/organization-defaults-faceted-filter-builder.svelte @@ -4,6 +4,8 @@ } const { includeDateFacets = true }: Props = $props(); + import CustomFieldFilterBuilders from '$features/organizations/custom-fields/components/custom-field-filter-builders.svelte'; + import * as FacetedFilter from './index'; type KnownTermsFilterConfig = { @@ -106,3 +108,5 @@ {#each eventsVersionFilters as { priority, term, title } (term)} {/each} + + diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/string-faceted-filter-builder.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/string-faceted-filter-builder.svelte index 4cc6d3c12d..36e608a880 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/string-faceted-filter-builder.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/filters/string-faceted-filter-builder.svelte @@ -24,6 +24,13 @@ }; $effect(() => { - builderContext.set(`string-${term}`, builder as unknown as FacetFilterBuilder); + const key = `string-${term}`; + const registeredBuilder = builder as unknown as FacetFilterBuilder; + builderContext.set(key, registeredBuilder); + return () => { + if (builderContext.get(key) === registeredBuilder) { + builderContext.delete(key); + } + }; }); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/table/events-bulk-actions-dropdown-menu.svelte.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/table/events-bulk-actions-dropdown-menu.svelte.test.ts index 67e51e4f63..5a3a202532 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/table/events-bulk-actions-dropdown-menu.svelte.test.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/table/events-bulk-actions-dropdown-menu.svelte.test.ts @@ -1,5 +1,5 @@ -import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; -import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; const mutateAsync = vi.hoisted(() => vi.fn()); const deleteEvent = vi.hoisted(() => vi.fn(() => ({ mutateAsync }))); @@ -17,6 +17,13 @@ describe('EventsBulkActionsDropdownMenu', () => { toast.success.mockClear(); }); + afterEach(async () => { + cleanup(); + // Bits UI restores the body scroll lock on a short timer after overlays unmount. + // Let that cleanup finish while jsdom's document is still available. + await new Promise((resolve) => window.setTimeout(resolve, 30)); + }); + it('deletes the selected events and clears the selection', async () => { // Arrange const resetRowSelection = vi.fn(); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/environment.svelte.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/environment.svelte.test.ts index 2e7508e0cc..6632f1045c 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/environment.svelte.test.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/environment.svelte.test.ts @@ -2,6 +2,7 @@ import { render, screen } from '@testing-library/svelte'; import { describe, expect, it, vi } from 'vitest'; vi.mock('$features/events/components/filters', () => ({ StringTrigger: null })); +vi.mock('../extended-data-item.svelte', () => ({ default: null })); import type { PersistentEvent } from '../../models'; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/extended-data.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/extended-data.svelte index 249e0c9350..0d65204929 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/extended-data.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/events/components/views/extended-data.svelte @@ -1,4 +1,5 @@ - + diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/api.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/api.svelte.ts new file mode 100644 index 0000000000..b5a738370e --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/api.svelte.ts @@ -0,0 +1,133 @@ +import type { + CustomFieldDefinitionResponse as ApiCustomFieldDefinition, + NewCustomFieldDefinition as ApiNewCustomFieldDefinition, + UpdateCustomFieldDefinition as ApiUpdateCustomFieldDefinition +} from '$lib/generated/api'; +import type { ProblemDetails } from '@foundatiofx/fetchclient'; + +import { accessToken } from '$features/auth/index.svelte'; +import { useFetchClient } from '@foundatiofx/fetchclient'; +import { createMutation, createQuery, useQueryClient } from '@tanstack/svelte-query'; + +import { type CustomFieldDefinition, type NewCustomFieldDefinition, parseApiIndexType, type UpdateCustomFieldDefinition } from './models'; + +export const queryKeys = { + customFields: (organizationId: string | undefined) => ['Organization', organizationId, 'custom-fields'] as const, + type: ['CustomField'] as const +}; + +export interface CreateCustomFieldRequest { + route: { + organizationId: string; + }; +} + +export interface DeleteCustomFieldRequest { + route: { + fieldId: string; + organizationId: string; + }; +} + +export interface GetCustomFieldsRequest { + route: { + organizationId: string | undefined; + }; +} + +export interface UpdateCustomFieldRequest { + route: { + fieldId: string; + organizationId: string; + }; +} + +export function createCustomFieldMutation(request: CreateCustomFieldRequest) { + const queryClient = useQueryClient(); + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId, + mutationFn: async (data: NewCustomFieldDefinition) => { + const client = useFetchClient(); + const response = await client.postJSON( + `organizations/${request.route.organizationId}/event-custom-fields`, + mapNewFieldRequest(data) + ); + return mapApiDefinition(response.data!); + }, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.customFields(request.route.organizationId) }); + } + })); +} + +export function deleteCustomFieldMutation(request: DeleteCustomFieldRequest) { + const queryClient = useQueryClient(); + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId && !!request.route.fieldId, + mutationFn: async () => { + const client = useFetchClient(); + await client.delete(`organizations/${request.route.organizationId}/event-custom-fields/${request.route.fieldId}`); + }, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.customFields(request.route.organizationId) }); + } + })); +} + +export function getCustomFieldsQuery(request: GetCustomFieldsRequest) { + return createQuery(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId, + queryFn: async ({ signal }: { signal: AbortSignal }) => { + const client = useFetchClient(); + const response = await client.getJSON(`organizations/${request.route.organizationId}/event-custom-fields`, { signal }); + return response.data?.map(mapApiDefinition) ?? []; + }, + queryKey: queryKeys.customFields(request.route.organizationId) + })); +} + +export function updateCustomFieldMutation(request: UpdateCustomFieldRequest) { + const queryClient = useQueryClient(); + return createMutation(() => ({ + enabled: () => !!accessToken.current && !!request.route.organizationId && !!request.route.fieldId, + mutationFn: async (data: UpdateCustomFieldDefinition) => { + const client = useFetchClient(); + const response = await client.patchJSON( + `organizations/${request.route.organizationId}/event-custom-fields/${request.route.fieldId}`, + mapUpdateFieldRequest(data) + ); + return mapApiDefinition(response.data!); + }, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: queryKeys.customFields(request.route.organizationId) }); + } + })); +} + +function mapApiDefinition(definition: ApiCustomFieldDefinition): CustomFieldDefinition { + return { + createdUtc: definition.created_utc, + description: definition.description ?? undefined, + displayOrder: definition.display_order, + id: definition.id, + indexType: parseApiIndexType(definition.index_type), + name: definition.name, + updatedUtc: definition.updated_utc + }; +} + +function mapNewFieldRequest(data: NewCustomFieldDefinition): ApiNewCustomFieldDefinition { + return { + description: data.description, + display_order: data.displayOrder, + index_type: data.indexType, + name: data.name + }; +} + +function mapUpdateFieldRequest(data: UpdateCustomFieldDefinition): ApiUpdateCustomFieldDefinition { + return { + description: data.description, + display_order: data.displayOrder + }; +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/custom-field-filter-builders.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/custom-field-filter-builders.svelte new file mode 100644 index 0000000000..691abe8495 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/custom-field-filter-builders.svelte @@ -0,0 +1,51 @@ + + +{#each activeFields as field (field.id)} + {@const filterType = getFilterType(field)} + {@const term = `idx.${field.name}`} + {@const title = field.description || field.name} + + {#if filterType === 'boolean'} + + {:else if filterType === 'number'} + + {:else if filterType === 'date'} + + {:else} + + {/if} +{/each} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/index-as-custom-field-action.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/index-as-custom-field-action.svelte new file mode 100644 index 0000000000..f00de65229 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/components/index-as-custom-field-action.svelte @@ -0,0 +1,144 @@ + + +{#if organizationId && canCreate} + + + + Index "{fieldName}" as Custom Field + + This will start indexing "{fieldName}" from future events, making it available for filtering and search. Existing events will not be + retroactively indexed. + + + +
{ + e.preventDefault(); + form.handleSubmit(); + }} + > + state.errors}> + {#snippet children(errors)} + + {/snippet} + + + + {#snippet children(field)} + + Index Type + field.handleChange(parseIndexType(value))}> + + {INDEX_TYPE_LABELS[field.state.value]} + {INDEX_TYPE_DESCRIPTIONS[field.state.value]} + + + + {#each INDEX_TYPES as type (type)} + + {INDEX_TYPE_LABELS[type]} + {INDEX_TYPE_DESCRIPTIONS[type]} + + {/each} + + + + Choose the type that best matches this field's data. + + {/snippet} + + + + + state.isSubmitting}> + {#snippet children(isSubmitting)} + + {/snippet} + + +
+ + +{/if} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/index.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/index.ts new file mode 100644 index 0000000000..b8338eb412 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/index.ts @@ -0,0 +1,12 @@ +export { createCustomFieldMutation, deleteCustomFieldMutation, getCustomFieldsQuery, queryKeys, updateCustomFieldMutation } from './api.svelte'; +export type { CustomFieldDefinition, IndexType, NewCustomFieldDefinition, UpdateCustomFieldDefinition } from './models'; +export { INDEX_TYPE_DESCRIPTIONS, INDEX_TYPE_LABELS, INDEX_TYPES, parseApiIndexType, parseIndexType } from './models'; +export { + type CreateCustomFieldFormData, + CreateCustomFieldSchema, + CustomFieldNameSchema, + type QuickCreateCustomFieldFormData, + QuickCreateCustomFieldSchema, + type UpdateCustomFieldFormData, + UpdateCustomFieldSchema +} from './schemas'; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.test.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.test.ts new file mode 100644 index 0000000000..ea83d53a6a --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from 'vitest'; + +import { parseApiIndexType } from './models'; +import { CustomFieldNameSchema } from './schemas'; + +describe('parseApiIndexType', () => { + it('accepts supported server values', () => { + expect(parseApiIndexType('keyword')).toBe('keyword'); + expect(parseApiIndexType('long')).toBe('long'); + }); + + it('fails loudly for an unknown server value', () => { + expect(() => parseApiIndexType('decimal')).toThrow("Unsupported custom-field index type 'decimal'."); + }); +}); + +describe('CustomFieldNameSchema', () => { + it.each(['keyword-7', 'bool-1', 'session-r', 'sessionend-d', 'haserror-b'])('rejects internal storage name %s', (name) => { + expect(CustomFieldNameSchema.safeParse(name).success).toBe(false); + }); +}); diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.ts new file mode 100644 index 0000000000..91f24247df --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/models.ts @@ -0,0 +1,59 @@ +export const INDEX_TYPES = ['keyword', 'string', 'int', 'long', 'float', 'double', 'bool', 'date'] as const; +export interface CustomFieldDefinition { + createdUtc: string; + description?: string; + displayOrder: number; + id: string; + indexType: IndexType; + name: string; + updatedUtc: string; +} + +export type IndexType = (typeof INDEX_TYPES)[number]; + +export interface NewCustomFieldDefinition { + description?: string; + displayOrder?: number; + indexType: IndexType; + name: string; +} + +export interface UpdateCustomFieldDefinition { + description?: string; + displayOrder?: number; +} + +export const INDEX_TYPE_LABELS: Record = { + bool: 'Boolean', + date: 'Date/Time', + double: 'Double', + float: 'Float', + int: 'Integer', + keyword: 'Keyword', + long: 'Long', + string: 'Text' +}; + +export const INDEX_TYPE_DESCRIPTIONS: Record = { + bool: 'True/false values.', + date: 'ISO 8601 date or timestamp.', + double: '64-bit binary floating-point. Use Keyword when formatting such as "4.90" matters.', + float: '32-bit binary floating-point. Use Keyword when formatting such as "4.90" matters.', + int: '32-bit whole number (-2B to 2B).', + keyword: 'Exact-match string. Best for IDs, codes, tags.', + long: '64-bit whole number. For very large integers.', + string: 'Full-text search. Best for messages and descriptions.' +}; + +export function parseApiIndexType(value: string): IndexType { + const indexType = INDEX_TYPES.find((candidate) => candidate === value); + if (!indexType) { + throw new Error(`Unsupported custom-field index type '${value}'.`); + } + + return indexType; +} + +export function parseIndexType(value: null | string | undefined, fallback: IndexType = 'keyword'): IndexType { + return INDEX_TYPES.find((indexType) => indexType === value) ?? fallback; +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/schemas.ts new file mode 100644 index 0000000000..94b3553967 --- /dev/null +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/custom-fields/schemas.ts @@ -0,0 +1,32 @@ +import { type infer as Infer, object, string, enum as zodEnum } from 'zod'; + +import { INDEX_TYPES } from './models'; + +export const CustomFieldNameSchema = string() + .trim() + .min(1, 'Field name is required.') + .max(100, 'Field name cannot exceed 100 characters.') + .regex(/^[a-zA-Z0-9_.-]+$/, 'Use only letters, digits, underscore, dot, and dash.') + .refine((name) => !name.startsWith('@'), 'Field names cannot start with @.') + .refine( + (name) => !/^(?:bool|date|double|float|int|keyword|long|string)-\d+$/i.test(name) && !/^(?:session-r|sessionend-d|haserror-b)$/i.test(name), + 'This name is reserved for internal search storage.' + ); + +export const CreateCustomFieldSchema = object({ + description: string().max(500, 'Description cannot exceed 500 characters.'), + indexType: zodEnum(INDEX_TYPES), + name: CustomFieldNameSchema +}); + +export const UpdateCustomFieldSchema = object({ + description: string().max(500, 'Description cannot exceed 500 characters.') +}); + +export const QuickCreateCustomFieldSchema = object({ + indexType: zodEnum(INDEX_TYPES) +}); + +export type CreateCustomFieldFormData = Infer; +export type QuickCreateCustomFieldFormData = Infer; +export type UpdateCustomFieldFormData = Infer; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/shared/components/faceted-filter/faceted-filter-number.svelte b/src/Exceptionless.Web/ClientApp/src/lib/features/shared/components/faceted-filter/faceted-filter-number.svelte index 2ad87f8296..a446b61d26 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/shared/components/faceted-filter/faceted-filter-number.svelte +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/shared/components/faceted-filter/faceted-filter-number.svelte @@ -7,13 +7,13 @@ import { onDestroy } from 'svelte'; interface Props { - changed: (value?: number) => void; + changed: (value?: string) => void; hidden?: boolean; open: boolean; remove: () => void; title: string; toggleHidden?: () => void; - value?: number; + value?: string; } let { changed, hidden = false, open = $bindable(), remove, title, toggleHidden, value }: Props = $props(); @@ -21,7 +21,7 @@ const DEBOUNCE_MS = 500; // eslint-disable-next-line svelte/prefer-writable-derived - let updatedValue = $state(); + let updatedValue = $state(); let debounceTimer: ReturnType | undefined; onDestroy(() => clearTimeout(debounceTimer)); @@ -32,9 +32,14 @@ function scheduleApply() { clearTimeout(debounceTimer); + const candidate = updatedValue; + if (!isValidNumericLiteral(candidate)) { + return; + } + debounceTimer = setTimeout(() => { - if (updatedValue !== value) { - changed(updatedValue); + if (isValidNumericLiteral(candidate) && candidate !== value) { + changed(candidate); } }, DEBOUNCE_MS); } @@ -51,6 +56,10 @@ function applyAndClose() { clearTimeout(debounceTimer); + if (!isValidNumericLiteral(updatedValue)) { + return; + } + if (updatedValue !== value) { changed(updatedValue); } @@ -76,8 +85,13 @@ } export function onClearFilter() { + clearTimeout(debounceTimer); updatedValue = undefined; } + + function isValidNumericLiteral(candidate: string | undefined): boolean { + return candidate === undefined || /^-?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?$/.test(candidate); + } @@ -86,7 +100,7 @@ + {/if} +
+ + {#if organizationQuery.isSuccess && !canManageFields} +
+ Custom field definitions remain available for reference, but adding, editing, and deleting fields requires a paid plan. +
+ {/if} + + {#if customFieldsQuery.isPending} +
+ + + +
+ {:else if customFieldsQuery.isError} + + {:else if !hasFields} +
+ +

No custom fields yet

+ Add a field to start indexing event data properties for use in filters and search. + {#if canManageFields} + + {/if} +
+ {:else} +
+ + + + Name + Type + Description + Added + {#if canManageFields}Actions{/if} + + + + {#each allFields as field (field.id)} + + {field.name} + + + {INDEX_TYPE_LABELS[field.indexType]} + + + {field.description ?? '—'} + + + + {#if canManageFields} + +
+ + +
+ + {/if} + + {/each} + + +
+ {/if} + + + + + + + Add Custom Field + + Define a new indexed field from event data. The field name must match a key in your event's extended data. + Indexing applies to new events only — historical events are not backfilled. + + + +
{ + e.preventDefault(); + createFormState.handleSubmit(); + }} + > + state.errors}> + {#snippet children(errors)} + + {/snippet} + + + + {#snippet children(field)} + + Field Name * + field.handleChange(event.currentTarget.value)} + placeholder="e.g., customer_id" + required + maxlength={100} + autocomplete="off" + spellcheck={false} + aria-invalid={ariaInvalid(field)} + /> + + {#if field.state.meta.errors.length === 0} + Must match a key in event data. Letters, digits, _ . - only. + {/if} + + {/snippet} + + + + {#snippet children(field)} + + Index Type * + field.handleChange(parseIndexType(value))}> + + {INDEX_TYPE_LABELS[field.state.value]} + {INDEX_TYPE_DESCRIPTIONS[field.state.value]} + + + + {#each INDEX_TYPES as type (type)} + + {INDEX_TYPE_LABELS[type]} + {INDEX_TYPE_DESCRIPTIONS[type]} + + {/each} + + + + Determines how the field is stored and what filter operators are available. + + {/snippet} + + + + {#snippet children(field)} + + Description (optional) +