Skip to content

[Proposal] Replace 54 individual filter plugins with a single universal plugin#1161

Draft
haimofergmail wants to merge 2 commits intoIBM:mainfrom
haimofergmail:universal-filter-plugin
Draft

[Proposal] Replace 54 individual filter plugins with a single universal plugin#1161
haimofergmail wants to merge 2 commits intoIBM:mainfrom
haimofergmail:universal-filter-plugin

Conversation

@haimofergmail
Copy link
Copy Markdown

@haimofergmail haimofergmail commented Mar 20, 2026

👋 Hi team — this is an architectural suggestion, not a finished feature.
I'm opening it as a draft to spark discussion. Please treat the code as a reference
implementation to make the idea concrete, not as something ready to merge.


The suggestion

Replace all 54 logstash-filter-*-guardium plugins with a single generic plugin
where each datasource is a thin parser class (or ideally just a config file).

Every filter plugin in this repo is ~95% identical boilerplate. The only unique part
is the 50–150 lines of parsing logic per datasource. That boilerplate is copy-pasted
54 times, which means:

  • A security fix must be applied to 54 files
  • Adding a new datasource requires scaffolding a full Logstash plugin (~500 lines, new gem)
  • 54 separate gems to build, test, and ship

What the proposal looks like

Before (54 plugins, each with its own gem):

filter { mysql_filter_guardium {}    }
filter { mongodb_guardium_filter {}  }
filter { guardium_snowflake_filter{} }

After (one plugin, datasource selected via config):

filter { guardium_universal_filter { datasource => "mysql"     } }
filter { guardium_universal_filter { datasource => "mongodb"   } }
filter { guardium_universal_filter { datasource => "snowflake" } }

Adding a new datasource would go from ~500 lines + new gem → ~100 lines, no new gem.


What's in this PR

  • logstash-filter-guardium-universal/ — a working reference implementation
  • ARCHITECTURE.md — full design rationale, old-vs-new diagrams, migration strategy, and open questions for the team
  • MySQL fully migrated as a concrete example; MongoDB and Snowflake as thin connectors

Open questions for the team

  • Is this direction aligned with the project's goals?
  • Should IGuardiumParser live in common to allow independent parser JARs?
  • Should simple datasources eventually be driven by YAML config files, with Java only for complex ones?

Happy to iterate on the design or answer any questions. Thanks for your time!

@haimofergmail

…dividual filter plugins

Introduces a new architectural pattern where one generic Logstash plugin
(guardium_universal_filter) replaces all individual logstash-filter-*-guardium
plugins. Each datasource is now a thin parser class implementing IGuardiumParser,
with zero Logstash boilerplate.

Key components:
- GuardiumUniversalFilter: single @LogstashPlugin handling all event-loop boilerplate
- IGuardiumParser: contract for datasource parsers (one method: parseRecord)
- AbstractGuardiumParser: shared utilities (correctIPs, IP validation)
- ParserRegistry: maps datasource name → parser instance
- MySqlParser: fully migrated MySQL audit log parser
- MongoDbParser: thin connector to existing MongoDB parser hierarchy
- SnowflakeParser: thin connector to existing Snowflake parser hierarchy
- ARCHITECTURE.md: design rationale and old-vs-new diagrams

Adding a new datasource now requires one class + one line in ParserRegistry
instead of a full plugin scaffold (~500 lines → ~100 lines per datasource).

Signed-off-by: haimofergmail <haimofer@gmail.com>
Clarifies this is a suggestion for discussion, not a finished implementation.
Adds questions for the team and a clear migration strategy.

Signed-off-by: haimofergmail <haimofer@gmail.com>
@haimofergmail haimofergmail marked this pull request as draft March 20, 2026 21:18
@haimofergmail haimofergmail changed the title Add logstash-filter-guardium-universal: single plugin replacing 54 individual filter plugins [Proposal] Replace 54 individual filter plugins with a single universal plugin Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant