Hi HN,

I built goxe, a tool designed to tackle one of the most annoying (and expensive) problems in infrastructure: log spam.

When a service starts failing, it often floods your pipeline with thousands of identical log lines. This doesn't just create noise; it increases bandwidth usage and inflates storage costs in platforms like Datadog or Splunk.

Goxe sits in your pipeline (currently via Syslog/UDP), normalizes incoming logs by stripping timestamps and variable noise, and aggregates them in real-time. Instead of 10,000 lines of the same error, you get one line with an occurrence count: error: connection failed (x10000).

Key Features:

    Performance: Written in Go with a worker pool architecture and streaming design for low memory overhead.

    Normalization: Strips dates, converts to lowercase, and removes whitespace to ensure near-identical logs are grouped correctly.

    Clustering: It handles similarity clustering to catch messages that are almost the same.

    Transparency: Apache 2.0 license.

I’m currently moving towards v1.0 and just implemented similarity clustering and syslog ingestion. I’d love to get feedback on the architecture or features you'd like to see next (e.g., more ingestion backends like Kafka or Promtail).

Repo: https://github.com/DumbNoxx/Goxe

Thanks for checking it out!