One of the biggest bottlenecks in security is the "log onboarding" tax. If an engineer doesn't know the schema, the data often just sits there or breaks the pipeline.
Long ago, I started to build loops that uses schema inference in the ETL stage to enable true zero-touch ingestion. Once the data is in, we use Energy-Based Models (EBMs) to find anomalies (High Energy = Uncertainty).
The real kicker: the system then validates the response via simulation. If a playbook misses a high-energy event, the optimization engine flags it for an update. We've seen a 57% MTTR improvement using this "Build Once, Learn Always" approach. Curious if anyone else is using schema inference for high-volume security telemetry (40PB+ scale)?
projectnexus•1h ago
Long ago, I started to build loops that uses schema inference in the ETL stage to enable true zero-touch ingestion. Once the data is in, we use Energy-Based Models (EBMs) to find anomalies (High Energy = Uncertainty).
The real kicker: the system then validates the response via simulation. If a playbook misses a high-energy event, the optimization engine flags it for an update. We've seen a 57% MTTR improvement using this "Build Once, Learn Always" approach. Curious if anyone else is using schema inference for high-volume security telemetry (40PB+ scale)?