I've developed a fundamentally different compression algorithm called FSP (Find Similar Patterns) that breaks from traditional dictionary and statistical compression methods. While most compression algorithms struggle with small files due to inherent overhead, FSP takes a completely different approach that makes it uniquely effective for specific use cases.

How FSP Differs from Traditional Compression:

Unlike LZ77/LZ78 (used in ZIP), LZMA, Huffman coding, or other common algorithms, FSP doesn't rely on:

· Sliding window dictionary approaches · Frequency statistics or entropy coding · Block-based compression with fixed headers · Move-to-front transformations or Burrows-Wheeler transforms

Instead, FSP uses a pattern similarity approach that:

1. Identifies similar patterns across data chunks 2. Stores only differential changes between patterns 3. Uses position-based editing rather than token replacement 4. Has virtually no overhead for highly similar data

The Unique Advantages:

What makes FSP novel isn't just that it handles small files well – it's that it represents a different philosophical approach to compression:

1. No Dictionary Overhead: Unlike LZ variants that must store dictionaries, FSP only stores differences 2. Position-Aware Editing: Rather than replacing tokens, FSP uses exact positional editing 3. Similarity-Based: Excels where data has structural similarity rather than just byte-level repetition 4. Stream-Compatible: Processes data in logical chunks rather than fixed blocks

Technical Differentiators:

· Unlike LZ77: No sliding window or distance-length encoding · Unlike Huffman/Arithmetic: No frequency tables or probability models · Unlike RLE: Handles non-sequential patterns and similarities · Unlike BWT: Doesn't require full data transformation and rearrangement

Where FSP Excels:

The algorithm particularly shines in:

· Versioned data: Where consecutive versions have minor changes · Structured records: Database entries with similar schema but different values · Sensor data: Regular readings with small fluctuations · Log files: Similar log entries with varying parameters · Genomic data: Sequences with localized variations

Performance Characteristics:

In testing, FSP achieves what traditional compressors cannot: consistent compression of very small data chunks without the overhead that typically plagues small-file compression. Where ZIP might add 50+ bytes of overhead, FSP adds virtually none for similar patterns.

Open Questions for Discussion:

I'm particularly interested in the HN community's thoughts on:

1. Theoretical classification of this approach within compression taxonomy 2. Potential hybrid approaches combining FSP with traditional methods 3. Mathematical analysis of the similarity detection problem 4. Applications in distributed systems where small payload compression matters 5. Comparisons to other non-traditional compression approaches

The algorithm is open-source under LGPL 3.0, and I welcome both theoretical and practical contributions. Sometimes innovation comes not from better implementations of existing approaches, but from fundamentally different ways of thinking about problems.

GitHub: https://github.com/Ferki-git-creator/fsp Website(more info): https://ferki-git-creator.github.io/fsp/