For a long time, I’ve used Kafka for services for some purpose. like tracing user action log, event sourcing propagation pipeline step processing and so on.

Whenever I use Kafka, I need to set up kafka ecosystem such as, ELK ( Elasticsearch + Logstash + Kibana) for search Kafka messages for service operation, issue trace. and Kafka sink connector for some data forward other persistent DB, and ksqlDB( or Flink ) for count and monitor event stream. With additional infra and software

One day, a thought suddenly crossed my mind.

That is “Basically Kafka is processing message stream sequentially, with a consumer.

It means If I don’t know what the partition and offset of “needle” is, I have to find needle sequentially from beginning. but if I know Kafka message partition and offset, Can I find and access data directly like other DBMS?”

ELK is not necessary on my service at that time. Just I need to search trace events and it is in Kafka topic message. All I need to know is where the data I want to find is

My desktop and your desktop are too powerful, expensive to use only for web browser operation and AI question agent.

Let’s use local disk and local application for indexing storage and search engine about Kafka message.

This local-first concept is implemented as “Kaflow Search desktop tool”

If you don’t need to keep data for a long time or permanently You don’t have to go through the hassle of installing and managing ELK or other Kafka ecosystem infrastructure

Just keep and sync index with Kafka cluster server And find messages you want easily with just this tool installed It will save your operation time.

When operating company services, The costs of installing server, managing additional ELK/Kafka Connect, ksqlDB instances didn’t come out of my own pocket

so I felt free to spend my time and server resources to set additional data pipeline up.

but if it is my own service that I afford, I can’t help saving my server cost. I can’t afford the additional infra costs from cloud service (like AWS)

I’m not a rich developer, who just has a laptop to develop software. I’ll make the most of my laptop resources, until my service generates enough money, and then I can handle additional server costs. I’ll only use Kafka and this tool on my service, before I really need ELK and other Kafka ecosystem.

I dedicate this rich client Kafka tool to developers who, like me, are poor.

GitHub : https://github.com/whsoul/kaflow-search

=> Download From release assets ( macOS, Windows ) - https://github.com/whsoul/kaflow-search/releases/tag/v0.1.3

  From brew ( macOS )
     - brew install --cask whsoul/tap/kaflow-search

=> Download demo From release assets ( macOS, Windows ) - https://github.com/whsoul/kaflow-search/releases/tag/demo-v0... From brew ( macOS ) - brew install --cask whsoul/tap/kaflow-search-demo

Please try my tool ( on your macOS, Windows ) and feedback for me consumer ID, consumer Group do not remain on your cluster. It does only read operation, not managing function ( such as delete, create topic, repartition cluster topic). It normally only reads once messages that you select, and read local-first index for search your message. So feel free to install this.

But it is not developer signed.. (Sorry because I’m poor developer..) Please follow this install guide ( https://github.com/whsoul/kaflow-search#install ) about untrusted developer warning when install and first launch.

The Incredible Journey of a Migratory Shorebird

Radius

When the fractional part of a float fixes your shader

Geoffrey Litt: Dynamic Documents as Personal Software [video]

Hackers Got Inside a Flock Camera. Its Data Shows How the System Works

Hackers Stole Flock Camera Software Revealing How Company Tracks Cars and People

Chimps teach their kids tool tricks

The Roman telescope has enough gas for 22 years, double NASA's expectations

Why building a Rust LSP is hard

DOE Seeks New Path for Future Light Bulb Standard

Readest – free open-source ePub and PDF reader

One prompt, answers from multiple AI models side by side

I replaced my brown-noise browser tab with a menu bar app

Show HN: AI Subtools – 40 free online tools, no sign-up, 100% client-side

Personal Statement on AI Risk

Linux development is benefiting from AI, but not from its coding

I joined the 'anti-chatbox' interface movement, and build Moe because of it

Show HN: Airmash – HTML5 Multiplayer Missile Warfare

Show HN: Kaflow Search – A desktop tool for locally indexed Kafka message search

Benchmark Llamacpp on Intel Panther Lake: Vulkan vs. SYCL vs. OpenVINO vs. CPU

Building agents? Here is everything you need to know in one long-read

Show HN: Sloptris - If Tetris was like AI-assisted dev

OpenAI Considers Pre-IPO Funding Round at More Than $1.2T Valuation

Scaling Golang CI by Replacing actions/setup-go

Show HN: Picxel – Turn reference images into pixel-art game assets

Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

ImpactGate: A merge gate that scores the structural decay AI adds

Voyager – defense revenue as a patience subsidy for commercial space station bet

Corporate and political systems are efficient filters to promote psychopaths

Show HN: Git-chain – Simplify branch dependency management in Git

Comments