frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Old Bitly Links Are Being Hijacked

https://ransomnews.com/burger-king-tweet-bitly-link-hijack-2026/
147•hspeiser•2h ago•74 comments

Preserving 30 Years of Salon Journalism

https://blog.archive.org/2026/11/12/preserving-30-years-of-salon-journalism/
59•CharlesW•2h ago•11 comments

What's Next for Nvidia?

https://steveblank.com/2026/11/15/whats-next-for-nvidia/
20•zdw•1h ago•9 comments

Mini.nvim Is Now on Tangled

https://tangled.org/nvim-mini.org/mini.nvim
22•nerdypepper•1h ago•7 comments

Protomaps: Self-hosting map tiles has gotten cheap and easy

https://bruceroettgers.eu/blog/protomaps/
151•bcye•5h ago•34 comments

A short explainer on how El Niño works [video]

https://www.youtube.com/watch?v=7_JF3etiAv8
174•nharada•6h ago•71 comments

WebGL Flight Simulator with seed generated terrain

https://baron.by.dai.studio/
82•ranger_danger•4h ago•10 comments

'Birdbrain' Is a Misnomer. Crows Make Decisions Faster Than We Can

https://www.nytimes.com/2026/11/15/science/birdbrain-crows-birds-research-decision.html
162•gumby•7h ago•121 comments

The MacBook Pro's OLED PWM problem

https://macanorak.com/the-macbook-pros-oled-pwm-problem-when-a-better-display-becomes-worse-to-use/
416•MacAnorak•11h ago•231 comments

Handwriting, as a Font

https://ductus.sorenblank.com
115•coinfused•5h ago•29 comments

Claude Code and Codex reverse engineered my 1994 Jeep's ECU over a $3 Arduino [video]

https://www.youtube.com/watch?v=WPj3aQ-xxpI
330•envoked•3d ago•163 comments

Show HN: The Last Complication, a Zachtronics-style watchmaking puzzle game

https://lastcomplication.com/
31•tdeangelis•2h ago•5 comments

Microsoft 365 outage leads to chlorine gas poisoning of 15 hotel guests

https://www1.wdr.de/nrw/rheinland/rhein-erft-kreis/technischer-defekt-chlor-phantasialand-100.html
380•DeepLogin•6h ago•181 comments

1900s How do you Wash a Kimono?

https://www.oldphotosjapan.com/photos/877/laundry-washing-clothes-meiji-19th-century-vintage-albu...
8•Matrixik•2d ago•0 comments

Things That Turbo Pascal Is Smaller Than (2011)

https://prog21.dadgum.com/116.html
196•ksec•11h ago•103 comments

Eclipse Mosquitto – An open source MQTT broker

https://github.com/eclipse-mosquitto/mosquitto
61•gregsadetsky•6h ago•25 comments

Understanding Positional Embeddings in Modern LLMs

https://magazine.sebastianraschka.com/p/understanding-positional-embeddings
199•vismit2000•12h ago•27 comments

Aspidistra: A British wartime radio transmitter used for MitM attacks

https://en.wikipedia.org/wiki/Aspidistra_(transmitter)
25•BoxOfRain•3d ago•3 comments

Emacs Is My Browser

https://joshblais.com/blog/emacs-as-my-browser/
176•nyc111•9h ago•94 comments

A quest to create the perfect whiskey glass

https://www.nytimes.com/2026/11/13/dining/drinks/kurt-josef-zalto-whiskey-glass.html
47•Kaibeezy•2d ago•46 comments

Ask HN: Stripe closed us after one $1,500 invoice and never asked for evidence

150•snirgbs•5h ago•54 comments

Vulcan rocket fails during return-to-flight launch, 40 Amazon satellites lost

https://www.space.com/space-exploration/launches-spacecraft/ula-vulcan-centaur-return-to-flight-a...
104•jonbaer•4h ago•53 comments

Clice v1.0 Released: Welcome to Modern C++

https://ykiko.me/en/articles/2105384023094834022/
97•molenzwiebel•7h ago•64 comments

Why did some Apollo missions carry a grenade launcher? (2019)

https://space.stackexchange.com/questions/37775
228•Skyfall106•16h ago•146 comments

Jensen Huang might return to Nvidia as CEO

https://www.reuters.com/world/china/jensen-huang-discusses-comeback-nvidia-ceo-with-allies-ft-rep...
106•YakubYilmaz•1h ago•88 comments

Notes on Mathematics for Engineers (1959) [pdf]

https://websites.umich.edu/~ners311/CourseLibrary/MathNotes.pdf
70•the-mitr•4d ago•20 comments

Keylogger Detection and Containment [pdf]

https://www.cs.vu.nl/~ast/Theses/ortolani-thesis.pdf
17•movsx•2d ago•2 comments

Eval Awareness vs. Eval Paranoia

https://www.lesswrong.com/posts/mepsi7JNKYejq9wry/eval-awareness-vs-eval-paranoia
46•ezrasu•3d ago•21 comments

All AI-generated code should be Open Source

https://jakub.jankiewicz.org/blog/ai-code-open-source/
1073•jcubic•10h ago•583 comments

A DrRacket style editor for Clojure with some Lisp Machine inspired features

https://jonquil.b12n.app/
12•yogthos•2d ago•1 comments
Open in hackernews

Garbage collection of object storage at scale

https://www.warpstream.com/blog/taking-out-the-trash-garbage-collection-of-object-storage-at-massive-scale
96•ko_pivot•1y ago

Comments

juancn•1y ago
Another possible mechanism for doing GC at scale (a variation on Asynchronous Reconciliation in the article) in some file/object store, is doing a probabilistic mark and sweep using bloom filters.

The mark phase can be done in parallel building many bloom filters for the files/objects found.

Then the bloom filters are merged (or'ed together essentially) and then a parallel sweep phase can use the bloom filter to answer the question: is this file/object live?

The bloom filter then answers either "No" with 100% certainty or "Maybe" with some probability p that depends on the parameters used for the bitset and the hash function family.

cogman10•1y ago
What does the bloom filter solve?

The expensive portion of the mark and sweep for the object store is the mark phase, not the storage of what's been marked. 100s, 1000s, or even millions of live objects wouldn't hardly take any space to keep in a remembered set.

On the other hand, querying the S3 bucket to list those 1M objects would be expensive no matter how you store the results.

But this does tickle my brain. Perhaps something akin to the generational hypotheses can be applied? Maybe it's the case that very old, very young, or very untouched objects are more likely to be garbage than not. If there's some way to divide the objects up and only look at objects whose are in "probably need to be collected" regions, then you could do minor fast sweeps semi frequently and schedule more expensive "really delete untracked stuff" infrequently.

Cicero22•1y ago
I was thinking they could use something like cloudwatch events, or something, to trigger sweeps and significantly reduce scheduled sweeps.

They could even use cost allocation tags to predict if a bucket or group of buckets should be scanned if it's growing unexpectedly. Cost isn't a perfect metric but there's definitely signal there.

juancn•1y ago
Building the set of used files or objects (which is what mark does in a mark/sweep).

Sometimes it's too expensive to mark in place, even if it's a bit that you need to write to disk and keeping a set of references may be prohibitive (or the structure holding the references is mostly/effectively immutable).

If it's all memory and mutable it doesn't (normally) really matter, but when it's not, you ideally would have some mechanism to move the code to where the data is, rather than stream the data to where the compute is (it is really wasteful for large scale data processing).

In any case, you would not be moving/scanning the files themselves, but the metadata is what you want to read for the mark phase.

The article if I understood correctly implies that the files and the metadata of the files (Kafka queues and so on) are separate, so presumably, the metadata is much much smaller than the data itself, but still potentially large.

For example if you had a large scale content addressed store (think a massive version of git's blob storage), you typically add to something like that and keep a few mutable root references to start your GC from to seed a mark/sweep.

Following the git example, the roots would be the branches, tags and reflogs, and the metadata you scan the transitive closur of the trees that are reachable from those (simplifying a bit) but not the file blobs themselves.

I use git as an example because a a CAS lends itself very well to large scale distributed systems because you can reason about it as an immutable data structure, but you can still change it effectively with sane semantics.

donavanm•1y ago
If you like big beautiful storage and probabilistic structures check out https://www.usenix.org/conference/osdi14/technical-sessions/.... The coho data folks ended up in AWS S3 a few years later.
juancn•1y ago
Thanks! I hadn't seen it and it may come handy!
deathanatos•1y ago
> Why Not Just Use a Bucket Policy?

I've heard these words so many times, it's refreshing to see someone dig into why bucket policies aren't a cure-all.

As for "Why not use synchronous deletion?" — regarding the pitfall there, what about a WAL? I.e., you WAL the deletions you want to perform into an object in the object store, perform the deletions, and then delete the WAL. If you crash and find a WAL file, you repeat the delete commands contained in the WAL.

(I've used this to handle this problem where some of the deletions are mixed: i.e., some in an object store, some in a SQL DB, etc. The object store is essentially being used as strongly consistent storage.)

(Perhaps this is essentially the same as your "delayed queue"? All I've got is an object store though, not a queue, and it's pretty useful hammer.)

telotortium•1y ago
> HN Disclaimer: WarpStream sells a drop-in replacement for Apache Kafka built directly on-top of object storage.

First time I’ve seen one of these. That’s actually a better way to advertise your product than putting it at the end.

hencq•1y ago
Yes, though I think they meant to say disclosure instead of disclaimer.
siscia•1y ago
What I see working extremely well, arguably in a setting where cost was not really an issue was a much simpler approach.

Keep compacting files at some regular cadence `t` and keep a bucket policy to delete files older than `t+delta+buffer`.

Then have an alarm for files older than `t+buffer`