Ask HN: Anyone want my realtime-snapshots dataset of Hacker News?
so I've been maintaining a project [Track HN](https://track-hacker-news.com) since 2023. it continuously takes snapshots of Hacker News, recording how the score/rank/comments of stories/comments change over time. I think this is a unique HN dataset in its own way.
I don't have enough time to maintain this project anymore and wondering if anyone would find the data useful and want to take it over and keep it alive.
## the dataset
the dataset is over 200M rows in a postgres instance, plus a clickhouse instance to speedup the analytics.
- stories/comments/users
- ~6.38M Stories, ~43.2M comments, ~1.15M users, since the epoch (2006-10-09)
- story score/comment count history - ~21.5M snapshots for ~900K stories, since 2024-03-22
- story rank history - ~145M snapshots for ~178,120 stories, since 2024-03-28
- http status of popular posts
## the product and stack
I've build a product around this dataset with a stack:
- a web app (with rails) to crawl and serve the data;
- dashboards with metabase;
- another set of dashboards with retool (retool kindly sponsored a postgres instance back then);
- a browser extension;
- few data reports;
## ideas
things you could look at from the data, for example:
- analyze activities of specific HN user
- see how stories moving through the front page
- find patterns between early score and eventual score
- see how long stories survive at different ranks
- find posting time/weekday effects, domain/user patterns,
etc.
- detect/extract topics/keywords from comments for brands/products/languages, etc.
Let me know if you're interested, send your proposal/questions/cool ideas to me#nami.land. thanks!
namiwang•45m ago
so I've been maintaining a project [Track HN](https://track-hacker-news.com) since 2023. it continuously takes snapshots of Hacker News, recording how the score/rank/comments of stories/comments change over time. I think this is a unique HN dataset in its own way.
I don't have enough time to maintain this project anymore and wondering if anyone would find the data useful and want to take it over and keep it alive.
## the dataset
the dataset is over 200M rows in a postgres instance, plus a clickhouse instance to speedup the analytics.
- stories/comments/users
- ~6.38M Stories, ~43.2M comments, ~1.15M users, since the epoch (2006-10-09)
- story score/comment count history - ~21.5M snapshots for ~900K stories, since 2024-03-22
- story rank history - ~145M snapshots for ~178,120 stories, since 2024-03-28
- http status of popular posts
## the product and stack
I've build a product around this dataset with a stack:
- a web app (with rails) to crawl and serve the data;
- dashboards with metabase;
- another set of dashboards with retool (retool kindly sponsored a postgres instance back then);
- a browser extension;
- few data reports;
## ideas
things you could look at from the data, for example:
- analyze activities of specific HN user
- see how stories moving through the front page
- find patterns between early score and eventual score
- see how long stories survive at different ranks
- find posting time/weekday effects, domain/user patterns, etc.
- detect/extract topics/keywords from comments for brands/products/languages, etc.
Let me know if you're interested, send your proposal/questions/cool ideas to me#nami.land. thanks!