Hi all, I'm a UX architect with a past as a web developer.
I started digging for dubby, minimal and ambient electronic records many years ago, and as you may imagine I literally spent hours on Discogs.
For a crate digger like myself the most powerful thing is the credits: which label a release is on, who engineered, mastered or remixed something. The connections between these things is what allows you to discover new stuff.
Doing that on Discogs is definitely possible, but it comes with a lot of friction.
Dub Digger is a personal, side project and it's my attempt at a tool to remove that friction. It's essentially the tool I would use to dig out records.
You search an artist, label or release and follow the credits from one name to the next.
I needed something focussed on a core of specific genres, so I built a curated database centered on techno, deep, ambient and experimental labels and artists, based on Discogs data.
Each artist and label has a relevance grade based on how close each entity is to the core of the database, from very high down to extremely low.
On top of that I added a manual curation layer to mitigate the issues and inconsistencies the automated rules were showing. I am also constantly adding manual overrides, to promote or demote specific artists I think should sit at a different level of relevance from where the raw Discogs data places them.
It currently has about 305k artists, 109k labels and 1M releases.
One interesting thing is that I tried to avoid using user data as much as possible and I did my best to make it truly accessible.
How I built it may be of interest
Source: Discogs' monthly XML dumps (CC0). The releases file is over 100 GB unpacked. No calls to the Discogs API to keep it light and with as few dependencies as possible.
Ingest: a Node/TypeScript script streams the XML, keeps only the fields the tool needs and goes through about 19M releases.
Two stages:
Stage 1 - it takes records tagged dub techno, deep techno, dub, ambient or minimal, with some rules added on top (for example dub only if tagged along with electronic , otherwise you get the whole reggae catalogue). An artist is in the core only if at least 10% of their releases have those tags.
Labels work in the same way. A label belongs to the core if at least half of the artists on it are core artists.
Stage 2 - one step out: releases with a core artist tagged on them, and releases on core labels. It stops there, as I was afraid a further step would pull in most of techno.
Precomputing: relevance grades, counts, rankings and the search index are all calculated offline. The output is one read-only SQLite file (about 1.3 GB).
Serving: Astro with server-side rendering, on a small nginx VPS, reading the SQLite file directly. No database server and no live API calls from the server. Covers and 30-second previews come from Apple: I match them offline with the iTunes Search API, and the browser loads them straight from Apple (the previews only when the user press play).
Updates: I re-run the pipeline on a new monthly dump and reupload the file.
Curation: The rules worked well only to an extent. The data were patchy so I had to add my personal curation layer. As a starting point I defined nine "traditions/influences" in the config, among them roots dub, reggae, Detroit techno, afrobeat, dubstep and UK garage. Belonging to one of them raises an artist to a fixed grade. All of this is paired with manual overrides.
Client: no accounts, cookies or tracking. “My List” uses localStorage. Aims for WCAG 2.1 AA ( I checked it myself, no certifications from any third parties).
It’s a beta and a passion project. And of course is far from perfect. I’d love some (kind, please!) feedback on how it feels to navigate.
ferrarosimone•35m ago
I started digging for dubby, minimal and ambient electronic records many years ago, and as you may imagine I literally spent hours on Discogs.
For a crate digger like myself the most powerful thing is the credits: which label a release is on, who engineered, mastered or remixed something. The connections between these things is what allows you to discover new stuff.
Doing that on Discogs is definitely possible, but it comes with a lot of friction.
Dub Digger is a personal, side project and it's my attempt at a tool to remove that friction. It's essentially the tool I would use to dig out records.
You search an artist, label or release and follow the credits from one name to the next. I needed something focussed on a core of specific genres, so I built a curated database centered on techno, deep, ambient and experimental labels and artists, based on Discogs data. Each artist and label has a relevance grade based on how close each entity is to the core of the database, from very high down to extremely low.
On top of that I added a manual curation layer to mitigate the issues and inconsistencies the automated rules were showing. I am also constantly adding manual overrides, to promote or demote specific artists I think should sit at a different level of relevance from where the raw Discogs data places them.
It currently has about 305k artists, 109k labels and 1M releases.
One interesting thing is that I tried to avoid using user data as much as possible and I did my best to make it truly accessible.
How I built it may be of interest
Source: Discogs' monthly XML dumps (CC0). The releases file is over 100 GB unpacked. No calls to the Discogs API to keep it light and with as few dependencies as possible.
Ingest: a Node/TypeScript script streams the XML, keeps only the fields the tool needs and goes through about 19M releases. Two stages: Stage 1 - it takes records tagged dub techno, deep techno, dub, ambient or minimal, with some rules added on top (for example dub only if tagged along with electronic , otherwise you get the whole reggae catalogue). An artist is in the core only if at least 10% of their releases have those tags. Labels work in the same way. A label belongs to the core if at least half of the artists on it are core artists. Stage 2 - one step out: releases with a core artist tagged on them, and releases on core labels. It stops there, as I was afraid a further step would pull in most of techno.
Precomputing: relevance grades, counts, rankings and the search index are all calculated offline. The output is one read-only SQLite file (about 1.3 GB).
Serving: Astro with server-side rendering, on a small nginx VPS, reading the SQLite file directly. No database server and no live API calls from the server. Covers and 30-second previews come from Apple: I match them offline with the iTunes Search API, and the browser loads them straight from Apple (the previews only when the user press play).
Updates: I re-run the pipeline on a new monthly dump and reupload the file.
Curation: The rules worked well only to an extent. The data were patchy so I had to add my personal curation layer. As a starting point I defined nine "traditions/influences" in the config, among them roots dub, reggae, Detroit techno, afrobeat, dubstep and UK garage. Belonging to one of them raises an artist to a fixed grade. All of this is paired with manual overrides.
Client: no accounts, cookies or tracking. “My List” uses localStorage. Aims for WCAG 2.1 AA ( I checked it myself, no certifications from any third parties).
It’s a beta and a passion project. And of course is far from perfect. I’d love some (kind, please!) feedback on how it feels to navigate.
Cheers