Whether we like it or not – we are adding dependencies sooner or later. And it's arguably good practice to check a new dependency beforehand: Is it maintained? By whom? How many issues does it have and how many of those are bugs? Are they being fixed? What's on the roadmap? What's the release frequency and how often do APIs break?

One of our favorite solutions that already exist to answer such questions is the OpenSSF Scorecard project (https://github.com/ossf/scorecard) – we use this ourselves and can only recommend it.

We built shouldiuse.dev around it to make results accessible as a website, and used the opportunity to dive deep into heavily LLM-assisted coding for the first time in a professional project.

Three people (devs and non-devs) each started vibe-coding an initial prototypes, one using v0, one using lovable and one using Cursor. At first blown away by how fast we were able to generate these and how great there were looking, we soon ran into issues merging different ideas as there were multiple different web frameworks and versions flying around. The most work on the frontend definitely went into getting the details and small adaptions right.

In parallel, on the backend we started to write a Go application that uses the ossf/scorecard library to do a lot of the checks we want. To also play around with AI on that end, we intentionally made heavy use of Copilot and tried around with different models and prompts. We also added more metrics to the dependency check that we gather via GitHub API, and finally generate textual summaries via OpenAI.

The Prompt to generate a final textual recommendation consists of:

* A header stating the role, capabilities and limitations, and the expected response format (JSON and no lists/bullet points) – We also tell it to be critical, objective and give short and concise answers. * The result of the scorecard check * Additional community-related data * The questions that are being shown in the FAQ section – The answers to those are also generated by the LLM.

Since such a check involves heavy use of the GitHub API, we require users to input a GitHub personal access token when requesting a check. The first time a repository is checked on shouldiuse.dev it will take a few seconds, but then the results are stored in a postgres for faster retrieval later on.

For now it only works for public GitHub repos, but we might add other platforms if there is interest.

We also added a remote MCP server with built-in authentication, so you can directly access shouldiuse from your IDE and automatically check new dependencies anytime a coding assistant introduces one to ensure that only safe dependencies are added to the project.

What started as a fun internal experiment quickly surprised us with how useful it turned out to be. We didn’t plan to release it publicly, but we think might be useful for other devs and therefore we wanted to share it here. Any feedback is welcome!