Right now, it’s an early MVP — it works, but still has a few bugs and missing features (like Database catalog, API catalog, pipeline merging, Existing DLT script onboarding, etc). I built it because DLT is quite powerful Data pipeline tools where we develop the scripts for pipeline creation, this client idea is to provide an easy way not only for creating, but also scheduling, monitoring, orchestrating, versioning, and more. Also it adds an AI layer to our pipeline management flow.
You can check it out here: https://github.com/nbernardo/dlt-client demo: 1. Creating a basic pipeline: https://lnkd.in/eZw-isTV 2. Scheduling a job for pipeline: https://lnkd.in/eNjsuWT3 3. Pipeline versioning, Data transformation and File system sourceing: https://lnkd.in/eazgrTrM
I’d love feedback, suggestions, or even contributions (ideas, code, etc) if anyone’s interested.
Thanks!
Nakassony Bernardo
mucio•3mo ago
One of the reasons why I like dlt is because I can do everything via code, which makes things more maintanable, for me. I think this could be a starting point for a dlt SaSS, but there are a lot of things to consider and connectors change all the time. It's something I thinking about a lot... :)
I think you will get more feedback in the dlt slack/community.
nakassony•3mo ago