I am currently on a high visibility project at my company involving collaboration of many data scientists and data engineers in a non-purely engineering domain.
I am a non-senior data engineer supporting this project and i have a lot of thoughts about the direction of it, how things could be improved, and where we are going wrong. however, i don't have the space or title to be heard out. data scientists feel like they see the platform team as an enemy rather than a power multiplier. additionally, the senior platform engineer supporting this project is spread thin across many projects and doesn't have their hands on the data. yet, they are directing us on how to work on this project. i get very annoyed working with these people as i see and know what the project needs, but instead i am just instructed to follow the lead of a senior engineer who is not really in touch with the needs of the project but thinks they are.
currently we use a complex pipeline which i was instructed to build. i am the most in touch with the underlying data and intimately understand the needs on both the platform and DS side despite not receiving super concrete requirements from the side of the "scientists."
i would like to implement some visionary functionality, high level architectural differences in parallel with current development/platform support work, and then one day say something like "hey, i built a copy of what we are doing but faster, better, etc. here is the proof and let me know what you think." however, it would basically automate the jobs of the people on the platform side, making enemies with at least a few people and especially the lead engineer who makes these calls which i disagree with. at least, i would like the stakeholders to see how our current approach is hindering results and how i have a vision for a better one. however, it's not my place or my title to do so.
i want to just show a director or our manager that i can fix this project without doing it in a hostile way. but i don't see how to do this given the organizational structure/conditions. would it be better to leave and go to another company in hopes of a better atmosphere where i can be heard out for my ideas?
Stuck, Data Engineer
benoau•5mo ago
jamboca•5mo ago
pavel_lishin•5mo ago
It sounds like a bad culture fit for you, to be honest. Nothing wrong with looking for another job; you're not likely to be able to steer this ship around, if nobody is interested in supporting you.
panny•5mo ago
Want to really be a senior? Think this way: Everything I build as a senior developer is an attempt to make myself obsolete. I should be able to give a two weeks notice and walk away with no harm to the company. When I build things at work, I never worry about job security, not even a little. If there is something I built that they can't support without me, I've failed.
And you will ultimately know you failed when you can't even go on vacation without carrying the company laptop with you. I know you're proud of that thing you built and how you're the only one who knows how it all works, but that's the core problem.
benoau•5mo ago
If the project is not failing, talk about how you would have done things differently in your next one-on-one and make it be known you'd like to be part of the planning next time.
It's normal to wish things were tailored more to your preference, it's normal to look back on how something was done and see a better way it could have been done. If you have retrospectives that's another good place to talk about these thoughts.
JojoFatsani•5mo ago
The way to effect this change is to psy-op your teammates/bosses into thinking they came up with it.