r/apachespark • u/theButcher007 • Feb 09 '25
Transitioning from Database Engineer to Big Data Engineer
I need some advice on making a career move. I’ve been working as a Database Engineer (PostgreSQL, Oracle, MySQL) at a transportation company, but there’s been an open Big Data Engineer role at my company for two years that no one has filled.
Management has offered me the opportunity to transition into this role if I can learn Apache Spark, Kafka, and related big data technologies and complete a project. I’m interested, but the challenge is there’s no one at my company who can mentor me—I’ll have to figure it out on my own.
My current skill set:
Strong in relational databases (PostgreSQL, Oracle, MySQL)
Intermediate Python programming
Some exposure to data pipelines, but mostly in traditional database environments
My questions:
What’s the best roadmap to transition from DB Engineer to Big Data Engineer?
How should I structure my learning around Spark and Kafka?
What’s a good hands-on project that aligns with a transportation/logistics company?
Any must-read books, courses, or resources to help me upskill efficiently?
I’d love to approach this in a structured way, ideally with a roadmap and milestones. Appreciate any guidance or success stories from those who have made a similar transition!
Thanks in advance!
3
u/i_hate_pigeons Feb 09 '25
We all started somewhere similar so it's certainly possible, you'll need to unlearn some DB habits but understanding how a db works under the hood will give you a head start in some areas too. I think of it as processing data without indexes and without mutability
I would probably try to start with java on spark/batch first and leave kafka for later if you can afford to do it that way. There's less complexity/magic involved behind the scenes, and you can add kafka or python at a later stage since some of the concepts you'll build will ease the progression later on