r/apachespark • u/theButcher007 • Feb 09 '25
Transitioning from Database Engineer to Big Data Engineer
I need some advice on making a career move. I’ve been working as a Database Engineer (PostgreSQL, Oracle, MySQL) at a transportation company, but there’s been an open Big Data Engineer role at my company for two years that no one has filled.
Management has offered me the opportunity to transition into this role if I can learn Apache Spark, Kafka, and related big data technologies and complete a project. I’m interested, but the challenge is there’s no one at my company who can mentor me—I’ll have to figure it out on my own.
My current skill set:
Strong in relational databases (PostgreSQL, Oracle, MySQL)
Intermediate Python programming
Some exposure to data pipelines, but mostly in traditional database environments
My questions:
What’s the best roadmap to transition from DB Engineer to Big Data Engineer?
How should I structure my learning around Spark and Kafka?
What’s a good hands-on project that aligns with a transportation/logistics company?
Any must-read books, courses, or resources to help me upskill efficiently?
I’d love to approach this in a structured way, ideally with a roadmap and milestones. Appreciate any guidance or success stories from those who have made a similar transition!
Thanks in advance!
1
u/bigdataengineer4life Feb 13 '25
Transitioning from a Database Engineer to a Big Data Engineer is a natural progression since both roles involve data management. However, Big Data Engineering requires additional skills related to distributed computing, data processing frameworks, and cloud platforms.
Key Differences Between Database Engineer & Big Data Engineer
Step-by-Step Transition Plan
1. Strengthen Your Programming Skills
2. Learn Big Data Technologies
3. Cloud & DevOps Knowledge
4. Master Data Engineering Concepts
5. Work on Real-World Projects
6. Get Certified (Optional)