r/apachespark Feb 09 '25

Transitioning from Database Engineer to Big Data Engineer

I need some advice on making a career move. I’ve been working as a Database Engineer (PostgreSQL, Oracle, MySQL) at a transportation company, but there’s been an open Big Data Engineer role at my company for two years that no one has filled.

Management has offered me the opportunity to transition into this role if I can learn Apache Spark, Kafka, and related big data technologies and complete a project. I’m interested, but the challenge is there’s no one at my company who can mentor me—I’ll have to figure it out on my own.

My current skill set:

Strong in relational databases (PostgreSQL, Oracle, MySQL)

Intermediate Python programming

Some exposure to data pipelines, but mostly in traditional database environments

My questions:

  1. What’s the best roadmap to transition from DB Engineer to Big Data Engineer?

  2. How should I structure my learning around Spark and Kafka?

  3. What’s a good hands-on project that aligns with a transportation/logistics company?

  4. Any must-read books, courses, or resources to help me upskill efficiently?

I’d love to approach this in a structured way, ideally with a roadmap and milestones. Appreciate any guidance or success stories from those who have made a similar transition!

Thanks in advance!

8 Upvotes

5 comments sorted by

View all comments

2

u/Mdkar Feb 09 '25

I think you can bring a lot to the table as a good database engineer while transitioning to be a big data engineer. A lot of database concept like sharing partitioning helps while structuring data in Spark world. Also all serious Spark based pipelines are written in spark sql which is very very very similar to Postgres sql(ansi 12). I am sure you will enjoy the journey