I know that job titles varies from company to company, so the title of this post could be interpreted differently.
For my current job, I am a Data Engineer. I work on building batch ETL systems in Python and Airflow. And these systems write into a data warehouse that is built in AWS Redshift. So, I feel like a DW Engineer and an ETL dev. But honestly, I want to get more into concurrent systems, real time data pipelines, distributed computing, and big data. I feel like the big data engineering world is still using Java and Scala to do most of this work, but I have no real experience on big data systems in production. I would like to get some advice.
I love learning new programming languages, and I really want to learn Scala. Especially because I love the elegant programming principles that come with functional programming.
tl;dr Should I learn Scala for Big Data Engineering? Or since I’m most interested in the functional part of Scala, should I just learn how to write Python in a more functional way?