инженер данных for Big Data

No description

• Write and optimize complex SQL queries;
• Develop and support Spark applications for processing large data volumes;
• Design, develop, and support data pipelines;
• Monitor and diagnose data pipeline performance;
• Interact with related technical and business teams.

• Confident SQL skills, including complex query writing and optimization, normalization, and denormalization;
• Practical Big Data experience with Spark / PySpark;
• Experience with the Hadoop ecosystem: Hadoop, YARN, Hive;
• Experience with data orchestrators such as Airflow or equivalents;
• Understanding of SQL query and Spark job optimization, including execution plans, joins, partitioning, and data skew;
• Experience with data storage formats such as Parquet, Avro, and/or equivalents;
• Understanding of Spark internals: Catalyst, Tungsten, CBO, AQE;
• Understanding of Data Quality principles and experience controlling data quality;
• Ability to work with related technical and business teams;
• Nice to have: Data Lake / Lakehouse experience with Delta Lake, Iceberg, ACID transactions, and time travel, deep Spark optimization, experience with multiple orchestrators and complex scalable data pipelines, monitoring and logging tools such as Prometheus, Grafana, ELK, and Grafana Loki, Data Quality tools and Data Governance processes, mentoring or participation in technical and architectural decisions.

• No conditions specified