devops engineer for a data lake

EPAM provides enterprise software products, open source solutions, accelerators, and technology services. The role contributes to a project for a large customer in the e-commerce and fashion industry, supporting the customer’s analytical data platform and central data lake.

• Troubleshoot user issues related to Databricks (Spark) and suggest optimizations for long-running or resource-intensive jobs;
• Guide users on best practices for Databricks cluster management;
• Consult and help diverse teams develop, implement, and maintain sustainable, high-performance, growth-ready data-processing and data-integration systems;
• Implement services that improve the CI/CD experience through self-service;
• Work with other teams in the customer’s Data Infrastructure to deliver backbone services for the customer’s central data lake;
• Improve data access methods to provide a secure and compliant self-service platform;
• Guide and support users in finding cost-effective setups while maintaining high efficiency;
• Implement observability solutions in Databricks to reduce slack costs;
• Drive adoption of edge Databricks features such as Photon and Graviton instances;
• Review current infrastructure to identify cost-saving opportunities;
• Analyze, design, and develop solutions in an Agile team to support essential business needs for the platform search application.

• Proficient knowledge of distributed data-processing frameworks such as Apache Spark and relational database management systems;
• Strong hands-on production experience with Databricks;
• Practical knowledge of Python, Scala, and SQL;
• Hands-on production experience with AWS services, including IAM, S3, Lambda, and EC2;
• Expertise in software development processes, including testing, continuous integration/continuous delivery (CI/CD), monitoring, and documentation;
• Advanced English skills;
• Nice to have: AWS Networking or networking basics on AWS cloud, hands-on experience with Terraform.

• No conditions specified