Data engineer

We are looking for a Senior Data Engineer to join a team developing data ingestion processes for a data warehouse, a Big Data environment, the Google Cloud platform, and campaign systems. We build solutions that process large volumes of data in batch and near-real time. Our data is used to create data marts, analyses, reports, customer segments, and personalized offers and campaigns. If you want to build reliable data pipelines in a hybrid and cloud environment, we invite you to get in touch!

• Designing, building, and developing ETL/ELT pipelines operating in batch and streaming modes.
• Creating data ingestion processes for BigQuery, data warehouses, a Data Lake, and a campaign platform.
• Integrating data from on-premise systems, the GCP cloud, databases, files, APIs, and event systems.
• Designing logical and physical data models and preparing data layers and subject-specific data marts.
• Creating and optimizing SQL queries in relational and columnar databases.
• Programming data processing and loading processes in Python and/or Rust.
• Building event flows using Kafka or Pub/Sub.
• Ensuring data quality, completeness, consistency, and timeliness.
• Optimizing process performance and processing costs in GCP.
• Implementing monitoring, logging, error handling, and processing retry mechanisms.
• Translating business requirements into technical concepts, data models, and ready-to-use solutions.
• Testing, supporting deployments, diagnosing incidents, and maintaining solutions.
• Creating technical documentation and developing solutions in line with the DevSecOps approach.
• Independence, accountability, and consistency in completing tasks.
• Attention to the quality, reliability, and maintainability of solutions.
• Good work organization and the ability to acquire knowledge quickly.
• The ability to collaborate effectively within a team.
• Experience working with business clients.
• Good communication skills and an analytical approach to problems.

• Senior-level experience designing and building ETL/ELT processes and data pipelines.
• Good knowledge of SQL, including query optimization and working with large volumes of data.
• Practical knowledge of BigQuery: designing data structures, partitioning, clustering, and optimizing performance and costs.
• Experience programming in Python and/or Rust.
• Knowledge of batch and streaming data processing.
• Knowledge of event-driven architecture (Kafka or Pub/Sub).
• Experience designing logical and physical data models.
• Knowledge of relational databases, e.g. Oracle or PostgreSQL.
• Knowledge of Linux sufficient to work independently and diagnose problems.
• Ability to analyze business requirements and translate them into technical solutions.
• Будет плюсом: Experience with Google Cloud services: BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage; Knowledge of Apache Spark (especially on Dataproc) and Apache Beam / Dataflow; Experience with Kubernetes, Docker, and CI/CD; Knowledge of Informatica PowerCenter or other ETL tools; Experience building a Data Lake, data warehouse, or data platform in a hybrid environment; Knowledge of campaign systems, customer segmentation, or marketing automation; Experience testing data quality, monitoring pipelines, and data observability; Knowledge of data security, access control, anonymization, and sensitive data processing; Knowledge of AutomateNOW

• B2B cooperation.
• Rate: 160 PLN/hour net.
• Cooperation period: over 12 months.