описание
We are looking for an experienced Principal Data Engineer to join our growing team building our strategic data platform as we look to adopt best in class data governance and democratise data around the business. The successful candidate will play a pivotal role in shaping and executing our data strategy, leveraging our data assets to the full, elevating our decision-making, streamlining processes, and drive innovation within our organisation.
задачи
- Contribute to the solution design and implementation of a robust and scalable data platform using Snowflake with complementary tooling to meet the organisation's needs
- Help evolve the Insight Data Strategy, communicate outcomes from a leadership through to team level
- Set the firm-wide standards and conventions for ingesting, governing, transforming, persisting, and democratising data
- Provide expertise in ELT pipeline performance optimisation
- Enable the adoption of advanced analytics & AI by configuring best-in-class tooling
- Evolve the usage of infrastructure-as-code in collaboration with cloud engineering
- Align technical designs with internal data, privacy, security and compliance policies
- Stay abreast of industry trends and emerging technologies related to Snowflake and big data processing, recommending relevant advancements for consideration
требования
- Snowflake (Warehouses, Tasks/Streams, Snowpipe, Snowpark, Distributed UDF/Sproc, Time Travel, Cloning)
- Demonstratable experience of delivering end to end pipelines in Snowpark (transferable frameworks such as Pyspark, Polars or Dask are also welcome)
- DBT including Jinja templating or equivalent transformation frameworks
- Python including build tools, test automation frameworks and engineering best practices
- SQL (including Snowflake-optimised SQL)
- Experience with CI/CD pipelines (GitHub Actions / TeamCity / Artifactory)
- Demonstrate initiative and independence to solve problems.
- Be able to communicate abstract or complex ideas or solutions effectively.
- Stand by software engineering principles around quality and doing things the right way.
- Experience working in and with Agile teams performing iterative development.
- Ability to collaborate with end users, stakeholders, teams, and team members.
- Have a constant desire to improve knowledge and drive adoption of new tech or approaches within the teams.
- Own mistakes and work with your colleagues to redress them, learn from them and make improvements for the long-term
- Будет плюсом: Docker; Java / Scala; Event streaming technologies such as Kafka, Pulsar, or Azure Event Hub; Integration with Collibra; Apache Iceberg; Apache Arrow; Pyspark; AI engineering (Snowflake AI, Langchain, Llamaindex, Pinecone, etc)
условия
- Условий в вакансии нет