data engineer for healthcare and insurance data

The role builds automated data ingestion pipelines and scraping mechanisms on Azure Databricks to support an enterprise AI platform and semantic layer.

• Build high-throughput data ingestion pipelines and scraping engines using Python, PySpark, and Databricks Auto Loader to collect unstructured documents, external registry feeds, and enterprise data
• Structure and expose ingested data in the Databricks semantic layer using Open Knowledge Format standards for consumption by AI Search, Databricks Genie Spaces, and LLM reasoning engines
• Automate ingestion pipeline deployments, scraping job schedules, and code releases using Azure DevOps and Databricks Asset Bundles (DABs)
• Scrape and ingest healthcare and insurance data, including regulatory databases, public registries, and unstructured medical and claims files, to support AI agent workflows
• Implement schema enforcement, automated ingestion failure alerts, and job telemetry in Databricks Workflows and Azure DevOps

• Extensive experience (5+ years) building high-volume data ingestion pipelines using Azure Databricks, PySpark, and Auto Loader
• Hands-on experience developing data scraping routines and extraction engines for complex, unstructured data sources
• Proven ability to structure ingested datasets using Open Knowledge Format within Databricks semantic layers and AI Search for downstream GenAI consumption
• Track record of managing Databricks ingestion workflows and deployment pipelines using Azure DevOps and Databricks Asset Bundles (DABs)
• English proficiency at B2 level or higher

• Условий в вакансии нет