Anveta, Inc

Clinical Data Lake Databricks Technical Lead/Solution Architect

Anveta, Inc United States
Anveta, Inc

Clinical Data Lake Databricks Technical Lead/Solution Architect

Anveta, Inc United States
2 months ago 36 applicants

See who Anveta, Inc has hired for this role

Project Duration -6+ Months

Location- - Remote

Day To Day Responsibilities

  • Experience in implementation of AWS Data Lake and Data Publication using Databricks, Airflow and AWS S3
  • 8+ Years of experience in Databricks Data Engineering to create Data Lake solutions using AWS services.
  • Knowledge of Databricks cluster and SQL warehouse, Experience in Delta and Parquet file handling
  • Experience in Data Engineering and Data Pipeline creation on Databricks
  • Experience in Data Build Tool (DBT) using Python and SQL
  • Extensive Experience in SQL, PL/SQL, complex Join, Aggregation function and DBT, Python, Data frames and Spark
  • Experience in Airflow for Job Orchestration, dependency Setup and Job scheduling
  • Knowledge of Databricks Unity Catalog and Consumption patterns
  • Knowledge of GitHub and CI/CD Pipelines, AWS Infra like IAM Role, Secrets and S3 buckets
  • Experience with life sciences is a must.

Must Haves

  • Responsible for defining technical architecture and application landscape.
  • Responsible for authoring SQL and Python scripts on Databricks and DBT (Data Build tool) to create data pipelines to create Operational Data Mart.
  • Responsible for creation of Data Pipelines for Data processing of Delta files into ODM format for downstream data consumption
  • Responsible for identifying data set relationship, join criteria and implement it in code for ODM model development.
  • Responsible for creation of Delta Lake for ODM model and setup of consumption pattern using Databricks Unity catalog
  • Responsible for creation of Airflow DAGs for job orchestration and scheduling of data pipeline jobs

Thanks & Regards

Amit Sinha

Recruitment Lead

Email: amit.sinha@anveta.com |

  • Seniority level

    Mid-Senior level

  • Employment type

    Contract

  • Job function

    Research, Analyst, and Information Technology

  • Industries

    IT Services and IT Consulting

Referrals increase your chances of interviewing at Anveta, Inc by 2x

See who you know

Get notified about new Clinical Data Specialist jobs in United States.

Sign in to create job alert

Similar jobs

People also viewed

Looking for a job?

Visit the Career Advice Hub to see tips on interviewing and resume writing.

View Career Advice Hub