Senior Data Engineer (PySpark / SQL)
Job Description
We are seeking a Senior Data Engineer with deep PySpark, SQL and AWS experience for a remote contract (C2C), with an optional hybrid arrangement in Erie, PA. You will design and build data integration and architecture on a modern AWS lakehouse stack, including Glue, Iceberg, Snowflake, Redshift and Athena, while driving metadata management, data quality and engineering best practices. The interview process is four virtual rounds. The ideal candidate will have 10–15 years of data engineering or data architecture experience. Petabyte-scale and insurance industry experience are strong pluses, as is experience providing technical leadership and mentoring other engineers. Requirements • 10–15 years of data engineering / data architecture experience • Strong hands-on PySpark, Python and SQL • AWS experience (S3, IAM, VPC), including AWS Glue / Glue Catalog and AWS streaming services • Experience with Apache Iceberg, Snowflake, Redshift and Athena • Experience with Talend and dbt Core • Experience with Splunk, Git and CI/CD • Strong ETL/ELT, data integration, metadata management and data quality skills • Preferred: petabyte-scale data environments, insurance industry experience, and technical leadership and mentoring • Any work authorization except OPT/CPT
Responsibilities
- Design and build scalable ETL/ELT pipelines using PySpark and SQL
- Develop data integration solutions on AWS Glue and Glue Catalog
- Build and maintain lakehouse tables with Apache Iceberg
- Model and deliver data across Snowflake, Redshift and Athena
- Develop transformations with dbt Core and Talend
- Implement streaming ingestion using AWS streaming services
- Drive metadata management and data quality practices
- Contribute to overall data architecture decisions
- Manage code through Git and CI/CD pipelines
- Provide technical leadership and mentor engineers