Avatar for MARICI Solutions
Provides IT & management consulting, data analytics, & training services

AWS Data Engineer

Posted: 3 weeks ago
Job Location
Remote Work Policy

In office - WFH flexibility

Visa Sponsorship

Not Available

RelocationAllowed
Skills
Python
SQL
MongoDB
Swagger
XML
JSON
Amazon S3
Oracle
Flask
DynamoDB
Pandas
Amazon CloudWatch
HDFS
Docker
OAuth2
Jwt
Apache Spark
Parquet
Kubernetes
DataDog
Apache Kafka
Avro
Amazon EMR
Pyspark
AWS Lambda
Confluent
GPT
AWS Step Functions
OpenAPI
Boto3
FastAPI
AWS Glue
Amazon EKS
Amazon EventBridge
Requests
Amazon MSK
Apache Iceberg
GitHub Copilot
Orc
AWS Glue Data Catalog
Awswrangler

About the job

: AWS Data & Solution EngineerLocation:** New York City, NYWork Mode:** Hybrid 2 3 days onsite per weekExperience:** 10+ Years## Job SummaryWe are seeking a Senior AWS Data & Solution Engineer with strong hands-on expertise in Python, PySpark, and AWS cloud services to design, develop, and optimize enterprise-scale data solutions for a Fortune 500 client. The role focuses on building an enterprise data lake on AWS, developing scalable data pipelines, implementing event-driven integrations, and delivering secure REST APIs. The ideal candidate will have experience with Apache Iceberg, Kafka, AWS Glue, and agentic AI workflows, along with the technical leadership skills needed to drive complex engineering initiatives.## Key Responsibilities Design, develop, unit test, and maintain scalable data processing applications using Python, PySpark, and Apache Spark. Build data pipelines integrating relational databases such as Oracle, NoSQL databases such as DynamoDB and MongoDB, and data sources including Amazon S3 and HDFS. Develop AWS Lambda functions using Python libraries such as awswrangler, pandas, boto3, json, and requests. Build and optimize ETL pipelines using AWS Glue, AWS Glue Data Catalog, and Amazon EMR. Implement event-driven data integrations using Apache Kafka, Amazon MSK, and Confluent libraries. Use Apache Iceberg to manage schema evolution, change data capture (CDC), ACID-compliant data merges, and snapshot management. Design, develop, and secure RESTful APIs using FastAPI or Flask, including OAuth2/JWT authentication and Swagger/OpenAPI specifications. Implement workflow orchestration using AWS Step Functions and Amazon EventBridge. Optimize Spark workloads, SQL queries, data access patterns, and storage formats such as Avro, Parquet, ORC, JSON, and XML. Configure AWS Glue crawlers and catalog tables, collections, and objects from Oracle, MongoDB, and Amazon S3. Containerize applications using Docker and deploy them to Kubernetes-based environments, including Amazon EKS. Build and leverage agentic AI workflows and use AI-assisted development tools such as GitHub Copilot or GPT-based coding assistants. Monitor, troubleshoot, and resolve application and pipeline failures using Amazon CloudWatch and Datadog. Diagnose production issues, perform root-cause analysis, and implement proactive performance and reliability improvements. Participate in code reviews, release management, production deployments, and Agile/Scrum ceremonies. Prepare technical documentation, deployment guides, runbooks, and user support materials. Coordinate with onsite and offshore teams, communicate technical designs, and mentor engineers on complex development scenarios.## Required Technical Skills **Experience:** 10+ years in software engineering, data engineering, or related technical roles. **Programming:** Strong hands-on expertise in Python, PySpark, pandas, boto3, and requests. **AWS Services:** Amazon S3, AWS Glue ETL, Glue Data Catalog, AWS Lambda, Amazon EventBridge, AWS Step Functions, Amazon Athena, and Amazon EMR. **Data Engineering:** Enterprise data lakes, ETL/ELT pipelines, distributed processing, data integration, and pipeline optimization. **Messaging & Streaming:** Hands-on experience with Apache Kafka, Amazon MSK, and event-driven architectures. **Data Formats:** Practical experience with Avro, Parquet, ORC, JSON, and XML, including compression and performance optimization. **Databases & SQL:** Strong SQL skills, including complex queries, with experience in relational and NoSQL databases. **API Development:** Experience building REST APIs using FastAPI or Flask, including OAuth2/JWT security and Swagger/OpenAPI. **Apache Iceberg:** Experience with Iceberg tables, CDC merges, schema evolution, and snapshots. **AI-Assisted Development:** Hands-on experience building agentic AI workflows and using Copilot/GPT-based tools for development. **Containers:** Experience with Docker and Kubernetes, preferably Amazon EKS. **Monitoring:** Experience with CloudWatch, Datadog, troubleshooting, and production support. **Certification:** AWS certification in Solutions Architecture, Data Engineering, or Data Analytics is required.## Preferred Qualifications Experience with Snowflake, Amazon Redshift, and DynamoDB. Experience designing enterprise-scale cloud data platforms for large organizations. Strong understanding of application reliability, security, scalability, and cost optimization. Ability to translate business requirements into technical designs and deliver production-ready solutions. Experience leading technical discussions, coordinating distributed teams, conducting code reviews, and mentoring engineers. Strong communication, analytical, problem-solving, and stakeholder-management skills.

Similar Jobs

Stochastic company logo
Stochastic
Intelligent interface that finally adapts to you
Coast company logo
Coast
Coast is re-imagining B2B payments, beginning with fleet and fuel
Kick Health company logo
Kick Health
The Online Performance Medicine Clinic for Energizing Sleep and Confident Presentations
AMPAworks  company logo
AMPAworks
AI applications and devices that drive efficiency and cost savings in hospitals
Super company logo
Super
Super is the AI assistant for property management
Rocket Money company logo
Rocket Money
The Money App That Works for You