Job Description

Azure Data Engineer Databricks / PySpark / AI

Location: Chicago, IL Onsite

Experience: 9+ Years

Employment Type: Contract

Job Summary

We are looking for an experienced Azure Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, SQL, and Azure cloud technologies . The ideal candidate should have strong experience in data engineering, data warehousing, data modeling, and cloud migration projects.

One of the resources should also have exposure to AI-driven data pipelines, Machine Learning, and Recommendation Systems to support AI/ML initiatives.

Required Skills

• 9+ years of overall IT/industry experience with strong cloud technology experience.
• 8+ years of hands-on experience with:
  • Azure Databricks
  • PySpark
  • Azure SQL
  • SQL development
• Strong hands-on experience in Azure Databricks development .
• Strong expertise in PySpark and SQL development .
• Experience with Data Warehousing and Data Modeling concepts.
• Experience developing and integrating cloud-based data solutions .
• Strong understanding of ETL/ELT data pipelines and data engineering processes.
• Experience working on cloud migration projects , preferably Azure migration.
• Experience working in Agile methodology and offshore/onshore delivery models.
• Strong SQL skills, including complex queries, optimization, joins, CTEs, stored procedures, and performance tuning.
• Ability to gather requirements, understand business needs, and participate in technical/design discussions.
• Strong communication and collaboration skills.

AI / Machine Learning Preferred

At least one candidate should have experience with:

• AI-driven data pipelines
• Machine Learning data preparation and pipelines
• Recommendation Systems / Recommendation Engines
• Feature engineering and ML data processing
• Integration of ML models with data pipelines
• Exposure to Azure ML or similar cloud-based ML platforms is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark .
• Develop efficient SQL-based data solutions using Azure SQL.
• Build and maintain data warehouse and data lake solutions.
• Perform data transformation, cleansing, integration, and validation.
• Develop and optimize Databricks notebooks and PySpark jobs.
• Support cloud migration and modernization initiatives .
• Collaborate with architects, developers, business teams, and offshore/onshore teams.
• Gather business and technical requirements and translate them into scalable data solutions.
• Participate in solution design, development, testing, deployment, and production support.
• Support AI/ML initiatives by preparing data pipelines and datasets for Machine Learning and recommendation use cases.
• Troubleshoot and optimize data pipelines for performance, reliability, and scalability.

Job Tags

Similar Jobs