Lead II - SE - AWS, Apache Spark (PySpark/Scala), Apache Kafka

JOB DETAILS:

* Job Title:

Lead II - Software Engineering - AWS, Apache Spark (PySpark/Scala), Apache Kafka

* Industry:

Global digital transformation solutions provider

* Salary:

Best in Industry

* Experience:

5-8 years

* Location:

Hyderabad

Job Summary

We are seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms. The role involves working with large-scale batch and real-time data processing systems, collaborating with cross-functional teams, and ensuring data reliability, security, and performance across the data lifecycle.

Key Responsibilities

ETL Pipeline Development & Optimization

• Design, develop, and maintain complex end-to-end ETL pipelines for large-scale data ingestion and processing.
• Optimize data pipelines for performance, scalability, fault tolerance, and reliability.

Big Data Processing

• Develop and optimize batch and real-time data processing solutions using Apache Spark (PySpark/Scala) and Apache Kafka .
• Ensure fault-tolerant, scalable, and high-performance data processing systems.

Cloud Infrastructure Development

• Build and manage scalable, cloud-native data infrastructure on AWS .
• Design resilient and cost-efficient data pipelines adaptable to varying data volume and formats.

Real-Time & Batch Data Integration

• Enable seamless ingestion and processing of real-time streaming and batch data sources (e.g., AWS MSK ).
• Ensure consistency, data quality, and a unified view across multiple data sources and formats.

Data Analysis & Insights

• Partner with business teams and data scientists to understand data requirements.
• Perform in-depth data analysis to identify trends, patterns, and anomalies.
• Deliver high-quality datasets and present actionable insights to stakeholders.

CI/CD & Automation

• Implement and maintain CI/CD pipelines using Jenkins or similar tools.
• Automate testing, deployment, and monitoring to ensure smooth production releases.

Data Security & Compliance

• Collaborate with security teams to ensure compliance with organizational and regulatory standards (e.g., GDPR, HIPAA).
• Implement data governance practices ensuring data integrity, security, and traceability.

Troubleshooting & Performance Tuning

• Identify and resolve performance bottlenecks in data pipelines.
• Apply best practices for monitoring, tuning, and optimizing data ingestion and storage.

Collaboration & Cross-Functional Work

• Work closely with engineers, data scientists, product managers, and business stakeholders.
• Participate in agile ceremonies, sprint planning, and architectural discussions.

Skills & Qualifications

Mandatory (Must-Have) Skills

1. AWS Expertise

• Hands-on experience with AWS Big Data services such as EMR, Managed Apache Airflow, Glue, S3, DMS, MSK, and EC2 .
• Strong understanding of cloud-native data architectures.

1. Big Data Technologies

• Proficiency in PySpark or Scala Spark and SQL for large-scale data transformation and analysis.
• Experience with Apache Spark and Apache Kafka in production environments.

1. Data Frameworks

• Strong knowledge of Spark DataFrames and Datasets .

1. ETL Pipeline Development

• Proven experience in building scalable and reliable ETL pipelines for both batch and real-time data processing.

1. Database Modeling & Data Warehousing

• Expertise in designing scalable data models for OLAP and OLTP systems.

1. Data Analysis & Insights

• Ability to perform complex data analysis and extract actionable business insights.
• Strong analytical and problem-solving skills with a data-driven mindset.

1. CI/CD & Automation

• Basic to intermediate experience with CI/CD pipelines using Jenkins or similar tools.
• Familiarity with automated testing and deployment workflows.

Good-to-Have (Preferred) Skills

• Knowledge of Java for data processing applications.
• Experience with NoSQL databases (e.g., DynamoDB, Cassandra, MongoDB).
• Familiarity with data governance frameworks and compliance tooling.
• Experience with monitoring and observability tools such as AWS CloudWatch, Splunk, or Dynatrace .
• Exposure to cost optimization strategies for large-scale cloud data platforms.

Skills:

big data, scala spark, apache spark, ETL pipeline development

******

Notice period - 0 to 15 days only

Job stability is mandatory

Location: Hyderabad

Note: If a candidate is a short joiner, based in Hyderabad, and fits within the approved budget, we will proceed with an offer

F2F Interview: 14th Feb 2026

3 days in office, Hybrid model.