Senior Engineer - Data Engineering

Engineer - Data, Python, Pyspark, 5-7 yrs, Lead the design, development, and optimization of data pipelines using Python and PySpark to ensure efficient data processing and integration

• Advanced proficiency in Python for data manipulation, analysis, and development of machine learning algorithms.
• Utilization of libraries such as Pandas, NumPy, and scikit-learn for extracting insights from complex datasets and automating data processing tasks.
• Strong background in PySpark for efficient handling of large-scale data processing and analytics across distributed computing environments.
• Adept at writing optimized PySpark code to execute transformations and actions on large datasets, facilitating real-time data processing and analysis.
• Proficiency in both Python and PySpark expected at an advanced level, with the ability to integrate these technologies into production-grade data pipelines.
• Education: Master of Technology (M.Tech) in Data Science, and Bachelor of Engineering (B.E.) or Bachelor of Technology (B.Tech) in Computer Science or Information Technology.
• Strong foundation in both theoretical and practical aspects of data science and engineering from educational background.
• Certifications preferred: Databricks Certified Data Engineer Associate and AWS Certified Big Data – Specialty.
• Demonstrates commitment to continuous learning and expertise in cloud-based data solutions.

• Lead the design, development, and optimization of data pipelines using Python and PySpark to ensure efficient data processing and integration.
• Collaborate with cross-functional teams, including data scientists and analysts, to understand data requirements and provide robust data solutions that meet business needs.
• Develop scalable data models and architectures to support large-scale data processing and storage.
• Implement data validation and testing processes to ensure high data quality and integrity throughout the data lifecycle.
• Mentor junior data engineers by providing technical guidance and support, fostering a culture of continuous learning and improvement within the team.
• Conduct performance tuning and troubleshooting of data workflows to enhance processing efficiency.
• Stay updated with the latest industry trends and best practices in data engineering and big data technologies, applying this knowledge to drive improvements in processes and tools.
• Document data engineering processes and workflows for knowledge sharing and operational continuity.

Engineer - Data, Python, Pyspark, 5-7 yrs