Job Seekers, Please send resumes to resumes@hireitpeople.com

Detailed Job Description:

• 5+ years total experience in development on Bigdata, Hive & Hadoop, Spark, Scala, Python/Pyspark, AWS, and other cloud related technologies.
• Independent/lead developer who can work with minimal supervision.
• Solid understanding of distributed system fundamentals.
• Solid understanding of hadoop security and familiar with kerberos/keytabs etc and hands on experience with working with Spark/Hive/Oozie/Kafka etc on a kerberized cluster.
• Experience in developing, troubleshooting, diagnosing, and performance tuning of distributed batch & real-time data pipelines using Spark/PySpark at scale.
• Develop scalable and reliable data solutions to move data across systems from multiple sources in real time (Nifi, Kafka) as well as batch modes (Sqoop)
• Demonstrated professional experience working with various components of Big Data ecosystem: Spark/Spark Streaming, Hive, Kafka/KSQL, Hadoop (or similar NoSQL ecosystem) and orchestrate these pipelines using oozie, et. al, in a production system.
• Construct data staging layers and fast real-time systems to feed BI applications and machine learning algorithms.
• Strong software engineering skills with Python or Scala/Java.
• Knowledge of some flavor of SQL (MySQL, Oracle, Hive, Impala), including the fundamentals of data modeling and performance.
• Skills in real-time streaming applications.
• Develop scalable and reliable data solutions to move data across systems from multiple sources in real time (Nifi, Kafka) as well as batch modes (Sqoop).
• Experienced in Data Engineering with good understanding of Datawarehouse, Data Lake, Data Modelling, Parsing, Data wrangling, Cleansing & Transformation, and sanitizing.
• Agile work experience, build CI/CD pipelines using Jenkins, GIT, Artifactory, Anisble etc.
• Hands-on Development experience with Scala, Python using Spark 2.0, Spark Internals and Spark jobs performance improvement.
• Good understanding of Yarn, Spark UI, Spark resource management and Hadoop resource management and efficient Hadoop storage mechanisms.
• Good understanding & experience with Performance tuning in Cloud environment for complex S/W projects mainly around large scale and low latency.
• AWS knowledge is essential with good working experience in AWS Technologies EMR, S3, Cluster management, AWS Airflow automation, Snowflake Knowledge is plus.
• AWS development certification/Spark certifications is an advantage.
• Expert in data analysis in Python (Numpy, Scipy, Scikit-learn, Pandas, etc.)
• Strong UNIX Shell scripting experience to support data warehousing solutions.
• Process oriented, focused on standardization, streamlining, and implementation of best practices delivery. 
• Excellent problem solving and analytical skill, excellent verbal and written communication skills.
• Proven teamwork in multi-site/multi-geography organizations.
• Ability to multi-task and function efficiently in a fast-paced environment.
• Strong background in Scala or Java and experience with streaming technologies such as Flink, Kafka, Kinesis, and Firehose ,experience with EMR, Spark, Parquet, and Airflow.
• Excellent interpersonal skills, ability to handle ambiguity and learn quickly.
• Exposure to data architecture & governance is helpful.
• A degree in Computer Science or a related technical field; or equivalent work experience

Minimum years of experience*: 5+