SUMMARY:

• Around 7 years of comprehensive experience in Hadoop ecosystem tools and Big Data technologies.
• Currently working on Apache Kafka, Apache Spark, Apache Storm and Hadoop Map Reduce, Hbase and Phoenix.
• Developed Storm topologies to read data from kafka topics, populated staging tables and stored the refined data in partitioned hive tables in Google Cloud Storage.
• Developed application that serves a web application that performs a REST HBASE API calls to read and write data to hbase tables using PHOENIX
• Worked on ETL tool Syncsort. Responsible for writing a DMX job that reads data from kafka topics and writes it to partitioned hive tables in HDFS, GCS and finally loading it into Teradata.
• Experience in designing and developing real time big data processing applications using Kafka, Storm, HBase, Hive, Spark
• Hands on experience in developing end to end solutions in the data pipeline and handling massive amounts of Data.
• Actively Involved in User training for data analytics tool like Apache Zeppelin
• Comprehensive experience in analyzing data using HiveQL, Pig Latin and custom MapReduce/Spark programs in Java.
• Working experience in designing and implementing complete solutions using Hadoop Infrastructure including PIG, HIVE, Sqoop, Oozie
• Hands on experience in working with various data sources like csv files, xml files, Json files, Oracle/Mysql to load data into Hive tables.
• Active member of team in writing shell scripts for Unix OS for application deployments to production region.
• Involved in maintaining hadoop cluster in development and test environment
• Good knowledge in mining the data in hadoop file system for business insights using Hive, Pig
• Expertise in Relational Database design, data extraction, data transformation from data sources using MySql and Oracle
• Good working knowledge on Eclipse IDE for both developing and debugging java applications.
• Involved in code performance improvement and query tuning activities.
• Ability to take initiate, share ideas and skills and motivate co - workers from various backgrounds, creative problem solving and drive positive work environment by working collaboratively and in cohesion with rest of the team members.

PROFESSIONAL PROFILE:

Confidential

Responsibilities:

• Strong experience in Hadoop Administration & Big Data Technologies along with Linux & Database administration.
• Experience with complete Software Design Lifecycle including design, development, testing and implementation of moderate to advanced complex systems.
• Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera.
• Expertise in Hadoop Cluster capacity planning, performance tuning, cluster Monitoring, Troubleshooting.
• Design Big Data solutions for traditional enterprise businesses.
• Backup configuration and Recovery from a Name Node failure.
• Excellent command in creating Backups & Recovery and Disaster recovery procedures and Implementing BACKUP and RECOVERY strategies for off-line and on-line Backups.
• Involved in bench marking Hadoop/Hbase cluster file systems various batch jobs and workloads
• Expertise in Hadoop cluster ready for development team working on POCs.
• Experience in minor and major upgrades of Hadoop and Hadoop eco system.
• Experience monitoring and troubleshooting issues with Linux memory, CPU, OS, storage and network
• Hands on experience in analyzing Log files for Hadoop and eco system services and finding root cause.
• Experience on Commissioning, Decommissioning, Balancing, and Managing Nodes and tuning server for optimal performance of the cluster.
• Involved in Administration of Cluster maintenance, trouble shooting, Monitoring and followed proper backup& Recovery strategies.
• Created system security supporting multi-tier software delivery system by utilizing Active Directory and Kerberos.
• Installed and configured Hadoop eco system like hive, sqoop, pig, and hive.
• Handsome experience in Linux admin activities on RHEL.
• Familiar with writing Oozie workflows and Job Controllers for job automation - shell, hive, scoop automation.

Hadoop Developer

Confidential

Responsibilities:

• Contributed to Design of Application and Database Architecture.
• Provided design recommendations and thought leadership to sponsors/stakeholders that improved review processes and resolved technical problems.
• Managed and reviewed Hadoop log files.
• Tested raw data and executed performance scripts.
• Shared responsibility for administration of Hadoop, Hive and Pig. Created pig scripts and map reduce programs to filter the log files and aggregate the data.
• Documented design document and conducted various POC’s to validate the design
• Worked on Apache Kafka, Apache Spark, Apache Storm and Hadoop Map Reduce, Hbase and Phoenix.
• Developed Storm topologies to read data from kafka topics, populated staging tables and stored the refined data in partitioned hive tables in Google Cloud Storage.
• Developed application that serves a web application that performs a REST HBASE API calls to read and write data to hbase tables using PHOENIX.
• Excellent understanding of Hadoop architecture and its components.
• Developed a map reduce program to validate the raw data before loading it to database for analysis for the required columns and data format.
• Loaded data from Linux file system to HDFS using Pentaho Data Integration
• Created data flows in Pentaho Data Integration for aggregating the data and loading the data to Hive tables
• Moved the data to Oracle tables from Hive using Sqoop for reporting
• Created Bar graphs, Heat maps and Geo Maps from aggregated data using Pentaho User Console.
• Completed Unit Testing and Integration Testing.
• Documented the user manual and troubleshooting guides
• Designed & developed several use cases. Created Oozie workflows to streamline the data flow and shell scripts to load the raw data to HDFS
• Loaded log files data to Hive
• Used Sqoop to move data between HDFS and MySql
• Developed a spark code to migrate existing mapreduce code and pig scripts as part of proof of concept
• Unit tested the application
• Involved in making very important and major enhancements to the already existing Map Reduce programs and Pig scripts
• Used SVN i.e. Subversion for version control and maintained different versions based on the release
• Technically documented every design and development detail of each use case
• Administered and Maintained the hadoop clusters in Development and Test environments
• Provided support and maintenance
• Provided business insights on purchase patterns during promo periods by mining data in Hive

Associate Hadoop Developer

Confidential

Responsibilities:

• Installed and configured MapReduce, HIVE and the HDFS; implemented CDH3 Hadoop cluster on CentOS. Assisted with performance tuning and monitoring.
• Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Supported code/design analysis, strategy development and project planning.
• Created reports for the BI team using Sqoop to export data into HDFS and Hive.
• Developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
• Assisted with data capacity planning and node forecasting.
• Collaborated with the infrastructure, network, database, application and BI teams to ensure data quality and availability.
• Administrator for Pig, Hive and Hbase installing updates, patches and upgrades.