Lead Hadoop Developer/Analyst/

PROFESSIONAL SYNOPSIS:

• Over 9 years of experience in application development and design using Analytical R, Mahout, Weka, NLP packages, Hadoop echo system tools, Big Data, Big Data Analytics, Abinitio and Java /J2EE Technologies.
• Extensive experience in installing, configuring and using eco system components like Hadoop Map reduce, HDFS, HBase, Hive, Pig,Flume, Sqoop,mahout R packages.
• Strong knowledge of Software Development Life Cycle SDLC including Business interaction, Requirement Analysis, Software Architecture, Design, Development, Testing and Documentation phases
• Strong knowledge and understanding of Analytical R, Hadoop HDFS MapReduce concepts and Hadoop Ecosystem.
• Experience in Apache, Hortonworks HDP Cloudera distributions CDH .
• Created various use cases using massive public data sets. Ran various performance tests for verifying the efficacy of Map Reduce, PIG and HIVE in various modes standalone, pseudo distributed, cluster and Cloud.
• Experience in Vertica, Impala, Solr and NOSQL databases like HBASE and Cassandra and also performed benchmarking on BigSQL, Impala and Hive.
• Worked on Distributed/Cloud Computing Map Reduce/Hadoop, Pig, Hbase, AVRO, Zookeeper, etc. , Amazon Web Services S3, EC2, EMR, etc. ,
• Explored Spark, Kafka, Storm along with other open source projects to create a realtime analytics framework.
• Develop and maintain operational best practices for smooth operation of Cassandra/Hadoop clusters.
• Have continuous learning approach, learned and practices newer tools like Solr, Elastic search, Kibana, Lucerne, Spotfire.
• Delivered Working Widget Software using EXTJS4, HTML5, RESTFUL Web services, JSON Store, Linux, HADOOP, ZOOKEEPER, NO SQL databases, JAVA, SPRING Security, JBOSS Application Server for Big Data analytics
• Built Analytics KPI engine using Python and PIG.
• Expertise in design and development of various web and enterprise applications using various technologies like JSP, Servlets, Struts, Hibernate, MySql, Spring, JDBC, EJB, JMS,JNDI JSF, XML, IBM MQ Series, AJAX, SOAP and Web Services.
• Involved in the configuration management using ClearCase.
• Good working experience on Application Servers like WebSphere and WebLogic with IDEs like WSAD5.1, RAD6.0, RAD7.0, Eclipse and JBuilder.
• Extensively used Informatica Power Center in end-to-end of Data warehousing ETL routines, which includes writing custom scripts, data mining and data quality process.
• Work with peers in development to tune infrastructure and plan for resource management including adding / removing cluster nodes for maintenance or capacity needs.
• In depth understanding of data structures and algorithms.
• Experienced in application design using Unified Modeling Language UML , Sequence diagrams, Case diagrams, Entity Relationship Diagrams ERD .
• Solution Architecting skills in the Enterprise data warehousing and ETL functionalities using Abinitio
• Worked on deployed technologies for exclusively off-site using the Amazon infrastructure and ecosystem EMR, Redshift, Hive, DynamoDB
• A quick learner, punctual and trustworthy.
• A great team player ability to effectively communicate with all levels of the organization such as technical, management and customers.
• Senior-level technical product executive technologist with global operations background. Successful track at Technologies including Solr, Lucene and Hadoop, Developed consumer web application.

TECHNICAL Skills

DOMAIN : Life Sciences, Finance, Banking and Insurance, automobile retail.

TACTICAL

• Methodologies : Agile/Scrum, UML, Design Patterns.HDLC.
• Testing : JUNIT, MRUNIT, HP clear quest.
• IDE Testing Tools :Net Beans, Eclipse. IBM Rational Application Developer RAD 6, Rational Clearcase, Rational Rose

PROFESSIONAL EXPERIENCE

Confidential

Role: Lead Hadoop Data mining engineer

Responsibilities:

• Interacting with other data scientists and architected custom solutions for data visualization using tools like tabulae, Packages in R and R-Shiny.
• Working as a lead on Big Data Integration and Analytics based on Hadoop, SOLR, Spark, Kafka, Storm and webMethods technologies.
• Lambda Architecture - Planned and help execute real-time streaming and analysis of sentiment for twitter data.Plugged into twitter API to follow certain keywords and sentiment was calculated.
• Strongly recommended to bring in ElasticSearch and was responsible for installing, configuring and administration of a 6 node Elasticsearch Cluster. Designed the schemas of the lucene indexes for various projects.
• Identifying the problem areas by using elastic search Kibanna with Logstash to import .csv files. Using Solr over Lucene index provided a full text search for analysis and quantification.
• Working with SAP professionals in writing SAP BO queries and scripts to extract high volume data.
• Took part in identification of BI/Analytics tools that fit into the Hadoop Ecosystem. A criteria matrix was created and scored each product on the various criteria.
• Managed a team of Java developer for identification process called harm identification and then processing GB's TB's of data using the combination of Java module Hadoop map-reduce. The environment of work was Cloudera distribution CDH4 .
• End result of the Hadoop map-reduce was given to Scripting in R for data visualization and machine-learning algorithms.
• Apache Hive pig were used in intermediate processes of sampling and quick generation of stats data. Some of the data was feed for post market surveillance using Spotfire.
• Worked on Multi Clustered environment and setting up Cloudera Hadoop echo System.
• Also extract device related JSON data for statistical market performance using Apache Mahout machine learning libraries to wrote advanced data mining and statistical procedures like filtering, clustering and classification for detecting patterns of device behaviours.
• Lead a part of team of Risk Management council and provided tactical reports on product field performance.

Quantifiable Achievements

• Presenter in Symposium of Big Data Analytics for text mining, .
• Futher explored technologies like Lucene, Mahout, R, Elastic search, Kibana,logstash

Environment:

Analytical R / Rstudio/R-shinny, Weka, Spotfire, Mahout, RHadoop Hadoop, MapReduce, Java JDK 1.6 , SAP BO, Rapid Minner

Confidential

Role: Big Data Hadoop Developer

Responsibilities:

While working in cloudera environment, Participant in developing Apache Horton works POC for AWS usage using Ambari to reduce the overhead of server maintenance.

Did hive optimization like

I. Replacing the existing text storage to compressed rcfile and sequence file

II. Bucketing data and encouraging pulling sampling of data using buckets.

III. Rebuilding deferred indexes.

IV. Replacing reference to small queries with map joins and semi-joins.

• Experience in developing customized Hive UDFs and UDAFs in Java, JDBC connectivity with hive, development and execution of Pig scripts and Pig UDF's.
• Implemented business logic by writing UDFs in Java and used various UDFs from Piggybanks and other sources.
• Daily scheduled setup using oozie workflow for sqoop data extraction, transformation using pig and storing to external hive table locations.
• Most centralized source of information is the policy master file, this file was heart of the ever growing system hence was decided to be made to be part of Cassandra system.
• Responsible for building this scalable distributed data solutions using Datastax Cassandra.
• Involved in business requirement gathering and proof of concept creation.
• Created data models in CQL for customer data.Involved in Hardware installation and capacity planning for cluster setup.
• Used the Datastax Opscenter for maintenance operations and Keyspace and table management.Implemented advanced procedures like text analytics and processing using the in-memory computing capabilities like Spark.
• Worked with the Data architect and the Linux admin team to set up, configure, initialize and troubleshoot an experimental cluster of 12 nodes with 3 TB of RAM and 60 TB of disk space.
• Tuned and recorded performance of Cassandra clusters by altering the JVM parameters like -Xmx and -Xms. Changed garbage collection cycles to place them in tune with backups/compactions so as to mitigate disk contention.
• Created map-reduce code for encryption of data, data-cleansing and ETL processes of high volume Csv's, sas datasets, text-files and XML files.
• Generated reports using Pentaho report designer.

Quantifiable Achievements

• Got chance to learn AWS practice implementation of Hadoop cluster in aws cloud.
• More in-depth in wrapper level programming in Java using hive, Cassandra API's. Full scale implementation of cassandra
• Comfortable knowledge in .dot nets work flow tools.

Environment:

Hadoop, MapReduce, HDFS, Hive, Java JDK 1.6 , MySql, linux, Hadoop Distribution of HortonWorks, Cloudera, Ubuntu 12.04 LTS server, .net Workflow, ControlM, AWS cloud,Cassandra,Hive,pig,

Confidential

Hadoop/ Big data Developer

Responsibilities:

• Working with the administrator to setup HortonWorks Hadoop clusters. Used Ambari for Managing Hortonworks Distribution of Hadoop.
• Involved in lot of open issues with HortonWorks directly and communicated the problem and resolved with their fixes
• Installed and configured Hadoop, MapReduce, HDFS
• Developed multiple MapReduce jobs in java for data cleaning and reprocessing.
• Importing and exporting data into HDFS and Hive using Sqoop.
• Experienced in defining job flow using Oozie and supporting job stream.
• Wrote map-reduce programs for click log analysis and log processing on the logs extracted by Flume agents. Also experienced in configuring flume agents.
• Avro files extracted by Flume was processed with Hive serde's and stored in external hive tables. Hive views were created for data analysts.
• Pig UDF's written in java were used with Pig scripting for currency conversion and tax calculations.
• Experienced in running Hadoop streaming jobs to process terabytes of xml format data.
• Experience in designing ETL solutions of Informatica Power Center tools such as Designer, Repository Manager, Workflow Manager and Workflow Monitor to Hadoop and ego system managed technologies.
• Enterprise Service Bus Logs were loaded into HDFS everyday, and reports were generated that enabled the monitoring team to measure SLO's of various services and generated reports for the usage of ESB as a whole.
• Architected a solution to ingest and retrieve data into Hbase with indexes stored in Apache Solr.
• Designed and administered 40 TB 9 Node cluster and a 2 TB Node cluster for dev/R D hosted over cloud.
• Implemented Security using Kerberos for internode communication.
• Configured Knox for security through REST access and XA-Secure for audit security management.
• Used combination of Ganglia and custom shell scripts to monitor the file system and usage
• Set up automated 24x7x365 monitoring and escalation infrastructure for Hadoop cluster using Nagios and Ganglia.
• Scheduling cron jobs for file system check using fsck and also running the balancer for uniform load on the nodes.
• Data scrubbing and processing with Oozie.
• Utilized Agile Methodologies to manage full life-cycle development of the project.
• Gained very good business knowledge on banking and finance, fraud detection and general ledger reporting.
• Used Resilient Distributed Datasets RDDs to manipulate data, perform light analytics and create visualizations using the high performance distributed computing framework of Apache Spark.

Quantifiable Achievements

Witnessed the introduction of low cost Gold cards for Mexico and India, other markets were in progress. Hadoop being open source, it saved time and money that were getting spent on proprietary tools.

Environment:

Hadoop, MapReduce, HDFS, Hive, Oozie,Sqoop,Flume Java JDK 1.6 , Hadoop Distribution of CDH3, Teradata, Flat files, PL SQL, SQL PLUS, Toad 9.6, Windows NT, UNIX Shell Scripting, ControlM,Informatica Powercenter tools, Kerberos security, Nagios, Ganglia.

Confidential

BI ETL developer.

Responsibilities:

BI ETL :

• Giving technical estimations to Project/SR timelines and designing all kinds of Technical documents Tech design, Capacity Plans, Source-Target Mapping documents, Implementation Plans etc.
• Using Visio in designing and presenting data flows and process flows.
• Create traceability matrices to keep track of the requirement. Design of the ETL solution and Design review.
• Reviewing write Abinitio codes for Performance and tuning, monitoring all the processes till Staging and Production. Used BI solution as a custom application build using OBIEE as front end and ODI and PL/SQL used for ETL.
• Strong experience in Dimensional Modeling using Star and Snowflake schemas to build. Fact and Dimensional tables using Erwin Data Modeling tool.
• Used various transformations like Filter, Expression, Sequence Generator, Update Strategy. Joiner and SQL, Lookup File and Database to develop robust mappings in the Informatica Designer.
• Worked on Business Intelligence BI /Data Analytics, Data Visualization, SAS/R, Data warehouse Architecture Design and MDM/Data Governance

J2EE

• Extensive experience in JVM Performance tuning including tuning heap size, GCThresholds/Cycles Memory Management etc.
• Designed the application using Front Controller, Service Controller, MVC, Factory, Data Access Object, and Service Locator.
• Implemented the web-based application following the MVC II architecture using Struts framework. Validations were done on the client side as well as the server side.
• Developed entire application based on STRUTS framework and configured struts config.xml, web.xml.Implemented validation framework for creation of validation.xml and used validation rules.xml.
• Form classes of Struts Framework to write the routing logic and to call different services.
• Created tile definitions, Struts-config files, validation files and resource bundles for all modules using Struts framework.
• Implemented open source MVC architecture using Spring MVC and configured the beans and extensively used Spring IOC for dependency injection.
• Used Spring AOP for transaction management features.
• Integrated Hibernate ORM with Spring-Hibernate framework to facilitate DML and DQL queries and represent object-database mapping.
• Developed business layer components using spring beans.
• Expertise in developing the multi-tier web applications in both Spring MVC and Struts MVC that ensures clear separation of layers and rapid application development.
• Implemented logging by using Log4j, developed assertions to work with JUnit, and worked SVN control for version controlling.
• Implemented the application using the concrete principles laid down by several design patterns such as MVC, Business Delegate, Session Facade, Data Access Object, Singleton and Factory.

Quantifiable Achievements

• Learning Abinito BI technologies.
• Developing co-ordination and team building soft skills.
• Managing small teams as team lead.

Environment:

Ab Initio GDE 1.15.7,1.15.13/ CO-OP 2.15-8-7, UNIX shell scripting, Control M, Quality center 9.0, QTP, J2EE technologies : Struts, Spring, Hibernate, Log4j, SVN Junit.

Confidential

Java/J2EE Developer

Roles and responsibilities:

• Involved in System Analysis and Design methodology as well as Object Oriented Design and development using OOA/OOD methodology to capture and model business requirements.
• Proficient in doing Object Oriented Design using UML-Rational Rose. Created Technical Design Documentation TDD based on the Business Specifications.
• Developed UI using HTML, JavaScript, CSS and JSP for interactive cross browser functionality and complex user interface.
• Coded Servlets on the server side, which gets the requests from the client and processes the same by interacting the Oracle database.
• Good Exposure to Service Oriented Architecture SOA such as Web Services including SOAP, UDDI, WSDL, JMS Java Messaging Service
• Captured and described all the requirements of the system in co-operation with the end user in preparing functional specifications of the system in spread sheet.
• Involved in implementing interactive web designing web application using Servlets, JSPs and JSTL, Custom Tags.
• Used XML parser APIs such as JAXP SAX and JAXB in the web service's request response data marshalling as well as unmarshalling process.
• Used Eclipse as development environment for Java application.
• Development of the server and client side application on tracking and report modules.
• Actively involved in system, integration testing and product support.
• Designed database and created tables, written the complex SQL Queries and stored procedures as per the requirements.
• Involved in improved performance of whole system by moving some of the data retrieving logic into database using Pl-SQL/SQL, stored procedures and triggers.
• Developed the application using Java Beans, Servlets and EJB's.
• Created Stateless Session EJB's for retrieving data and Entity Beans for maintaining User Profile.
• Used WebSphere Application Server and RAD to develop and deploy the application.
• Involved in developing Custom Tags for common look and feel. Used Maven for built code

Environment: J2EE, JSP, JSTL, Custom Tags, CSS, Java Script, Oracle 8.X, PL/SQL, SOAP,SOA, JMS, XML parsers