SUMMARY

• Led team to plan, design, and implement applications and software into Hadoop BigData Echo systems from various source systems
• Collaborated wif business analysts, developers, and technical support teams to define project requirements and specifications.
• Designed, developed, and managed web - based applications, databases, network accounts, and programs dat drive in HDFS.
• Launched complex recovery solutions to safeguard mission critical data.
• Translated technical specifications into project scopes of work and product requirements while spearheading design and development of databases and enterprise solutions.
• Implemented application developments, resolved performance issues, and provided end-user training on hardware and software.
• 14 Years of IT Experience in Analyzing, Designing, Developing, Implementing and Testing ofSoftware Applications and Currently working as Hadoop Architect/Senior Hadoop Developer.
• Hands on experience in software design and development in Big Data/HADOOP (HDFS,PIG,HIVE, HBASE, MongoDB, SQOOP, MapReduce,SPARK,KAFKA,STORM and SCALA)
• Hands on experience in software design and development using CORE JAVA,JDBC in ZOS operating system in Mainframes.
• Experience in XML parsing using DOM parser.
• Good noledge in collection frameworks to collect the structured data.
• Good experience on REST webservices to pull data from different servers of CLOUD based API
• Extensively worked on Hadoop and Hadoop Eco Systems.
• Excellent understanding of Hadoop architecture and different components of Hadoop clusters which include componenets of Hadoop (Job Tracker, Task Tracker, Name Node and Data Node).
• Data Ingestion to HDFS from various data sources.
• Analyzed large data sets by running Hive queries and Pig scripts.
• Good experience in writing the pig scripts.
• Optimization of Hive Queries.
• Ability to analyze different file formats.
• Good exposure in cluster maintenance.
• Loading data from LINUX file system to HDFS.
• Importing and exporting the data from relational databases, NO SQL DB’S using SQOOP.
• Very good noledge on HBase and Mongo DB and CASSANDRA.
• Good noledge in TES and $ schdulers.
• Automated sqoop,hive and pig jobs using Oozie scheduling.
• Configuration and deployment modules.
• Knowledge in OOZIE workflows.
• Good noledge on writing and using the user defined functions in HIVE and PIG.
• Developed multiple Kafka Producers and Consumers from scratch as per the business requirements.
• Responsible for creating, modifying and deleting topics (Kafka Queues) as and when required by the Business team.
• Working on implementing Spark and Strom frame work.
• Knowledge on Map reduce frame work.
• Extensive exposure to all aspects of Software Development Life Cycle (SDLC) me.e. Requirements Definition for customization, Prototyping, Coding (JAVA,COBOL,DB2) and Testing.
• Migrating systems written in COBOL to JAVA in order to reuse JAVA programming skills and class libraries.
• DeployedJAVA tasksdat periodically query a database and gives the result to a dataset.
• Using Java programs to access APIs such as SOAP/Web services, WebSphere MQ client API, Java Database Connectivity (JDBC) databases, custom Transmission Control Protocol/Internet Protocol (TCP/IP) socket services, and so forth.
• Passing datasets created by traditional job steps to Java programs, which convert the data to XML and also reading and writing MVS datasets from Java.
• Proficient in SYNCSORT for sorting of data files.
• Flexible configuration of the Java Virtual Machine (JVM) and environment variables.
• Routing JVM output directly to JES SYSOUT datasets.
• Controlling output encoding.
• Passing condition codes between Java and non-Java steps.
• Executing the JVM under the original batch address space.
• Communicating wif the MVS system consol.
• All phases of Software Development Life Cycle starting wif Analysis, followed by design, development and testing.
• Acquired good noledge on Mainframes and extensively worked in VS COBOL II, JCL languages, and well versed wif DB2 and the file system like PS files and VSAM and having worked in different applications domains.

TECHNICAL SKILLS

Programming Languages: C,C++,CORE,Advance Java, Scala,Python,JAVA,COBOL,JCL, SQL,Unix

Data Handling: Oozie, Ambari, Flume, Hadoop, HDFS, Map Reduce, Yarn, Pig, Hive, HBase, Sqoop,Zoo Keeper,FLUME,IMPALA

Scripting Languages: Perl,Shell Scripting

Databases Explored: MS SQL 2008,Oracle 11g, DB2, MongoDB,post gres

File Formats: PARQUET,ORC,RC,TEXT,JSON,XML,AVRO

Descriptive and Predictive Analytics: NLP,R, SPARK

Software Experience: Oracle Data Miner(ODM), Visual Studio C++, Eclipse, Oracle SQL Developer, Web Services MQ communication

Libraries/Tools Explored: ECLIPSE,MQSeries,CHANGEMAN,TSO,FILE-AID,PANVALET,LCS,STS,NDM, SPUFI,QMF

Web Related: HTML, Extended Java Script, Servlet, JavaScript, JSP, XQuery, XPath, XML, XSLT

Operating System: Android, IOS,Microsoft Windows, Unix/Solaris, Linux, MVS-OS/390,MS-WINDOWS 2000/XP AND MS-DOS

PROFESSIONAL EXPERIENCE

Confidential, Owing Mills, Maryland

Senior Fullstack BigData developer

Responsibilities:

• Architect,Design,Code and implement next-generation data analytics platform using software engineering best practices in latest technologies using Apache Spark, Scala,Java,R
• Graph DataBase for schduling the workflows
• NoSQL technologies like Cassandra,HBase,DynamoDB
• Spark integration wif Big Data(Hadoop),Amazon EMR,Apache Zeplin
• Provide software expertise in these areas: Spark based applications, Java application Integration, web services, Cloud Computing
• Develop solutions to enable metadata/Rules engine driven data analytics application leveraging open source and/or cloud native components
• Develop solutions in highly collaborative and agile environment

Confidential, New Jersy /BridgeWaters

Senior Hadoop Developer

Responsibilities:

• Design, configure, implement and manage the SnappyData cluster platform dat processes Point of ERP data wif optimal performance and ease of maintenance.
• Created a scripts dat ingest the data from various regions in grid.
• Troubleshoot and resolve various process or data related issues. Will be on call and provide off-hour support as needed.
• Assist in the ongoing development and documentation of the standards for the system and data processes.
• Create project plans, manage milestones, create and distribute reports and manage risks
• Communicate TEMPeffectively wif senior management, direct reports and customers.
• Develop Hive, Pig scripts for data transformation.
• Develop Python, shell,JAVA and HQL scripts for data flow orchestration.

Confidential, Port Washington, NY

Senior Hadoop Developer

Responsibilities:

• Design, configure, implement and manage the Hadoop HBase platform dat processes Point of Sale data wif optimal performance and ease of maintenance.
• Created an Application Framework dat ingests data from various sources using SPARK framework.
• Use SCALA/Python as the programming language to process the framework to Ingest to HBASE/Phoenix.
• Use Apache Phoenix as SQL projection on top of No-SQL Hbase Projection.
• Create the data models and tables dat ingest data to HBASE/Phoenix.
• Troubleshoot and resolve various process or data related issues. Will be on call and provide off-hour support as needed.
• Assist in the ongoing development and documentation of the standards for the system and data processes.
• Create project plans, manage milestones, create and distribute reports and manage risks
• Communicate TEMPeffectively wif senior management, direct reports and customers.
• Develop Hive, Pig scripts for data transformation.
• Develop Python, shell,JAVA and HQL scripts for data flow orchestration.
• Use MAVEN as the build tool to compile and promote the code and SVN as version control tool.
• Worked wif SQL Server meta data system.

Confidential, Herndon, VA

Hadoop Developer

Responsibilities:

• Develop Hive, Pig scripts for data transformation.
• Exposure about ETL batch and the concept of datawarehousing.
• Develop Hadoop jobs through schedulers and use SSIS orchestration engine andalso through oozie
• Develop Python, shell,JAVA and HQL scripts for data flow orchestration;
• Manage software build when needed thru Microsoft TFS and GIT
• Support REST-Based ETL Hadoop software in higher environments like UAT, Production;
• Built the SSIS packages dat orchestrates the Green Plum Jobs and Troubleshoot SSIS Packages if needed
• Worked wif SQL Server meta data system; and
• Ability to troubleshoot asp.net web API based REST layer.
• Architect, Design and develop Hadoop ETL by using Kafka.
• To create SPARK,PIG and HIVE Jobs by using Python Rest Orchestration
• To build MR API's programs where we used in the combination of HIVE and HBASE
• Ability to work on Green Plum(Postgres DB) for the transformed data to store.
• To create MONGO collection to store in MONGO DB for the persistent storage.
• Developing multiple Java based Kafka Producers and Consumers from scratch as per the business requirements.
• Worked on XML formatted data,Text formatted Data,JSON formatted data.
• Used AVRO’s as a schema to Hive tables.

Confidential, Boston, MA

Big Data Application senior developer and Solutions Architect

Responsibilities:

• SME on Big data technologies( hdfs, yarn, mapreduce, impala, hive, oozie, spark, sqoop, hbase, platform architecture.) Worked wif Hortonworks technical team in resolving issues.
• Evaluating client needs and translating their business requirement to functional specifications theirby onboarding them onto the Hadoop ecosystem.
• Working on designing the mapreduce and Yarn flow and writing mapreduce scripts, performance tuning and debugging.
• Single point of contact for the lamda architetcure to develop on the Hadoop platform.
• Developing multiple Java based Kafka Producers and Consumers from scratch as per the business requirements.
• Responsible for creating, modifying and deleting topics (Kafka Queues) as and when required by the Business team.
• Working on implementing Spark and Strom frame work to ingest the data in real time and apply transfoemations in SCALA
• Creating Hive tables, loading the data and writing hive queries dat will run internally in a map reduce way.
• Data lineage in hadoop to track down the data from where its being ingetsed and also has sound noledge on various tools to figureout dat lineage.
• Imported data using Sqoop to load data from Oracle to HDFS on regular basis.
• Configure schedulers for the scripts.
• Written Hive queries for data analysis to meet the business requirements.
• Created HBase tables to store variable data formats coming from different portfolios.
• Implemented HBase custom co-processors, observers to implement data notifications.
• Used HBase thrift API to implement Real time analysis on HDFS system.
• Developed Pig scripts to implement ETL transformations including Cleaning, load and extract.
• Developed PIG UDFs to incorporate external business logic into pig scripts.
• Developed HIVE UDFs to incorporate external business logic into hive scripts
• Developed join data set scripts using HIVE join operations.
• Developed join data set scripts using Pig Latin join operations.
• Designed and implemented Map Reduce-based large-scale parallel relation-learning system.
• Implemented data injection process using flume sources, flume consumers and flume interceptors
• Validated the performance of Hive queries on Spark against running them traditionally on Hadoop
• Involved in Testing and coordination wif business in User testing.
• Importing and exporting data into HDFS and Hive using Sqoop.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate TEMPeffective querying on the log data
• Involved in creating Hive tables loading data and writing queries dat will run internally in MapReduce way.
• Used Pig as ETL tool to do transformations, event joins, filter and some pre-aggregations.
• Involved in processing ingested raw data using MapReduce, Apache Pig and HBase.
• Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
• Used Hive to analyze the partitioned and bucketed data to compute various metrics for reporting.

Confidential, Phoenix, Arizona

Big Data Application Architect

Responsibilities:

• SME on Big data technologies( hdfs, yarn, mapreduce, impala, hive, oozie, spark, sqoop, syncsort ingestion, platform architecture.) Worked wif Cornerstone technical team in resolving issues.
• Evaluating client needs and translating their business requirement to functional specifications theirby onboarding them onto the Hadoop ecosystem.
• Worked on designing the mapreduce flow and writing mapreduce scripts, performance tuning and debugging.
• Involved in creating Hive tables, loading the data and writing hive queries dat will run internally in a map reduce way.
• Figured out the data lineage in hadoop to track down the data from where its being ingetsed and also has sound noledge on various tools to figureout dat lineage.
• Imported data using Sqoop to load data from Oracle to HDFS on regular basis.
• Imported data from Terradata to HDFS thru Informatica maps using Unix scripts..
• Configure Dollar View schedulers for Hive and Pig jobs to run the Hive and Pig scripts.
• Written Hive queries for data analysis to meet the business requirements.
• Created HBase tables to store variable data formats coming from different portfolios.
• Implemented HBase custom co-processors, observers to implement data notifications.
• Used HBase thrift API to implement Real time analysis on HDFS system.
• Developed Pig scripts to implement ETL transformations including Cleaning, load and extract.
• Developed PIG UDFs to incorporate external business logic into pig scripts.
• Developed HIVE UDFs to incorporate external business logic into hive scripts
• Developed join data set scripts using HIVE join operations.
• Developed join data set scripts using Pig Latin join operations.
• Designed and implemented Map Reduce-based large-scale parallel relation-learning system.
• Implemented data injection process using flume sources, flume consumers and flume interceptors.
• Validated the performance of Hive queries on Spark against running them traditionally on Hadoop
• Tested and coordinated wif business in User testing.
• Importing and exporting data into HDFS and Hive using Sqoop.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate TEMPeffective querying on the log data
• Involved in creating Hive tables loading data and writing queries dat will run internally in MapReduce way.
• Used Pig as ETL tool to do transformations, event joins, filter and some pre-aggregations.
• Involved in processing ingested raw data using MapReduce, Apache Pig and HBase.
• Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
• Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs.
• Used Hive to analyze the partitioned and bucketed data to compute various metrics for reporting.

Confidential, SAN JOSE, CA

Big Data Technology Analyst

Responsibilities:

• Created the existing Mainframes functionality in HDFS.
• Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop.
• Migration of all the data from mainframes to HDFS thru FTP protocol to local Unix Box.
• Migrating all the programs,Jobs and schdules to Hadoop.
• Configure the Dollar View schedulers for the jobs migrated from the mainframes schduler.
• Worked on designing the mapreduce flow and writing mapreduce scripts, performance tuning and debugging.
• Involved in creating Hive tables, loading the data and writing hive queries dat will run internally in a map reduce way.
• Figured out the data lineage in hadoop to track down the data from where its being ingetsed and also has sound noledge on various tools to figureout dat lineage.
• Designed the rules dat dynamically creates the XML parsing using DOM parser.
• Implemented and navigated the structured data using collection frameworks
• Used the REST webservices to pull data from different servers of CLOUD based API.
• Imported data using Sqoop to load data from Oracle to HDFS on regular basis.
• Imported data from Terradata to HDFS thru Informatica maps using Unix scripts..
• Configure Dollar View schedulers for Hive and Pig jobs to run the Hive and Pig scripts.
• Written Hive queries for data analysis to meet the business requirements.
• Created HBase tables to store variable data formats coming from different portfolios.
• Implemented HBase custom co-processors, observers to implement data notifications.
• Used HBase thrift API to implement Real time analysis on HDFS system.
• Developed Pig scripts to implement ETL transformations including Cleaning, load and extract.
• Developed PIG UDFs to incorporate external business logic into pig scripts.
• Developed HIVE UDFs to incorporate external business logic into hive scripts
• Developed join data set scripts using HIVE join operations.
• Developed join data set scripts using Pig Latin join operations.
• Designed and implemented Map Reduce-based large-scale parallel relation-learning system.
• Implemented data injection process using flume sources, flume consumers and flume interceptors
• Presently implementing KAFKA to collect the logs from the HIVE jobs.
• Configured deployed and maintained multi-node Dev and Test Kafka Clusters.
• Developed multiple Kafka Producers and Consumers from scratch implementing organization’s requirements
• Responsible for creating, modifying and deleting topics (Kafka Queues) as and when required wif varying configurations involving replication factors, partitions and TTL.
• Designed and developed tests and POC’s to benchmark and verify data flow through the Kafka clusters.
• Validated the performance of Hive queries on Spark against running them traditionally on Hadoop
• Involved in Testing and coordination wif business in User testing.
• Importing and exporting data into HDFS and Hive using Sqoop.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate TEMPeffectiv querying on the log data
• Involved in creating Hive tables loading data and writing queries dat will run internally in MapReduce way.
• Used Pig as ETL tool to do transformations, event joins, filter and some pre-aggregations.
• Involved in processing ingested raw data using MapReduce, Apache Pig and HBase.
• Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
• Populated HDFS and HBASE wif huge amounts of data using Apache Kafka.
• Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs.
• Used Hive to analyze the partitioned and bucketed data to compute various metrics for reporting.
• Experienced in managing and reviewing the Hadoop log files.
• Expertise wif NoSQL databases like HBase and MongoDB.
• POC work is going on using Spark,STORM and Kafka for real time processing.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs,Python and Scala.
• Used Spark-SQL for querying and quick data analysis.
• Real time streaming the data using Spark wif Kafka.
• Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala
• Design technical solution for real-time analytics using Kafka and HBase.
• Gained noledge in creating QlikView and Tableau dashboards for reporting analyzed data.

Confidential

Team Member

Responsibilities:

• Analysis and understanding the HLD
• Involved in preparing the LLD
• Preparation of UTP and Unit test cases
• Participation in making the code changes
• Executing the Unit test cases and obtaining the unit test results
• Peer Reviews
• Promoting the components to system testing
• Supporting the system testing team
• Promoting the components from Dev to Production wif proper approvals
• Analysis the Defect change request/Live Defect when received
• Migrating systems written in COBOL to Java in order to reuse Java programming skills and class libraries.
• Employing long-running started tasks for Java dat periodically query a database to find new work to process.
• Using Java programs to access APIs such as SOAP/Web services, WebSphere MQ client API, Java Database Connectivity (JDBC) databases, custom Transmission Control Protocol/Internet Protocol (TCP/IP) socket services, and so forth.
• Passing datasets created by traditional job steps to Java programs, which convert the data to XML.
• Flexible configuration of the Java Virtual Machine (JVM) and environment variables.
• Routing output directly to JES SYSOUT datasets
• Controlling output encoding.
• Passing condition codes between Java and non-Java steps
• Reading and writing MVS datasets from Java
• Executing the JVM under the original batch address space.
• Communicating wif the MVS system console
• Running Java batch jobs wif BPXBATCH and Running Java batch jobs wif a custom JVM launcher.

Confidential

Team Member

Responsibilities:

• Analysis and understanding the HLD
• Involved in preparing the LLD
• Preparation of UTP and Unit test cases
• Participation in making the code changes
• Executing the Unit test cases and obtaining the unit test results
• Promoting the components to system testing
• Supporting the system testing team.
• Promoting the components from Dev to Production wif proper approvals
• Analysis the Defect change request/Live Defect which received from vantive.
• Involved in Design, Development, Testing and Integration of the application.
• Involved in development of user interface modules using HTML, CSS and JSP.
• Involved in writing SQL queries
• Involved in coding, maintaining, and administering Servlets, and JSP components to be deployed on Apache Tomcat application servers
• Database access was done using JDBC. Accessed stored procedures using JDBC.
• Worked on bug fixing and enhancements on change requests.
• Coordinated tasks wif clients, support groups and development team.
• Worked wif QA team for test automation using QTP
• Participated in weekly design reviews and walkthroughs wif project manager and development teams.
• Migrating systems written in COBOL to Java in order to reuse Java programming skills and class libraries.
• Employing long-running started tasks for Java dat periodically query a database to find new work to process.
• Using Java programs to access APIs such as SOAP/Web services, WebSphere MQ client API, Java Database Connectivity (JDBC) databases, custom Transmission Control Protocol/Internet Protocol (TCP/IP) socket services, and so forth.
• Passing datasets created by traditional job steps to Java programs, which convert the data to XML.
• Flexible configuration of the Java Virtual Machine (JVM) and environment variables
• Routing output directly to JES SYSOUT datasets
• Controlling output encoding
• Passing condition codes between Java and non-Java steps
• Reading and writing MVS datasets from Java
• Executing the JVM under the original batch address space
• Communicating wif the MVS system console
• Running Java batch jobs wif BPXBATCH and Running Java batch jobs wif a custom JVM launcher.