PROFESSIONAL SUMMARY:

• Over 8 years of professional experience involving project development, implementation, deployment and maintenance using Java/J2EE, Hadoop and Spark related technologies using Cloudera, Hortonworks.
• Hadoop Developer with 5 years of working experience in designing and implementing complete end - to-end Hadoop based data analytical solutions using HDFS, MapReduce, Spark, Yarn, Kafka, PIG, HIVE, Sqoop, Storm, Flume, Oozie, Impala, HBase etc.
• Experience in installation, configuration, management and deployment of Hadoop Cluster, HDFS, Map Reduce, Pig, Hive, Sqoop, Apache Storm, Flume, Oozie, HBase and Zookeeper.
• In-depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Map Reduce, Spark.
• Hadoop related eco-systems as a Data Storage and Retrieval systems.
• Good experience in creating data ingestion pipelines, data transformations, data management, data governance and real time streaming at an enterprise level.
• Profound experience in creating real time data streaming solutions using Apache Spark/Spark Streaming, Kafka.
• Good knowledge on Spark Ecosystem and Spark Architecture.
• Experience developing Pig Latin and HiveQL scripts for Data Analysis and ETL purposes and extended the default functionality by writing User Defined Functions (UDFs), (UDAFs) for custom data specific processing.
• Good Hands-on full life cycle implementation using CDH (Cloudera) and HDP (Hortonworks Data Platform) distributions.
• Strong Knowledge on Architecture of Distributed systems and parallel processing, In-depth understanding of MapReduce programing paradigm and Spark execution framework.
• Profound understanding of Partitions and Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
• Experience in handling messaging services using Apache Kafka.
• Experience with migrating data to and from RDBMS into HDFS using Sqoop.
• Experience in job workflow scheduling and monitoring tools like Oozie and Zookeeper.
• Worked on NoSQL databases including HBase, Cassandra and Mongo DB.
• Strong experience in collecting and storing stream data like log data in HDFS using Apache Flume.
• Experience in working with Java HBase API for ingestion processed data to HBase tables.
• Experience with Oozie Workflow Engine to automate and parallelize Hadoop Map/Reduce, Hive and Pig jobs.
• Experienced with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Extensive knowledge in programming with Resilient Distributed Datasets (RDDs)
• Proficient in using Cloudera Manager, an end-to-end tool to manage Hadoop operations in Cloudera Cluster.
• Assisted in Cluster maintenance, Cluster Monitoring, Managing and Reviewing data backups and log files.
• Experience in Extraction, Transformation and Loading (ETL) of data from multiple sources like Flat files, XML files and Databases.
• Extensive experience in ETL process consisting of data transformation, data sourcing, mapping, conversion and loading in Talend.
• Used Talend for ETL processing based on business needs and extensively used Oozie workflow engine to run multiple Hive and Pig jobs.
• Experience with Talend and Informatica/Data Exchange.
• Solid experience in developing workflow using Oozie for running Map Reduce jobs and Hive Queries.
• Experience in managing and reviewing Hadoop log files.
• Responsible in performing advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
• Experience with Apache Solr in replication, distribution, rebalancing and fault tolerance out of the box.
• Experience in architectural patterns like Apache Lucene search development, full-text search development, cross-platform, High Performance Indexing and ranked searching.
• Good experience in working with cloud environment like Amazon Web Services (AWS) EC2 and S3.
• Experienced in Java Application Development, Client/Server Applications, Internet/Intranet based applications using Core Java, J2EE patterns, spring, Hibernate, Struts, JMS, Web Services (SOAP/REST), Oracle, SQL Server and other relational databases.
• Profound knowledge on core Java concepts like Exceptions, Collections, Data-structures, I/O. Multi-threading, Serialization and deserialization.
• Experience writing Shell scripts in Linux OS and integrating them with other solutions.
• Expert at creating UML diagrams Use Case diagrams, Activity diagrams, Class diagrams and Sequence diagrams using Microsoft Visio and IBM Rational Rose.
• Good experience in development of software applications using Core Java, JDBC, Servlets, JSPs, Spring and RESTful Web Services.
• Experience in using PL/SQL to write Stored Procedures, Functions and Triggers.
• Excellent technical and analytical skills with clear understanding of design goals of ER modeling for OLTP and dimension modeling for OLAP.
• Experience in Requirements Gathering/Analysis, Design, Development, Versioning, Integration, Documentation, Testing, Build and Deployment.
• Efficient in packaging & deploying J2EE applications using ANT, Maven & Cruise Control on WebLogic, WebSphere & JBoss.
• Experience in using Jenkins and Maven to compile the package and deploy to the Application Servers.
• Deployment, Distributed and Implementation of Enterprise applications in J2EE environment
• Good Understanding of bootstrap, spring rest and integration.
• Strong Knowledge of Version Control Systems like SVN, GIT & CVS.
• Familiar with multiple software systems, ability to learn quickly new technologies, adapt to new environments, self-motivated, team player, focused, adaptive and quick learner with excellent interpersonal, technical and communication skills.

TECHNICAL SKILLS:

Big Data Technologies: Hadoop 1.x/2.x(Yarn), HDFS, MapReduce, Pig, Hive, HBase, Cassandra, Zookeeper, Oozie, Sqoop, Flume, HCatalog, Apache Spark, Scala, Impala, Kafka, Storm, Tez, Ganglia, Nagios, Splunk, Elastic Search, Kibana

Hadoop Distributions: Cloudera, Horton Works, AWS

Operating Systems: Windows, Macintosh, Linux, Ubuntu, Unix, CentOS, Redhat.

Programming Languages: C, JAVA, J2EE, SQL, PigLatin, HiveQL, Scala, Python, Unix Shell Scripting

Java Technologies: JSP, Servlets, Spring, Hibernate, Maven

Databases: MS-SQL, Oracle, MS-Access, NoSQL, MySQL

Reporting Tools/ETL Tools: Tableau, Informatica, Data stage, Talend, Pentaho, Power View

Methodologies: Agile/Scrum, Waterfall, DevOps

Development Tools: Eclipse, NetBeans, IntelliJ, Hue, Microsoft Office Suite (Word, Excel, PowerPoint, Access)

PROFESSIONAL EXPERIENCE:

Confidential, Longwood, FL

Spark/Scala Developer

Responsibilities:

• Handled importing of data from various data sources, performed data control checks using Spark and loaded data into HDFS.
• Worked on the Spark SQL for analyzing the data.
• Used Scala to write code for all Spark use cases.
• Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Pair RDD'S, YARN.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD'S and Scala.
• Worked in Spark SQL on different data formats like JSON and Parquet.
• Developed Spark scripts by using Scala shell commands as per the requirement.
• Used Spark API over Cloudera Hadoop YARN to perform analytics on data in HDFS.
• Load the data into Spark RDD and performed in-memory data computation to generate the output response.
• Familiarity with Hadoop open source stack including Yarn, Kafka, Hive
• I have been experienced with KAFKA to ingest data into Spark engine
• Worked on streaming pipeline that uses Spark to read data from Kafka, transform it and write it to HDFS.
• Extensive experience in using the mom with active MQ, apache storm, apache Spark & Kafka maven and zookeeper.
• Developed Kafka producer and consumers, HBase clients, Spark and Hadoop MapReduce jobs along with components on HDFS, Hive.
• Developed MapReduce (YARN) programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
• Designed and developed Map Reduce jobs to process data coming in different file formats like XML, CSV, JSON.
• Developed workflows using Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
• Involved in preparing JIL's for AutoSys jobs.
• Used Sqoop to import the data from RDBMS to Hadoop Distributed File System (HDFS) and later analyzed the imported data using Hadoop Components.
• Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries.
• Used Oozie for automating the end to end data pipelines and Oozie coordinators for scheduling the work flows.
• Implemented daily workflow for extraction, processing and analysis of data with Oozie.
• Hadoop installation & configuration of multiple nodes on AWS EC2 system
• Implementing Hadoop with the AWS EC2 system using a few instances in gathering and analyzing data log files.
• Implemented a Continuous Delivery pipeline with Docker, Jenkins and GitHub, Nexus, Maven and AWS.

Environment: Spark - 1.5.2, Spark SQl, Java 1.8, Hive, HDFS, HQL, YARN, HBase, MapReduce, Sqoop, Flume, Oozie, Kafka, Scala, AWS Oracle 12c.

Confidential, Chicago, IL

Hadoop Developer

Responsibilities:

• Worked on live 60 nodes Hadoop Cluster running CDH5.4.4, CHD5.2.0, CDH5.2.1
• Worked on Hadoop cluster using different Bigdata analytic tools including Kafka, Pig, Hive and Map Reduce.
• Developed simple to complex MapReduce streaming jobs using Python language that are implemented using Hive and Pig.
• Implemented data access jobs through Pig, Hive, HBase (0.98.0), Storm (0.91)
• Involved in loading data from LINUX file system to HDFS
• Importing and exporting data into HDFS and Hive using Sqoop.
• Altered existing Scala programs to enhance performance and obtain partitioned results Spark tool.
• Used SparkSQL for Scala &, Python interface that automatically converts RDD case classes to schema RDD.
• Used SparkSQL to read and write table which are stored in Hive.
• Involved in importing the real time data to Hadoop using Kafka and implemented the Oozie job daily.
• Involved in developing Hive DDLs to create, alter and drop Hive tables and storm, & Kafka.
• Experienced in transferring data from different data sources into HDFS systems using Kafka producers, consumers and Kafka brokers
• Experience in data migration from RDBMS to Cassandra.
• Created data-models for customer data using the Cassandra Query Language.
• Experienced in developing Spark scripts for data analysis in both python and Scala.
• Worked on processing unstructured data using Pig and Hive.
• Collected and aggregated large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
• Used Impala to read, write and query the Hadoop data in HDFS or HBase.
• Involved in scheduling Oozie workflow engine to run multiple Hive and Pig jobs.
• Developed Pig Latin Scripts to extract data from the web server output files to load into HDFS.
• Responsible in taking backups and restoration of Tableau repository.
• Converted ETL operations to Hadoop system using Pig Latin operations, transformations and functions.
• Experience in Talend migration project from one version to another is preferred.
• Worked on majority of Talend components and can design simple ETL Jobs to handle complex Business Logic.
• Knowledge of error handling and Performance tuning in Talend and SQL.
• Created and maintained technical documentation for launching Hadoop clusters and for executing Hive queries and Pig Scripts.
• Exported the result set from Hive to MySQL using Shell Scripts.
• Actively involved in code review and bug fixing for improving the performance.
• Experience with Cassandra (DataStax distribution preferred)
• Collaborate with development teams on architecture and deployment of NoSQL database systems like Cassandra

Environment: Hadoop, HDFS, Pig, Hive, Map Reduce, Sqoop, Storm, Kafka, LINUX, Hortonworks distribution, Bigdata, Java APIs, Java collection, SQL, NoSQL, MongoDB, Cassandra.

Confidential, Malvern, PA

Hadoop Administrator/Developer

Responsibilities:

• Responsible for installation, configuration, maintenance, monitoring, performance tuning and troubleshooting Hadoop Clusters in different environments such as Development Cluster, Test Cluster and Production.
• Used Job Tracker to assign MapReduce tasks to Task Tracker in cluster of nodes.
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and extracted the data from MySQL into HDFS using Sqoop.
• Processed HDFS data and created external tables using Hive and developed scripts to ingest and repair tables that can be reused across the project.
• Implemented Kerberos security in all environments.
• Defined file system layout and data set permissions.
• Implemented Capacity Scheduler to share the resources of the cluster for the MapReduce jobs given by the users.
• Responsible for troubleshooting issues in the execution of MapReduce jobs by inspecting and reviewing log files.
• Involved in loading data from Linux and Unix file system to HDFS.
• Involved in emitting processed data from Hadoop to relational databases or external file systems using Sqoop, HDFS GET or Copy to Local.
• Involved in Cluster planning and setting up the multimode cluster.
• Commissioned and Decommissioned nodes from time to time.
• Involved in HDFS maintenance and administering it through HDFS-Java API.
• Worked with Hadoop developers and designers in troubleshooting MapReduce job failures and issues.

Environment: Hadoop 1.2.1, MapReduce, HDFS, Pig, Hive, Sqoop, Cloudera Hadoop Distribution, HBase, Windows NT, LINUX, UNIX Shell Scripting.

Confidential

Hadoop Administrator/Developer

Responsibilities:

• Installed and configured Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
• Imported and exported data into HDFS from Oracle database and vice versa using Sqoop.
• Installed and configured Hadoop Cluster for major Hadoop distributions.
• Used Hive and Pig as an ETL tool for event joins, filters, transformations and pre-aggregations.
• Created partitions, bucketing across state in Hive to handle structured data.
• Developed workflow in Oozie to orchestrate a series of Pig scripts to cleanse data such as removing personal information or merging many small files into a handful of very large, compressed files using Pig pipelines in the data preparation stage.
• Involved in moving all log files generated from various sources to HDFS for further processing through Kafka, Flume & and process the files by using Piggybank.
• Extensively used PIG to communicate with Hive using HCatalog and HBase using Handlers.
• Performed Sqooping for various file transfers through the HBase tables for processing of data to several NoSQL DBs- Cassandra, MangoDB.
• Created tables, secondary indices, join indices viewed in Teradata development environment for testing.
• Captured data logs from web server into HDFS using Flume & for analysis.
• Managed and reviewed Hadoop log files.

Environment: Hive, Pig, MapReduce, Sqoop, Oozie, Flume, Kafka, Storm, HBase, Unix, Linux, Python, SQL, Hadoop 1.x, HDFS, GitHub, Talend, Python Scripting.

Confidential

Java Developer

Responsibilities:

• Involved in development of JavaScript code for client-side validations.
• Developed the HTML based web pages for displaying the reports.
• Developed front-end screens using JSP, HTML, jQuery, JavaScript and CSS.
• Performed data validation in Struts from beans and Action Classes.
• Developed dynamic content of presentation layer using JSP.
• Accessed stored procedures and functions using JDBC Callable statements.
• Involved in designing use-case diagrams, class diagrams and interaction using UML model with Rational Rose.
• Implemented Hibernate to persist the data into Database and wrote HQL based queries to implement CRUD operations on the data.
• Developed coding using SQL, PL/SQL, Queries, Joins, Views, Procedures/Functions, Triggers and Packages.
• Developed Web Applications with rich internet applications using Java applets, Silverlight, Java.
• Used JDBC for database access.
• Played a key role in the high-level design for the implementation of the application.
• Designed and established the process and mapping the functional requirement to the workflow process.

Environment: Java, Servlets, Java Beans, JSP, EJB, J2EE, STRUTS, XML, XSLT, JavaScript, HTML, CSS, Spring 3.2, SQL, PL/SQL, MS Visio, Eclipse, JDBC, Windows XP.