SUMMARY:

• 14 years of Experience in architectural design and development in Hadoop eco - system along with Java, Scala and Python technologies at Retail,Financial.
• Extensively having 10 years experience at Retail,Insurance and Healthcare Domains.
• Vast exposure in analyzing in the application bottlenecks or issues and propose the resolutions to get rid of it.
• Strong in documenting the architecture design / use cases / solution recommendations, etc
• Worked in building application platforms in the Cloud by leveraging AWS, Azure, Databricks, and open source technologies and with best engineering practices.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Leveraging Azure Synapse to build analytics ML model data pipeline
• Robot built using Qualcomm API & Tensor Flow object detection to navigate the robot through tight spaces with Autonomous and TeleOp mode process.
• Defining implementation plan for migrating the existing legacy applications to adopt PAAS (Auto Scaling, Elastic Beanstalk), Containers(Docker), Containers Orchestration (Kubernetes) andAWS Cloud Serverlessarchitecture.
• Hands on experience in creating API/Micro service applications using Spring Boot framework by adopting Micro services patterns, AWS Cloud, Hibernate, JWT and Angular JS.
• Migration of existing Batch and Workflow applications to Serverless applications using AWS Services such as Step Functions, Lambda, APIGatewayetc.
• Deploying highly available and scalable applications using PAAS (Elastic Beanstalk) and Container Orchestration, methodologies (Docker /Kubernetes) by following the AWS well architecture framework and adopting best practices in Continuous Integration and Delivery process.
• Integration Architecture design and development of payment system with various other insurance applications using Web Services and Web Sphere MQ /JMS.
• Worked with Kubernetes to make deployment more robust and to execute ETL pipelines using Spark on Kubernetes.
• Expertise in Java/J2EE development along with Business Process Management (BPM),SOA,Micro Services
• Thorough knowledge in core Java concepts like OOP, JDBC, JMS, Multi-Threading, JUnit and advanced Java concepts like JSP, Servlets, HTML, XML, CSS, Hibernate, JPA and Spring.
• Digital experience to build webapp and mobile app using AngularJS,ExtJS with back end Microservices
• Very good experience in evaluating, installation, configuration, management and deploymet of Hadoop Cluster and its components along with security (kerberize/ssl/tls/encryption/AD) enabling.
• Extensive knowledge in design, architecting, analyzing and implementing and building end-to-end ETL pipeline in all the phases of life cycle.
• Well versed in using software development methodologies like Agile (SCRUM), Waterfall and Test Driven Development and also worked in various fast-paced agile development environments.
• Demonstrated ability to handle multiple projects/tasks and leading the team. Good self-starter and enthusiastic in learning new technologies and adapt to new environment quickly. Had a good ability to think and work creatively and analytically in a problem-solving and challenging environments
• Very keen in to doing various POC’s suggest various approaches and also list out pros and cons of approaches along with suggesting right technologies and platforms. Had a very good ability to perform detailed analysis of business problems and technical environments and use this in designing the solution
• Excellent communication, analytical, presentation and interpersonal skills with the ability to work with leadership and business users/client. Ease in working with team of employees with various disciplines.

TECHNICAL SKILLS OVERVIEW:

Programming Lang.: Java, Scala, PL/SQL, Python

Big Data Ecosystem: ProcessingSpark (SQL, Streaming, Machine Library, DataFrame, DataSet, RDD API, GraphX), Flink, Kafka, Sqoop Distributions / Cloud AWS, Azure Cloudera, HortonWorks, MapR,Databricks Container OrchestrationDocker, Kubernetes ML platforms Spark, TensorFlow Pipeline / Data flow Streamsets, NiFi, CDAP, Cloudera Navigator Frameworks Akka (Core, Http, Stream, JWT), Scalatest, mockito, playtests, Zookeeper, etc RDBMS / Storage / Oracle, MySQL, DB2, SQL ServerNoSQL / SearchHive, Impala, MongoDB, HBase, Solr Notebooks Zepplin, Jupyter, Cloudera Data Science Workbench (CDSWS), Livy Logging ELK Logging (Elasticsearch, Logstash, Kibana), Splunk

UI / Web: JQuery, Angular JS, Ext JS, D3 JS, ReactJS

J2EE / Other Stack: Java / J2EE (JSP, Servlets, EJB, Spring, Struts, ORM - IBatis / Hibernate, Web Services - SOAP / Restful)

Application Servers: Websphere, Oracle Apps, WebLogic, Pramati, Tomcat, JBOSS, IIS, Jetty

Scripting: NodeJS, Bash, Python, ANT

Source Control: Git/ Bitbucket /Git lab, Rational Clear Case, CVS and SVN, Tortoise

Build /other tools: Maven, Gradle, SBT, Jenkins, Autosys, RAD, IntelliJ, Jdeveloper, Eclipse, Bladelogic

Operating System: Windows, Ubuntu, CentOs, RedHat.

PROFESSIONAL EXPERIENCE:

Confidential, Austin, TX

Sr. Cloud Engineer

Responsibilities:

• Architecture decisions are made around Resource Group management and Azure Components
• Expertise in Architecting and Implementing Azure Service Offering, such as Azure cloud services, Azure storage, IIS, Azure Active Directory (AD), Azure Resource Manager (ARM), Azure Storage, Azure, Blob Storage, Azure VMs, SQL Database, Azure Functions, Azure Service Fabric, Azure Monitor, and Azure Service Bus.
• Architecture decisions are made for supporting functions and resources (networking, language, service plan for functions, Databricks metastore, VM creation)
• Architecture decisions are made for how the Stage environment will be managed (duplicates, folder structures, data typing, etc.)
• Implemented a CI/CD pipeline using Azure DevOps in both cloud and on-premises with GIT, Maven along with Jenkins plugins.
• Experience in writing Infrastructure as a code (IaC) in Terraform, Azure resource management. Created reusable Terraform modules in both Azure and AWS cloud environments.
• Data Center Operations: Evaluate processes and infrastructure design to more efficiently manage operations and reduce costs while increasing predictability
• Enterprise Directories: Deploy enterprise-wide business directories to lower administrative costs, coordinate end-user accounts, and reduce development costs
• Detailed understanding of Azure database offerings Relational, NoSQL, Datawarehouse
• Deep familiarity with Azure Security Services Azure Active Directory, RBAC, KeyVault, ADFS
• Architecture and Implementation experience with medium to complex on-prem to Azure migrations
• Implementation experience with Data Lakes and Business Intelligence tools in Azure
• Expertise in building Azure native enterprise applications and migrating applications from on-premises to Azure environments Database/Application health and performance monitoring experience using services such as OMS and application insight
• Demonstrated ability to architect and deliver scalable enterprise solutions combining various Azure services
• I provide information solutions as a member of a strategic data architecture team
• I've created solutions that use a combination of appropriate technologies like Azure SQL, Data Factory, Event Hub, Cosmos DB, Azure Functions, Event Grid, or simple Storage Accounts to deliver data to Power BI and other reporting systems.
• Data sources to be ingested are prioritized
• Building Pipelines using PySpark with Multitenancy architecture in the Databricks platform

Technologies: PySpark, Scala, Azure Databricks, Azure DevOps, Azure Synapse, Azure Cosmos DB, Azure Data Factory, Azure Data Lake, Snowflake,Tableau,PowerBI,Alteryx

Confidential, Mason, OH

Sr. Cloud Engineer

Responsibilities:

• Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines and also managing the development team
• Created Pipelines inADFusingLinked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources likeAzure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that processes the data using the Cosmos Activity.
• Leveraging Azure Synapse to build analytics ML model data pipeline
• Azure Bot Service and Bot Framework provide tools to build, test, deploy, and manage intelligent bots, all in one place.
• Developed Spark applications usingScalaandSpark-SQLfor data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Responsible for estimating the cluster size, monitoring and troubleshooting of the Hadoop cluster.
• UsedZeppelin,Jupyternotebooks andSpark-Shellto develop, test and analyze Spark jobs before Scheduling Customized Spark jobs.
• Creating development builds and deployables and also helping in setting up the environments like Production, Development, Integration, etc
• Extensively worked on Spark execution tuning to improve the performance of complete application.
• Developed various security utilities and also generic utilities to launch jobs for Impala, Spark, etc from API.
• Heavily usage of Spark Dataframe, Dataset and RDD API.
• Creating reusable components or Utilities
• Parsing the Mainframes ebcdic and copybooks from Spark and saving into Hive and HBase
• Built configurable utility from config file to read the data from Hive or HDFS or apply transformations specified and save the data to HDFS or HBase or Kafka, etc
• Configurable component to read Kafka data from Flink or Kstream or Flume and apply transformations or ML pipeline and place the data back to HDFS or Kafka or Hive.
• Created connector to HBase from Spark dataframes to put get and scan functionalities
• Configuring and launching the Spark jobs from ControlM
• Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.

Technologies: Spark/Scala, PySpark, Spark SQL, Kafka and Hive, Mongo DB, Java, Azure Blob, Azure Synapse, Azure Cosmos DB, Azure Data Factory, Azure Data Lake, Azure Databricks, Azure Event Hubs Azure AI, Azure ML, Bot Framework Emulator

Confidential, Blue Ash, OH

Sr.Cloud Engineer

Responsibilities:

• Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines and also managing the development team
• Worked in building application platforms in the Cloud by leveraging AWS, Azure Databricks
• Design and Migration Activities from on-prem to AWS Cloud environment.
• Responsible for Designing and configuring Network Subnets, Route Tables, Association of Network ACLs to Subnets and Open VPN.
• Designed AWS Cloud Formation templates to create VPC, subnets, NAT to ensure successful deployment of Web applications and database templates.
• Enhance the existing product with newly features like ELB, Auto scaling, S3, Cloud Watch, Cloud Trail and RDS-Scheduling.
• Backup and Recovery, Replication activities from on-prem to cloud.
• Act as technical liaison between customer and team on all AWS/BigData technical aspects.
• Creating development builds and deployable and also helping in setting up the environments like Production, Development, Integration, etc
• Extensively worked on Spark execution tuning to improve the performance of complete application.
• Configuring and launching the Spark jobs from ControlM
• Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.

Technologies: Cloudera, PySpark, Spark SQL, Scala/Python, Java, Impala, HBase, Hive, Jetty, Kafka, AWS Cloud, Azure Databricks

Confidential, Blue Ash, OH

Sr.Cloud Engineer

Responsibilities:

• Architectural discussions, design, implementation, etc
• Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines.
• Interactions with management on the goals and its tracking
• Discussion with Clients(ember clients) and their needs
• Azure cloud enablement for on premises Big data platform application into Cloud.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Managing and guiding the development team along with few implementations and helping them on POC’s like below
• Develop Spark 2.1/2.4 Scala component to process the business logic and store the computation results of 10 TB data into HBase database to access the downstream web apps using Big SQL db2 database.
• Worked in building application platforms in the Cloud by leveraging Azure Databricks
• Created Pipelines inADFusingLinked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources likeAzure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that processes the data using the Cosmos Activity.
• Used Spark API to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from upstream in near real time and Persists into HBase.
• Working with different Hive file formats like Text file, Sequence file, ORC file, Parquet and Avro to analyze the data to build data model
• Worked extensively develop Spark/Scala/BigSQL framework to Process and Persist
• Customize BIGSQL Load Hadoop component Insert/Update Spark Data Frame/Dataset into HBase.
• Integrate Spark Jobs with TWS for Jobs scheduling, Develop spark Jobs shell scrips
• Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.
• Developed and Implement HBase capabilities for Big de-normalized data set and then apply transformation on the de-normalized data set using Spark/Scala
• Developed and Implement Spark ETL custom component to extract the data from upstream systems and push the data to HDFS and finally store the data in HBase with wide row format.
• Uploaded Inbound data from Different source s to HDFS and HBase
• Implement and Develop Order Price component using IBM BigSQL

Technologies: HDP, Azure Cloud, Spark 2.4, Scala, Java, HBase, BigSQL, IBM DB2,Java, Hive, Dynatrace, Pepper Data Tool,SQL,Spring Boot,IntelliJ

Confidential, GA

Sr.Cloud Engineer

Responsibilities:

• Architecting, design and development of application and effectively contributing end to end involvement and also managing development teams.
• Doing various POC’s and propose pros and cons of various approaches.
• Creating development builds and deployables and also helping in setting up the environments like Production, Development, Integration, etc
• Extensively worked on Spark execution tuning to improve the performance of complete application.
• Worked on building the REST custom interface to Spark and other components using LIVY and Databricks for job invocation in Microsoft Azure platform
• Worked on building the custom logging wrapper over Splunk for Analyzing, Alerting and monitoring the jobs
• Architectural discussions, design, implementation, etc
• Exploring or doing POC’s to suggest on the right approach to Business team, Clients and development teams
• Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines.
• Interactions with management on the goals and its tracking
• Discussion with Clients(ember clients) and their needs
• Managing and guiding the development team along with few implementations and helping them on POC’s like below
• Develop Spark 2.1/Scala component to process the business logic and store the computation results of 10 TB data into HBase database to access the downstream web apps using Big SQL db2 database.
• Used Spark API to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from upstream in near real time and Persists into HBase.
• Working with different Hive file formats like Text file, Sequence file, ORC file, Parquet and Avro to analyze the data to build data model
• Worked extensively develop Spark/Scala/BigSQL framework to Process and Persist
• Customize BIGSQL Load Hadoop component Insert/Update Spark Data Frame/Dataset into HBase.
• Integrate Spark Jobs with TWS for Jobs scheduling, Develop spark Jobs shell scrips
• Hands on experience in setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobs in Talend TAC through wrapper scripts
• Design and Develop web app using Restful,AngularJS,Spring Boot,Datastax API
• Build and Design UI screens using AnjularJs, interact with UX team and coordinate with business users
• Code reviews using GitHub and build, deployment using team city

Technologies: HDP, Spark2.1, Scala, HDFS,YARN,HBase, BigSQL, IBM DB2, Map Reduce, Java, Hive, Dynatrace, Pepper Data Tool,SQL,Spring Boot,IntelliJ,AngularJS 1.6,JAX-RS,HTML 5.0,BootStrap,Git,Team City

Confidential, MN

Responsibilities:

• Architectural discussions, design, implementation, etc
• Exploring or doing POC’s to suggest on the right approach to Business team, Clients and development teams
• Business needs or requirement discussions with the Business users to come up with effective architectural design and timelines.
• Interactions with management on the goals and its tracking
• Discussion with Clients(ember clients) and their needs
• Managing and guiding the development team along with few implementations and helping them on POC’s like below
• Spark session cluster aware inside the docker container and also providing impersonation for the users logged in based on their roles using TTL cache.
• Worked with Kubernetes to make Optum deployment more robust and to execute ETL pipelines using Spark on Kubernetes.
• Code enhancements to improve maintainability including creating the code in patterned approach - DAO, services and REST Controllers, etc.
• Wrote many generic or reusable components along with logging wrapper generic exception handling, etc
• Testing and validation of various integration's to cloud (AWS) and big data distributions (MapR, Hortonworks)
• Architecture discussions / proposing new ideas to enhance the Ember functionality to improve development and deployment
• Developed and Implement HBase capabilities for Big de-normalized data set and then apply transformation on the de-normalized data set using hive.
• ELDM and ECMM Jobs developed using Spark RDD and Data Frame
• Developed a data pipeline using Kafka and Storm to store data into HDFS.
• Uploaded streaming data from Kafka to HDFS, HBase and Hive by integrating with storm.
• Integration Spark with Mark Logic for MLCP jobs to push the documents.
• Using Splunk to visualization of Dashboards based on MLCP Logs
• Using Talend to integrate with Hive/Map Reduce Jobs to Schedule Jobs

Technologies: MapR, AWS, Hadoop, HDFS, HBase, Apache Spark 1.6, Hive, Pig, Scala, Sqoop, SQLEclipse, Zookeeper, Java, JAXB, Scala 2.10, Talend 5.1, Mark Logic, Oozie, Splunk

Confidential, Columbus, OH

Responsibilities:

• Building generic components or pipeline like
• Machine learning pipeline
• Exporting the ML models to JPMML file and reading back to creating models
• Reading or invoking R file using RServe from Spark and creating Spark dataframes, etc
• Extensively working on integrating the components like Kafka, Flume, Hbase, Solr, Spark, HBase
• Heavily usage of Spark Dataframe, Dataset and RDD API.
• Creating reusable components or Utilities
• Parsing the Mainframes ebcdic and copybooks from Spark and saving into Hive or HBase
• Built configurable utility from config file to read the data from Hive or HDFS or apply transformations specified and save the data to HDFS or HBase or Kafka, etc
• Configurable component to read Kafka data from Flink or Kstream or Flume and apply transformations or ML pipeline and place the data back to HDFS or Kafka or Hive.
• Created connector to HBase from Spark dataframes to put get and scan functionalities
• Configuring and launching the Spark jobs from Spark Launcher and Livy
• Worked on Spark GraphX and pregel api
• Setting up and working on Zeppelin, Jupyter and Ipython notebook
• Wrote testing and exception components utilities like
• Embedded Kafka or Mini HBase
• Unit test cases using Scalatest framework, Akka HTTP, etc
• Explored on various pipeline tools like NiFi, CASK- CDAP, Streamsets
• Used Zeppelin, Jupyter notebooks and Spark-Shell to develop, test and analyze Spark jobs before Scheduling Customized Spark jobs.

Technologies: Kafka, Flume, Spark sink, HBase, Apache Solr, Spark - Machine Learning, Steaming, Spark SQL, DataSet, DataFrame, RDD API, Scala, Java, Flink, NiFi, CASK- CDAP, Streamsets, HBase, H20, Zeppelin, Jupyter and Ipython notebooks, etc

Confidential, NY

Responsibilities:

• Architecting and designing the application as per suggestions and requirements from business team and application team.
• Designing and working on models / components for Proof of concepts (POC's)
• Working with Cloudera support team, in case of any issue.
• Discussing with Business team and Technical partners
• Responsible for writing Hive and Mapreduce programs to replace Netezza and Teradata queries.
• Created Oozie workflow for automating the process using Jenkins and Autosys.
• Importing and exporting data into HDFS using Sqoop from various like Teradata, Oracle, Netezza, etc
• Involved in creating the reusable components (for transformations and aggregations) like Universal key component, Data Quality, Data Integrity, etc using some frameworks like PIG, HIVE, Mapreduce, Spark etc.
• Performed DQ and DI checks on the Sqoop data before saving it into HDFS, Hive tables
• Coded the UDF, UDAF and UDTF in the HIVE queries based on the business requirement.
• Worked extensively with Avro, parquet, ORC, JSON file formats.
• Exposing the Hive views to Impala queries and Creating the UI portal from Impala views to present and generate reports using D3 JS, Jquery, Spring, JAX-RS web service, etc.
• Writing Hive and PIG UDFs for doing transformations and aggregations. Also writing custom Hive SerDes for reading semi-structured data.
• Writing MapReduce along with Combiners, Custom Partitions, Custom Splits, Custom Record Readers, Distribute cache, join and filtering
• Migrating the existing Map Reduce, PIG scripts and components to Spark components in Scala
• Developed Oozie workflows by integrating all tasks relating to a project and schedule the jobs as per requirements.
• Developed Shell scripts and schedule jobs on Cron to import data files from various sources to HDFS to archive
• Created multiple Hive tables, implemented Partitioning, Dynamic Partitioning and Buckets in Hive for efficient data access.
• Worked on Spark SQL and Data frames for faster execution of Hive queries using Spark SqlContext.
• Implemented Spark scripts using Scala and Spark SQL for faster processing of data.
• Involved in writing UNIX shell scripts, JAVA reusable components for development and automated testing.

Technologies: Hadoop - MapReduce, Hive, Pig, Oozie, Sqoop, Autosys scheduler, Spark - DSL, Spark SQL, Dataframes, Streaming, Scala, Java/J2EE, Spring, REST - Web Services, Jenkins, D3js, JQuery, AJAX, Tomcat, etc.

Confidential, NY

Technical Architect

Responsibilities:

• Responsible for writing Mapreduce programs to place DB data to HDFS and also NOSQL- MongoDB using Custom Record Reader and Input Format.
• Importing and exporting data into HDFS and Hive using Sqoop.
• Created Oozie workflow for automating the process and archiving the DB data.
• Wrote Hive queries to data analysis and reporting.
• Worked on implementing callback parameters for Hadoop job running and its status, worked on PathFilters to read the required on HDFS, worked on reading JSON data. Also, exposed in writing Combiners, Custom Partitions, Custom Splits, Custom Record Readers, Distribute cache, join and filtering.
• Configured Hive and also written Hive UDFs and also worked on reading JSON data using JSON SerDes.
• Exposing the MongoDB data as a Restful Web Services for reporting and also analyzing the data in NoSQL DB - MongoDB
• Designing the MongoDB schema including indexing and sharing of data.
• Working and maintaining Cloudera Hadoop cluster and also maintaining MongoDB DB cluster.
• Reading the log files using Elasticsearch Logstash and alerting users on the issue and also saving the alert details to MongoDB for analyzations.
• Worked on various POC’s like using Flume for reading logs files, analyzing the data using PIG, worked on various NoSQL DB’s like HBase that suits the requirement, processing the data using Spark

Confidential

Sr. Consultant

Responsibilities:

• Followed agile software development (Scrum) practice paired programming and test driven development.
• Installed and configured Hadoop Cluster for development and testing environment.
• Involved in consuming/Integrating the external Web Services application by generating stubs and interaction with DB by writing DAO classes.
• Developed generic UI framework/utility for making dynamic Ajax calls and consuming Web Services details to UI using Spring MVC, Jquery and DWR.
• Used Axis2 usage to prepare Web Services calls.
• Exported data from DB2 to HDFS using Sqoop.
• Developed mapreduce programs for applying business rules on the data.
• Developed and executed hive queries for denormalizing the data.
• Implemented fair scheduler on the Job tracker to share the resources of the cluster for the map reduces jobs.
• Automated the work flow using shell scripts.
• Created Hive queries to compare the raw data with Warehouse tables and performing aggregates
• Performance tuning of the hive queries, written by other developers.
• Defining the Oozie workflow to automate the tasks of loading the data into HDFS.
• Developed unit test cases using JUnit.
• Involved in security authentication, architectural design and code review.
• Testing and validating the changes to meet the requirements.

Technologies: Java/J2EE( EJB, Spring - MVC, AOP, Spring Batch, Web Services - SOAP, Restful), iBatis, Savvion, Blaze Advisor 6.9, HTML, JSON, jQuery, DWR, AJAX, Ext JS, DB-Oracle, Hadoop, Hive, Pig, Sqoop, etc

Confidential

Project Lead

Responsibilities:

• Worked on setting up the Hadoop cluster for the Environment for development and testing environment.
• Developed mapreduce jobs for applying business rules on the data.
• Developed Hive queries to pre-process the data for analysis and executed Hive queries for denormalizing the data.
• Creating and managing the instances, which includes no of CPUs, RAM and Disk storage required using Openstack

Technologies: Java/J2EE, Hadoop, Hive, HDFS, Openstack, Oracle