SUMMARY:

• Experienced Big Data Engineer/Architect with demonstrated results such as:
• Architect and build batch and real time applications with Lambda Architecture using Spark, Scala, Hive, Kafka, Hbase and Elastic Search
• Defined database technologies for ingestion, storage and processing according to different uses cases
• Data modeling in Big Data
• Developed performance monitoring and alerting using Grafana, Influx DB and OpenTsDB

PROFESSIONAL EXPERIENCE:

Lead Data Engineer/Architect

Confidential

Responsibilities:

• Architect and Build Merchant Big Data Platform as single source of truth for internal and external use cases
• Data Ingestion and transformation from Oracle, Teradata and third party platform into Hadoop through Kafka
• Developed end to end pipeline for Aggregates and Snapshot on consumer and payments datasets using Spark Scala framework, Spark SQL and Kafka
• Data persistence in Elastic Search Data store for faster query processing from Serving Layer.
• Build Lambda architecture on Spark Streaming for Webpage click stream datasets On conversion and fallouts
• Developed Monitoring/alerting Platform using Grafana, OpenTSDB on message events on Kafka and Elastic Search clusters
• Developed Disaster recovery plan and storage estimates on each stages on pipeline.

Senior Data Architect

Confidential

Responsibilities:

• Implemented Data Hub, MDM customer Master for BOW, collecting code value pair data sets and external systems like securities data as well in ODS
• Worked on establishing the Big Data standards, software architecture and functional specifications for development, Data quality, Retention, tuning and implementations
• Worked with business users in collecting requirements to translate into use case and design
• Responsible for Analysis, Design, Data modeling and BI report solutions using Spark, Spark SQL, Erwin, Oracle, Informatica, Microsoft and OBIEE

Senior Data Engineer/Architect

Confidential

Responsibilities:

• Worked on EDW and Capacity Planning projects, to forecast Cloud Platform infrastructure capability for Confidential transactions from Market Interest predictions data using Spark and python.
• Work on proof of concept and prototyping activities to measure the performance and scalability
• Implemented API based integration for data pipelines and data parsing using python library
• Work closely with the product management and development teams to rapidly translate the understanding of customer data and requirements to product and solutions.
• Working on Agile SDLC for deliverable on cloud platform services.

Hive, Pig, Spark, Spark SQL, Flume, Python, Java, Talend, Cassandra, SQL Server 2012, Erwin, RestApi, Web service and Tableau

Senior Data Engineer

Confidential

Responsibilities:

• Worked on Visa Enterprise Data Warehouse system project, responsible for design, model - ing and BI reporting solutions using Ab Initio 3.2.5, Hive, Pig, Hadoop streaming, Sqoop, Impala, Hbase, Flume
• Analyze multiple sources of structured and unstructured data to propose and design data architecture solutions for scalability, high availability, fault tolerance, and elasticity.
• Architect, Design and implement High performance large volume data processing using Ab Initio, Hive, Pig, Sqoop, Impala, Hbase, and Flume.
• Developed Map Reduce programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW
• Supported QA and UAT by preparing the environments, providing execution instructions, troubleshooting issues.
• Involved in troubleshooting performance issue and failures in all environments.

Map reduce, Hive, Pig, Oozie, Sqoop, Impala, Erwin, Hbase, Python, Java, Ab Initio GDE3.1.3, Micro strategy, Tableau, UNIX, IBM DB2 UDB, Oracle Exadata, Scrum

Lead ETL Developer

Confidential

Responsibilities:

• Worked in Merrill lynch Data Conversion and Data Management Team on Data Extraction, Fictionalization, Subset, Data Cleansing, and Data Validation
• Involved in all the stages of SDLC during the projects. Analyzed, designed and tested the new system for performance, efficiency and maintainability using ETL tool AB INITIO andDataStage
• Responsible for requirement gathering, analysis and development for data synchronization with Brokerage, Annuity and Sales NFS Trades data.
• Co-ordinate with different testing groups to accommodate their testing data requirements and translate them to data selection criteria in Ab-Initio and Data stage format
• Worked on legacy account conversion from ML account to BOA in cleansing to implementation

Ab Initio GDE 1.14.26, Co>Operating System 2.14.62, Data Stage 8.1, UNIX, Oracle 11g, Teradata 6.0, Autosys

Oracle/ETL Developer

Confidential

Responsibilities:

• Worked on Enterprise Data warehouse project FDW (Financial Data warehouse) project, responsible for design, develop BI reporting solutions using ODI, BAM and Oracle SOA Suite. Worked on creating ETL processes using ODI to populate data into staging and data ware-house tables and analysis cubes.
• Installed and configured ODI 10.1.3.5, Oracle 10g & Oracle BIEE in the Window environment
• Worked on migration projects, that includes database migration from SQL Server 2000 to Oracle 11g, and consolidation of databases. Worked on migrating DTS packages from SQL Server 2000 to ODI Interfaces in Oracle 11g.

ODI 10.1.3, Oracle 11g, SQL Server 2005, BAM, Oracle SOA Suite, PL/SQL, UNIX, Autosys

Sr. ETL Developer

Confidential

Responsibilities:

• Involved in ETL design, coding using Ab Initio ETL tool in order to meet requirements for ex-tract, transformation, cleansing, and loading of data from source to target data structures
• Prepared the estimation, design effort and SDLC documents for the enhancements and lead a team and responsible for on-time delivery of allocated DW project/project modules.
• Created complex transformations and multifile system in using aggregate, scan, normalize, parallelism, rollup, de-normalize, conditional dml, sequence generator, lookup, joiner and stored procedure transformations

Oracle 10g/11g, SQL*Loader, PL/SQL, SQL Server 2005, Teradata, DB2, Ab Initio Co>Op 2.14 GDE 1.14, UNIX, Autosys, XML, Informatica 7.1, Erwin, Trillium 7.6, Caliber, Dimensions, CONTROL-M