Big Data Engineer Resume
Newark, NY
SUMMARY:
- Over 11+ years of Professional IT experience in all aspects of Software Development Life Cycle (SDLC) including requirements analysis, design specification, code development, testing and deployment using various methodologies with Big data, Traditional ETL and Oracle System Developer.
- 2.5+years of extensive Analysis, design, Development and Implementation experience using Big Data years of experience ecosystem components usingSCB SCUDEE framework for data ingestion into Hadoop T1 layer.
- Currently working on NRT (near Real Time Replication) using CDC for Kafka on Kafka cluster using Apache Nifi flows and ingestion on hbase/hive for near real time analytics for SAIL Payments application.
- Also have 4+ years of experience in working on traditional DWH solutions using Netezza, Teradata, Datastage and IBMCDC replication tool.
- Well versed with SCB bank process to implement the development changes via CAB using remedy.
- Experienced in handling End to End DWH implementations for various clients, using big data Hadoop and traditional data warehouse implementation.
- Implemented the framework designing/Implementation of hive queries and SQOOP to import data from Oracle Database to Hadoop Hive tables.
- Good hands on experience in writing Pig Script &HiveQL
- Ability to adapt to any technology/domain/programming language within a short interval of time.
- Capable of working at great pressure and tight deadlines
- Excellent problem - solving skills and good interpersonal skills
- Experience in Implementation and Client Handling
- Profound function knowledge in Telecom domain with experience in working with Customer Care and Telecom Billing systems of the Amdocs Ensemble system
- Profound functional knowledge in Investment Banking, Finance Trade banking application.
- Good Exposure to global Trade Finance Domain which involves in Letters of Credit (LC), Bill Processing, Bankers Acceptances, Reimbursements and Collections.
- Have a very good understanding of Financial Market and Securities.
- Exceptionally well organized, strong work ethics and willingness to work smart and hard to achieve project/team objective.
TECHNICAL SKILLS:
Programming and Scripting: PL/SQL, UNIX Shell Scripting, Java, Scala, Pro *C.
Big Data Technologies: Hadoop, Sqoop, Hive, Pig, Spark, Apache nifi, Kafka
Replication Tools: IBM CDC for Oracle, IBM CDC for DB2, IBM CDC for Kafka
ETL Tools: IBM Datastage, MSBI SSIS
Scheduling Tools: Control M, Crontab
Version Control: SVN, PVCS, Merant Version Manager
Databases: Oracle, MySQL, Netezza, Teradata
OLAPTools: MSBI SSAS
Frontend: Power Builder, Adobe Cold Fusion 7, Adobe Dreamweaver 8
Middleware: Oracle tuxedo
Markup Languages: XML, HTML.
Java Development IDEs: Eclipse and IntelliJ Idea.
Operating Systems: Linux/AIX, Windows 2000
Tools: Toad, Telnet, DB debugger, Sql Developer, Teradata Client
PROFESSIONAL EXPERIENCE:
Confidential, Newark, NY
Big Data Engineer
Responsibilities:
- DCUT for Sourcing and Ingestion of SAIL STS data to Datalake T1 layer in hive for batch processes using SCB SCUDEE (ingestion framework).
- The replication layer being used is IBM CDC for Oracle and DB2 for mirroring delta. Wrote the code in CDC for handling DDL schema evolution from source.
- MetaApp layer for schema evolution done using shell script, java and hive retrofit driver using spark dataframes.
- Conduct POC's using IBM CDC for Kafka as producer using nifi dataflow tool on Kafka cluster (kafka, nifi and HBase) to validate the feasibility and benchmark the near Real time replication within the Enterprise Data warehouse.
- As part of SIP metadata management develop and enhance the metadata management process using shell scripts, teradata bteq and mload across the various application in edmp.
- Key member of CDC infra team perform CDC metadata management in Hadoop.
- End to End owner of implementation of CDC dashboard metadata to Hadoop in hive.
- Well versed with end to end SCB process for implementing Cr’s in production.
- Participate in the CAB process to implement the changes.
Environment: and Technologies used: Horton Works, HDFS, Hive, Sqoop, Kafka, Apache nifi, Spark, Scala, Spark SQL, Oracle 11g, IBM CDC for Oracle, IBM CDC for DB2, IBM CDC for Kafka, PL/SQL, SQL*PLUS, Windows NT, UNIX/Linux Shell Scripting, Toad, Telnet, Control M
Confidential
Team Lead
Responsibilities:
- Design and development of for ODS for Confidential .
- ODS replication layer consisted of IBM CDC for oracle and delta was generated using flat files.
- The population of delta, shadow and ODS DWH database was done using datastage jobs with control M as scheduling tool and shell script wrapper.
- Successfully migrated 24 million subscribers to ODS from Optus Legacy DWH to ODS production.
- Successfully migrated 1.5 million subscribers to ODS from Singtel Legacy DWH to ODS production.
- Designed and developed critical various end reports to Optus
- End to End owner of implementation of usage module for ingesting into hive.
- Managing team of 8 on various milestones of implementation for Confidential .
- Responsible for HLD, DD and LLD for the ODS.
- Implemented Control M automation wrapper using shell scripting to deploy 1000+ jobs in dev and production. Deploying new jobs now just took 5 minutes as opposed to 4 hours earlier.
- Handled complete IBM CDC replication implementation for Confidential .
- Developed the Unix Shell scripts for creating the reports from Hive data for Usage layer.
Environment: and Technologies used: IBM CDC for Oracle, IBM Datastage, Oracle SQL, UNIX, bash Shell Scripting, HortonWorks, HDFS, Hive, Sqoop, Toad, Telnet, Control M
Confidential
Team Lead
Responsibilities:
- Design and development of Usage processing for ODS in Peru using shell scripting and Oracle as ODS.
- End to End owner of implementation of usage module for TEF Peru.
- Design and development of 12 reports using OLAP-cube using SSAS.
- End to End owner of implementation of reports using OLAP-Cubes.
- Managing team of 8 on various milestones of implementation
- Responsible for HLD, DD and LLD for the ODS Usage, OLAP Cubes.
- Handled complete IBM CDC replication implementation for TEF Peru. Implemented the logic for populating the delta/staging layer in Oracle from CDC landing using shell scripting wrapper and control M for scheduling.
- Key member of Design review team for the entire TEF galaxy project.
Environment: and Technologies used: IBM CDC, IBM Datastage, Microsoft MSBI SSAS, SSIS, Oracle SQL, PL/SQL, UNIX, bash Shell Scripting, Toad, Telnet, Control M
Confidential
Team Lead
Responsibilities:
- End to End owner of implementation Confidential ODS for.
- Implemented the wrapper for parsing the CDC delta files and loading to Delta layer in Netezza.
- Used datastage framework to transform into shadow leyer.
- Wrote nzsql scripts to populated the shadow to ODS layer 6 subject areas.
- Successfully migrated 44 million customers from Claro legacy to ODS DWH over 3 migrations for Costa, Oriente and Occidente
- Managing team of 8 on various milestones of implementation
- Responsible for HLD, DD and LLD for the ODS Colombia
- Handled complete IBM CDC replication implementation
- Key role in DCUT for various critical reports (Ventas, Clientes, Equipos) to the end user.
- Key member of Design review team for the entire Claro galaxy project.
Environment: and Technologies used: IBM CDC, IBM Datastage, Netezza, Oracle SQL, PL/SQL, UNIX, bash Shell Scripting, Toad, Telnet, Control M, Datastage scheduler
Confidential, Dallas, TX
Senior Software Developer
Responsibilities:
- Developing new requirements in Customer Service Management for Telecom project
- Solving issues in the application
- Production Support
- Participation in critical bridge calls for the issues
- Presentation of new version kintanas
- Have a good technical knowledge of PowerBuilder
- Complete knowledge of SDLC.
- Complete knowledge of Telecom, ERP Domains.
Environment: and Technologies used: Oracle SQL, PL/SQL, SQL performance tuning, C, Power Builder 11.5, Pro*C, TUXEDO, UNIX, bash Shell Scripting, Toad, Telnet, Crontab
Confidential
Software Developer
Responsibilities:
- Following water fall model for SDLC processes, through HCL internal PM Smart application,
- Analysis, Coding, Implementation.
- CMMi documentation and testing of the enhancements,
- Preparation of FP estimate,
- Preparation of FS, TS, HLD, LLD, Impact Analysis and Handover,
- Interaction with Client,
- Implemented business logic in PL/SQL blocks and did query tuning.
- Post-Cutover Production Support for Release activities, this includes SIT, UAT support,
- Preparation of Unit Test Cases for Sanity testing and
- Causal Analysis
Environment: and Technologies used: Oracle SQL, PL/SQL, Unix IBM AIX, bash Shell Scripting, Perl, Adobe Cold Fusion 7, Toad, Telnet, Adobe Dreamweaver 8, Crontab
Confidential
Software Developer
Responsibilities:
- Following water fall model for SDLC processes, through HCL internal PM Smart application
- Analysis, Coding, Implementation
- Production Support / Enhancement
- CMMi documentation and testing of the project,
- Interaction with Client,
- Post-Cutover Production Support for Release activities, this includes SIT, UAT support,
- Preparation of Unit Test Cases for Sanity testing
Environment: and Technologies used: Oracle SQL, PL/SQL, Unix IBM AIX, Unix Scripting, PowerBuilder 10.5, Toad, Telnet, Control M
Confidential
Software Developer
Responsibilities:
- Analysis and Development,
- Production Support / Enhancement,
- Configuration Management,
- Managing Release activity,
- Managing Version control of PB objects,
- Interaction with Client,
- Post-Cutover Production Support for Release activities,
- Preparation of Unit Test Cases for Sanity testing and
- Causal Analysis
Environment: and Technologies used: Oracle SQL, PL/SQL, UNIX, Windows 2000, PowerBuilder 10.5, Toad, Merant version Control, Powergen
Confidential
SoftwareDeveloper
Responsibilities:
- Implemented Enhancements, bug fixes and Production Support
- Performed INFRA Support backup,
- Performed SIT Support,
- Performed UAT and Post-Cutover Production Support during Sanity testing
Environment: and Technologies used: Oracle SQL, PL/SQL, and Pro *C, Power Builder 10, HP UNIX, Toad, GDB debugger, Telnet