Technology Lead Big Data Engineer Resume

SUMMARY:

Big data professional with 11+ years of IT experience and 3+ years of extensive experience as aBig data Engineer and as Hadoop Developer.
Extensive knowledge in Hadoop eco - system technologies like Apache Hive, Sqoop, Apache Spark core, Spark SQL
Involved in developing Big Data applications by using different frameworks like Hadoop, Hive, Sqoop, Spark
Expertise at designing tables in Hive, SQL server using SQOOP and processing data like importing and exporting of databases to the HDFS.
Having experience in all stages of the project including requirements gathering, designing & documenting architecture, development, performance optimization, cleaning, and reporting.
Strong experience on AWS-EMR, Spark Installation, HDFS & Map-reduce Architecture. Along with that having a good knowledge on Spark, Scala and Hadoop distributions like Apache Hadoop, Cloudera.
Handle the JSON, XML, Log data using Hive (SERDE) and filter the data based on query factor.
Proficient and experienced in developing and changing COBOL, DB2 Modules. Very good in developing and changing JCLs and PROCS. Worked on developing components using COBOL, VSAM and Flat files.
Expertise in DB2, VSAM, COBOL, JCL and IMS Technologies.
Used SQL for, retrieving Data from a Single Table, from Multiple Tables, Creating and Updating Tables and Views, Programming with the SQL Procedure.
Involved in writing complex SQL queries based on the given requirements such as complex DB2 Joins, Sub Queries etc.
Have excellent experience working with Scrum, Agile and Waterfall Software Development Life Cycle (SDLC) methodologies.

TECHNICAL SKILLS:

Domain: Banking

Technical: Sqoop, HBase, Hive, Apache Spark, Map Reduce, Yarn, Cassandra, Oozie.

Languages: Scala, Spark SQL, COBOL, Core Java

Relational Databases: MySQL, DB2 on ZOS

PROFESSIONAL EXPERIENCE:

Technology Lead Big Data Engineer

Confidential

Responsibilities:

Understanding the requirements for the project.
Based on Mapping document, import data from Oracle, SQL Server, DB2 databases and store data in Hive based on business logic.
Worked with different file formats like Text files, Sequence Files, Avro, Parquet.
Schedule spark Jobs using Oozie and shell script.
Query performance optimization.
Performance Analysis using spark web UI.
Write alternative code to optimize Spark performance.
Validating and testing the code.

Confidential

Responsibilities:

For injection data from Oracle, SQL server to HDFS which can be queried using hive and spark SQL tables.
Worked on Sqoop jobs for ingesting data from SQL server to HDFS
Created hive external tables for querying the data
Use Spark Dataframe APIs to inject Oracle data to HDFS.
Process the datasets and apply different transformation rules on the top of different datasets.
Apply Transformation rules on the top of DataFrames.
Validate the source and final output data.
Debug & test the process is reaching Client's expectations or not.
Query execution is trigger. Improve the process timing.

Confidential

Responsibilities:

Co-ordination with SME and all the implementation teams based in London (Client Location). Taking decisions independently at critical situations based on the requirement of Business/Client.
Leading a team of 8 members which includes Technical Analysts, Senior Software Engineers and Software Engineers.
Preparing application design document.
Develop application for migrating customers of Royal Bank of Scotland to the new branches.
Review code of other team members.
Co-ordinate with SMEs and business.

Confidential

Responsibilities:

Analyzing and resolving tickets within stipulated time
Resolving the tickets based on Service Level Agreement.
Involved in Analysis, Coding, testing and UAT.
Involved in interacting with clients and business users.

We provide IT Staff Augmentation Services!

We'd love your feedback!

Resume Categories

Client Services

Job Seekers

Visa Sponsorship