Data Engineer (agile Methodology) Resume
Raleigh, NC
SUMMARY
- 8+ years of IT Experience in Data Warehousing, Database Design and ETL Processes in the complex business environments of various business domains like finance, manufacturing and health care industries.
- Strong experience in Extraction, Transformation, Loading (ETL) data from various sources into Data Warehouses and Data Marts using Informatica Power Center.
- Worked in the Requirement Analysis,DataAnalysis, Application Design,DataModeling, Application Development, payment data and Transactional data at various Business Application Systems of Financial (JPMC), Confidential . Had an experience working with pricing and contracting at JPMC client.
- Good experience in Data Modeling with expertise in creating Star & Snow - Flake Schemas, FACT and Dimensions Tables, Physical and Logical Data Modeling using Erwin.Designed and developed efficient Error handling methods and implemented throughout the mappings in various projects.
- Extensive experience in integration of various heterogeneous data sources definitions like SQL Server, Oracle, Teradata, Flat Files, Excel and XML files loaded data into Data warehouse and Data marts using Power center.
- Strong skills in SQL and PL/SQL, backend programming, creating database objects like Stored Procedures, Functions, Triggers and Packages.
- Worked extensively in creating complex Mappings using various transformations like Unconnected and Connected lookups, Source Qualifier, Joiner, Rank, Sorter, Router, Filter, Expression, Aggregator, Joiner and Update Strategy, Normalizer.
- Very good knowledge in data warehouse designs using Star Schema and Snowflake Schema Architectures, Fact and Dimension Tables, UML concepts, OLTP and OLAP applications.
- Developed Unix Shell Scripts and PL/SQL procedures. Extensively used UNIX shell scripts to create the parameter files dynamically and scheduling jobs using Autosys. Scheduled sessions and batches on the Informatica Server using Informatica Server.
- Experienced in various data sources like DB2, SQL Server, Oracle, Fixed Width and Delimited Flat Files. Automation of ETL process, error handling and reporting purposes.
- Knowledge with Informatica DVO, IDE/IDQ tools for Data Analysis / Data Profiling and Data Governance.
- Used Teradata utilities like for Teradata ETL processing huge volumes of data throughput Fast Load, Fast Export, Multiload, Tpump and also Involved in implementation and batch monitoring.
- Used Basic Teradata Query utility for reporting Purpose. Built the Operational data store by using BTEQ, Teradata, Oracle, SQL, and UNIX.
- Expertise in different types of loading like Normal and Bulk loading challenges. Involved in Initial Loads, Incremental Loads, Daily loads and Monthly loads.
- Performed tuning and optimization of complex SQL queries using Teradata Explain and Run stats.
- Experience in Ingesting data to different Tenants in Data Lake and creating snapshots tables for Consumption. Experience in working on HIVE tables with good understanding on Filters.
- Experienced in development of a talendframework for common and consistent use of the intended design for batch data ingestion, data processing and delivery of massive datasets.
- Very good understanding of Partitions, bucketing concepts in Hive. Designed and Managed External tables in Hive to optimize performance.
TECHNICAL SKILLS
Hadoop Ecosystem: Hive, Spark, MapReduce.
ETL Tools: Informatica Power Center 10.x,9.x,8.x/7.x, OLAP, OLTP, SQL*Loader, Informatica Power Connect for DB2, IDQ, Informatica Power Exchange 8.x,Talend7.1/6.5/6.1
Dimensional Data Modelling: Data Modelling, Star Schema Modelling, Snow-Flake Modelling, FACT and Dimensions tables, physical and logical data modelling, Erwin.
Operating Systems: Linux, Unix(Solaris, AIX), Windows 95/98/NT/2000/XP,Windows7
Databases: Oracle 12g,10g/9.x/8.x, SQL Server, DB2, Teradata
Tools: Exadata, Toad, Visio, Share Point, Putty, Telnet, WINSCP, Oracle, Teradata SQL Assistance.
Scheduling: Control M, Autosys, Tivoli
Agile Tools: Jira, Confluence.
Programming Languages: Python,PLSQL, UNIX/Linux Shell Scripting.
Version Control Tools: PVCS, GIT Hub
PROFESSIONAL EXPERIENCE
Confidential, Raleigh, NC
Data Engineer (Agile Methodology)
Responsibilities:
- Worked on the projects like FRAUD Reporting and COMMERCIAL Reporting.
- Worked On call support for production environment.
- Created new mapping designs using various tools in Informatica Designer like Source Analyzer, Warehouse Designer, Mapplet Designer and Mapping Designer.
- Develop the mappings using needed Transformations in Informatica tool according to technical specifications
- Created complex mappings that involved implementation of Business Logic to load data into staging area.
- Involved in the performance tuning of the Informatica mappings and stored procedures and the sequel queries inside the source qualifier.
- Created, launched & scheduled sessions.
- Involved in the Performance Tuning of Database and Informatica. Improved performance by identifying and rectifying the performance bottle necks.
- Used Server Manager to schedule sessions and batches.
- Involved in creating Business Objects Universe and appropriate reports
- Wrote PL/SQL Packages and Stored procedures to implement business rules and validations.
- Used Informatica reusability at various levels of development.
- Developed mappings/sessions using Informatica Power Center 10.4 for data loading.
- Developed Workflows using task developer, Worklet designer and workflow designer in Workflow manager and monitored the results using workflow monitor.
- Building Reports according to user Requirement.
- Extracted data from Oracle and SQL Server then used Teradata for data warehousing.
- Write Shell script running workflows in Unix environment.
- Optimizing performance tuning at source, target,mapping and session level
- Participated inweeklystatus meetings and conducting internal andexternal reviews as well as formal walk through among various teams and documenting the proceedings.
- Worked on Unix commands and shell scripting.
- Partner with the Lead Developer to identify, analyze and document new business requirements and translate them to data integration and workflow requirement specifications.
- Worked on Jil Scripts to schedule the jobs in Autosys.
- Worked on EDW architecture and Data flow using Visio Professional.
- Worked on git commands to check in and checkout the code. Updated the Jira tickets and created DCR along with the steps to execute in uat and prod deployment.
Environment: Informatica Power Center10.4/ 10.2, Oracle 12c, sql oracle developer, PL/SQL, UNIX, Autosys Scheduling job, Shell Scripting, Hive, JIRA, Confluence, Waterfall/Agile Methodology and flat files. Python 3.0, Bit Bucket, UNIX, Shell Scripting, Hive, JIRA, Visio, Agile Methodology, confluence.
Confidential, Irving, TX
ETL Developer (Waterfall/ Agile Methodology)
Responsibilities:
- Identify, prioritize, and execute on DI and Workflow enhancements to continuously improve the end user experience.
- Partner with the Lead Developer to identify, analyze and document new business requirements and translate them to data integration and workflow requirement specifications.
- Complete data integration and workflow changes as assigned and assist with developing, documenting and performing system and user acceptance tests.
- Worked on AGENTS, POLICIES and PROFIT SHARING data.
- Involved on the projects PROFIT SHARING, ADM REMEDIATION, STATE PRIVACY LAW.
- Worked on delivering Big Data Management and Integration Solutions.
- Worked on hive to analyze the data by writing HQL queries.
- Strong technical skills with concentrated abilities in relational database
- In-depth understanding of the movement of sets of data, master systems anddownstream systems and the way data sets are related to one another
- Extensively worked on confirmed Dimensions for the purpose of incremental loading of the target database.
- Development of scripts for loading the data into the base tables in EDW using Fast Load, Multiplied and BTEQ utilities of Teradata.
- Worked on ODS, Data mart and also involved in designing the ETL Uses Cases.
- Performed Unit, Systems and Regression Testing of the mappings. Involved in writing the Test Cases and also assisted the users in performing UAT.
- Developed/Enhanced mappings using transformations like Source Qualifier, Aggregator, Expression, Joiner, Filter,Lookups, Router, Union and Update Strategy.
- Familiar with SCD concepts.
- Experience in using basic UNIX commands and writing scripts.
- Hands on in writing SQL for data verifications.
- Proficient in designing and executing test cases and test scripts.
- Handled multiple post and parallel production runs.
- Hands on on Data Warehouse/Data mart, OLTP and OLAP implementations System testing, Implementation and production support.
- Handled documentation on release deployment checklist for the production release.
Environment: Informatica Power Center10.1/10.2, Oracle 11g, sql oracle developer, PL/SQL, UNIX, Tivoli Scheduling job, Shell Scripting, Hive, Talend7.1,JIRA, Hive,Confluence, Water Fall/Agile Methodology and flat files.
Confidential, Charlotte, NC
ETL/BIG DATA Developer (Waterfall/ Agile Methodology)
Responsibilities:
- Worked in all the phases of Data warehouse life cycle involving Requirement Analysis, Design, Coding, Testing, and Deployment.Worked with business analysts to identify and understand requirements and translated them into ETL code in Requirement Analysis phase. Designed various Mappings for extracting data from various sources involving relational tables & flat files.
- Worked closely with Business Users regarding the volume issues of GE Lift and Shift Project. Worked on identifying the flow of jobs and analysis of BO reports in GE lift and Shift project.
- Used ETL process to load data from multiple sources to staging area (Oracle) using Informatica Center 9.6.1.
- Worked extensively on Informaticatransformations like Source Qualifier, Expression, Filter, Router, Union, Aggregator, Lookup, Update Strategy, etc.Involved in creation of Mapplets, Work lets, Reusable transformations.
- Worked on the databases Oracle, Teradata, Mysql.
- Worked on Error Handling, Scheduling, Identifying ETL Job Dependency and Recovery mechanisms.
- Redesigned and migrated existing mappings in informatica power center to meet new requirements.
- Worked in optimizing the hive queries to improve the performance of hive queries. Performed extensive data analysis using Hive.
- Performed data analysis in Hive by creating tables, loading it with data and writing hive queries which will run internally in a Map Reduce way.
- Worked on a Hadoop framework for Hive processing, HQL processing and outbound processing and be able to frequently bring in data from the source and make it available for consumption.
- Involved in Data Ingestion in ingesting the data from different sources like Oracle, Teradata and SQL server.
- Worked on mapping documentation by identifying the sources, targets, hashed files, Transformation stage and logics in between by using Data Stage XML Reports.
- Involved in migrating the Mappings, Sessions, Workflows from Test environment to Production environment.
- Performed fine tuning of existing mappings in order to increase the performance of the mappings.
- Involved in implementation ofInformatica and Unix shell script code to SIT, UAT and Production environment.
- Involved in documenting the database changes, informatica changes and JIL scripts. Worked on post-install validation scripts and roll-back of database changes.
- Intensively worked on the Debugger from within the Mapping Designer to troubleshoot predefined mapping.
- Worked on creating the Problem tickets and Work Requests to work on the production issues. Worked in 24/7 production support, ETL executions and resolving root causes.
- Developed and Scheduled jobs for daily, Monthly, Weekly loads by creating JIL Scripts in CA Workload automations by creating the scripts that invoked Informatica workflows & sessions associated with the mappings.
- Worked on SFTP project in creating the new JIL scripts to transfer the files every day by creating the shell scripts to call the files from the folder.
- Involved in migration of SFTP to DTS transmission process by creating the new connections and folders.
- Created SQL scripts to drop & recreate the indexes on source and target tables by calling it with Autosys scheduling jobs.
- Involved in Developing the solutions for business challenges by leveraging UNIX scripting, Oracle SQL, and Autosys tools and environment.
- Member of On-call team for providing supportfor daily, weekly and monthly batch loads.
Environment: Informatica Power Center9.6/10.1, Oracle 11g, Teradata, sql oracle developer, PL/SQL, SQL Server, UNIX, Autosys Scheduling job, Hive, Talend7.1, Shell Scripting, Water Fall/Agile Methodology and flat files.
Confidential, Salem, NH
ETL/SQLDeveloper(Agile-Lead)
Responsibilities:
- Involved in gathering and analyzing the requirements with IT Functional and Technical product managers and Business Users(BAU’s).Prepared source to target data mapping and business rules for the ETL processes.
- Worked with data modeler and business users for designing of tables.
- Implemented restart abilityframework to recover failed sessions. Worked on data integrity and data validation checks for ETL load. Worked on exceptions handling and exception reprocessing for ETL load.
- Created Time Estimate Proposal document with estimation of hours required for completion of each ETL task.
- Designed, Developed, Documented and Build Teradata UNIX scripts for the Fast Load, Fast Export, BTEQ and MLoad Utilities to load data into Teradata tables.
- Conversion of business requirements into technical documents - Business Requirement Document, explained business requirements in terms of technology to the developers. Data integrity maintained through session control.
- Worked on data masking to hide the data of credit transaction in production
- Performed datacleansing, massaging, masking, IDQ and data profiling of data.
- Used structured and Un-Structured Data like PDF files, spreadsheets, Word documents and print streams option to get normalized data using B2B Exchangeof informatica.
- Used ODI to effectively integrate heterogeneous data sources and converting raw data into useful information.
- Worked closely with Architect and Data Modeler to enhance the existing Model with the new attributes as per the Business needs.
- Hands on experience on working withODIKnowledge Modules like LKM, IKM and JKM, CKM.
- Responsible for creating batches & scripts for implementing logical design to T-SQL.
- Extended the baseODIKnowledge Modules to create customized versions that are re-used across other data integration projects.
- Implemented Change Data Capture using Informatica Power Exchange 9.1.
- Deployed data stage jobs to various environments like Test, UAT.
- Configured and used B2B data exchange for end to end datavisibility through event monitoring and to provide a universal data transformation supporting numerous formats, documents, and filters.
- Worked on unit testing, documented positive and regression testing on the mappings. Worked on workflow design documents and process control documents on the workflows.
- Created mappings to load dimensions and fact tables from multiple source systems like Oracle, SQL server, flat files, etc. for the data warehouse.Worked in a critical project EIS to built data warehouse and also involved in Data Mart.
- Extensive knowledge in Data Analysis, Data Requirement gathering and Data Mapping for ETL processes and the scheduler tool Control-M.
- Well versed in OLTP DataModeling,Datawarehousing concepts.
- Involved inDatamodeling to determineDatadefinitions and establish referential integrity of the system.
- Experience in Agile methodology as a lead and worked with offshore team.
- Designed and developed complex mappings involving SQL Transformation, Lookup, Expression, Update, Sequence generator, Aggregator, Router, Stored Procedure, etc.,
- Created reusable transformations, sessions, mapplets to simplify ETL processes.
- Used PLSQL Stored Procedures for truncating the tables, drop partitions, gather stats through informatica mappings.
- Extensively used Informatica Debugger for trouble shooting purpose using data breakpoints.
- Coding using Teradata Analytical functions, BTEQ SQL of TERADATA, write UNIX scripts to validate.
- Worked with DBA’s on creation of tables, indexes, stored procedures, partitions for the data model identified. Worked with PVCS (Polytron version control systems), GIT for moving ETL and SQL code.
- Used Batch Control to schedule the Control M jobs by calling ETL informatica workflows Worked with file watchers, cyclic, hourly, daily, monthly, annual and on-demand jobs in CTM
Environment: Informatica Power Center 9.1/ 9.6/10.1 , Informatica DVO 9.1, IDQ 9.6, Oracle 11g, SQL Server 2012, ODI 12C, Informatica B2B Exchange, T-SQL, Tableau, Putty, JIRA, Teradata, Toad10.6, SQL Loader, MS Visio,Control-M, MDM, oracle EBS, WinScp tool, Unix/LINUX Shell scripting, Flat files, ERWIN, GIT, PVCS, Share point, Confluence.
Confidential, Charlotte, North Carolina
Informatica/Database Developer
Responsibilities:
- Responsible for gathering the requirements both functional and technical and documentation of the same.
- Worked with the business analysts in requirement analysis to implement the ETL process.
- Involved in Low-level Design for the scripts of the database sequences, constraints, triggers and stored procedures.
- Extensively used Informatica to load data from various Data Sources like Flat files, Oracle, SQL Server, into the Enterprise Data Warehousing.Created Data Lineage through Informatica Metadata Manager
- Extensively worked on confirmed Dimensions for the purpose of incremental loading of the target database.
- Developed logical and physical data models that capture current state/future state data elements and data flows. Designed and developed Complex mappings like Slowly Changing Dimensions Type 1, Type 2. Created mappings using informatica as per 3rd normal form design model.
- Extensively worked in performance tuning of programs, ETL procedures and processes.
- Developed and invoked PL/SQL stored procedures and functions for data processes used in the Informatica mappings.
- Worked on ODS, Data mart and also involved in designing the ETL Uses Cases.
- Performed Unit, Systems and Regression Testing of the mappings. Involved in writing the Test Cases and also assisted the users in performing UAT.
- Extensively used Informatica to load data from various Data Sources like Flat files, Oracle, SQL Server, into the Enterprise Data Warehouse.
- Extensively worked on confirmed Dimensions for the purpose of incremental loading of the target database. Performed Unit, Systems and Regression Testing of the mappings. Involved in writing the Test Cases and also assisted the users in performing UAT.
- Guided and coached multiple team members to utilize agilemethodology and provide constructive performance feedback.
- Worked on data modellingand produced datamapping and data definition documentation.
- Build Data stage jobs to Transforming, integrating, and loading data into data warehouse database using various stages which has available in data stage like merge, lookup, change capture and transformer stage.
- Executed and tested required queries and reports before publishing dashboards.
- Extensively worked with Slowly Changing Dimensions Type1, Type2, for data Loads.
Environment: Informatica Power Center 9.5, ORACLE11g, PL/SQL, AGILE/SCRUM, TOAD, Reports 9i/10g, SQL Server 2012, Flat Files, Autosys, GIT, UNIX Shell Scripting, LINUX.
Confidential
Software/SQL Programmer
Responsibilities:
- Responsible for documentation, version control of schema and version release.
- Analyzed specifications and identified source data needs to be moved to data warehouse, participated in the Design Team and user requirement gathering meetings.
- Participated in the analysis of development environment of Extraction process and development architecture of ETL process.
- Coordinated with customer in finding the sources and targets for data conversion.
- Analyzed the source data with business users, developed critical mappings using Informatica Power Center to load the data from DB2 to Oracle.
- Scheduled sessions and batches on the Informatica Server using Informatica Server. Developed Unix Shell Scripts and PL/SQL procedures.
- Extensively used LINUX shell scripts to create the parameter files dynamically and scheduling jobs using Autosys.
- Created Reusable transformations and Mapplets for use in Multiple Mappings.
- Worked on Power Center Designer client tools like Source Analyzer, Warehouse Designer, Mapping Designer and Mapplet Designer.
- Creation of Transformations like Lookup, Joiner, Rank and Source Qualifier Transformations in the Informatica Designer.
- Coordinated with source system owners, day-to-day ETL progress monitoring, Data warehouse target schema Design (Star Schema) and maintenance.
- Datamodeling experience using DimensionalDatamodeling, Star Schema modeling, Physical & logicalDatamodeling in ERWIN. Designed and developed reusable Transformations for ETL using Informatica Power Center 8.6.1.
- Worked extensively on different types of transformations like normalize, expression, union, filter, aggregator, update strategy, lookup, stored procedure, sequence generator and joiner.
- Extensively worked in performance tuning of programs, ETL procedures and processes.
- Developed and invoked PL/SQL stored procedures and functions for data processes used in the Informatica mappings.
- Tactical Execution of messaging and product content to meet aggressive timelines for deliverables.Worked on extracting data from different sources likes IBM MQ, Oracle, MS Access, Flat files.
- Create and Maintain Teradata Tables, Views, Macros, Triggers and Stored Procedures.
- Coding usingTeradataAnalytical functions, write UNIX scripts to validate, format and execute the SQLs on UNIX environment.
- Extensively dealt with the performance issues and made necessary coding changes for improving the system Performance.
- Responsible for development of robust and reliable Database solutions requiring low maintenance, thorough testing of solutions. Developed SQL scripts to Insert/Update and Delete data in MS SQL and Oracle database tables.
Environment: Informatica Power Center 8.6, Teradata, informatica power exchange 8.1, Oracle 10g, PL/SQL, Flat Files, AGILE METHODOLOGY, SQL Server 2007, ERWIN, SQL Workbench, UNIX Shell Scripting, Mainframes, MS PowerPoint, SQL, ERWIN, PL/SQL.