SUMMARY

• 8+ years of work experience in Development and Implementations of Data Warehousing solutions.
• 4+ years of development experience on cloud platforms(Azure) relevant to Azure Data Engineer. Solid experience on building ETL ingestion flows using Azure Data Factory.
• Experience in building Azure stream Analytics ingestion spec for data ingestion which helps users to get sub second results in Real Time.
• Experience in building ETL(Azure Data Bricks) data pipelines leveraging PySpark, Spark SQL.
• Experience in building the Orchestration on Azure Data Factory for scheduling purposes.
• Experience working with Azure Logic APP Integration tool.
• Experience working with Data warehouse like Oracle, SAP, HANA.
• Expertise on working with databases like Azure SQL DB, Azure SQL DW.
• Hands - on experience in Azure Analytics Services - Azure Data Lake Store (ADLS), Azure Data Lake Analytics (ADLA), Azure SQL DW, Azure Data Factory (ADF), Azure Data Bricks (ADB) etc.
• Orchestrated data integration pipelines in ADF using various Activities like Get Metadata, Lookup, For Each, Wait, Execute Pipeline, Set Variable, Filter, until, etc.
• Programming experience on working with Python, Scala.
• Happy to work with the team who are in middle of the road with some Big Data challenges for both on prem and cloud.
• Experience working in a cross-functional AGILE Scrum team.
• Good knowledge on Polybase external tables in SQL DW.
• Involved in production support activities

TECHNICAL SKILLS

ETL/Middleware Tools: Talend, SSIS, Azure Data Factory, Azure Data Bricks

Big Data: Cosmos, Hadoop, Azure Data Lake

Azure: Azure Data Factory, Azure Data Bricks, PolyBase, Azure DW, ADLS, Azure Synapse Analytics, BLOB, Azure SQL Server, Azure DW

RDBMS: Oracle, Netezza, Teradata, Redshift, MS SQL Serv, MySQL

Programming Skills: T-SQL, Java, Python, MS-SQL, SOQL

Tools: TOAD, SQL developer, Azure Data Studio, Soap UI, SSMS, GitHub, Share Point, Visual Studio, Teradata SQL Assistant.

PROFESSIONAL EXPERIENCE

Confidential

Azure Data Engineer

Responsibilities:

• Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements
• Design and developed Batch processing and real-time processing solutions using ADF, Databricks clusters and stream Analytics
• Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate sourcesystems by using different Azure Activities like Move &Transform, Copy, filter, for each, Databricks etc. Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.
• Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.
• Created, provisioned different Databricks clusters, notebooks, jobs and autoscaling.
• Performed data flow transformation using the data flow activity.
• Used Polybase to load tables in Azure synapse.
• Implemented Azure, self-hosted integration runtime in ADF.
• Improved performance by optimizing computing time to process the streaming data by optimizing thecluster run time.
• Perform ongoing monitoring, automation, and refinement of data engineering solutions.
• Scheduled, automated business processes and workflows using Azure Logic Apps.
• Designed and developed a new solution to process the NRT data by using Azure stream analytics,Azure Event Hub and Service Bus Queue.
• Created Linked services to connect the external resources to ADF.
• Worked with complex SQL views, Stored Procedures, Triggers, and packages in large databases fromvarious servers.
• Used Azure Devops & Jenkins pipelines to build and deploy different resources(Code and Infrastructure)in Azure.
• Ensure the developed solutions are formally documented and signed off by business.
• Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issueidentification, and management.
• Worked on the cost estimation, billing, and implementation of services on the cloud.
• Experience managing Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of howto integrate with other Azure Services.
• Migration of on premise data (Oracle/ Teradata) to Azure Data Lake Store(ADLS) using Azure DataFactory(ADF V1/V2).
• Work closely across teams (Support, Solution Architecture) and peers to establish and follow bestpractices while solving customer problems
• Created infrastructure for optimal extraction, transformation, and loading of data from a wide varietyof data sources.
• Designed and created optimal pipeline architecture on Azure platform.
• Created pipelines in Azure using ADF to get the data from different source systems and transform thedata by using many activities.
• Created Linked service to land the data from different sources to Azure Data Factory.
• Created different types of triggers to automate the pipeline in ADF.
• Created, provisioned different Databricks clusters needed for batch and continuous streaming dataprocessing and installed the required libraries for the clusters.

Environment: Azure Data Factory (ADF v2), Azure SQL Database, Azure functions Apps, Azure Data Lake, BLOB Storage, SQL server, Windows remote desktop, UNIX Shell Scripting, AZURE PowerShell, Data bricks, Python, ADLS Gen 2, Azure Cosmos DB, Azure Event Hub, Azure Machine Learning.

Confidential

Sr. Talend / ETL Developer

Responsibilities:

• Worked on SSAS in creating data sources, data source views, named queries, calculated columns, cubes, dimensions, roles and deploying of analysis services projects.
• SSAS Cube Analysis using MS-Excel and PowerPivot.
• Implemented SQL Server Analysis Services (SSAS) OLAP Cubes with Dimensional Data Modeling Star and Snow Flakes Schema.
• Developed standards for ETL framework for the ease of reusing similar logic across the board.
• Analyse requirements, create design and deliver documented solutions that adhere to prescribed Agile development methodology and tools.
• Responsible for creating fact, lookup, dimension, staging tables and other database objects like views, stored procedure, function, indexes and constraints.
• Monitoring the Data Quality, Generating weekly/monthly/yearly statistics reports on production processes - success / failure rates for causal analysis as maintenance part and Enhancing exiting production ETL Process
• Developed complex Talend ETL jobs to migrate the data from flat files to database.
• Implemented custom error handling in Talend jobs and also worked on different methods of logging.
• Followed the organization defined Naming conventions for naming the Flat file structure, Talend Jobs and daily batches for executing the Talend Jobs
• Responsible for develop the jobs using ESB components like tESBConsumer, tESBProviderFault, tESBProviderRequest, tESBProviderResponse, tRESTClient, tRESTRequest, tRESTResponse to get the service calls for customers DUNS numbers..
• Exposure of ETL methodology for supporting Data Extraction, Transformation and Loading process in a corporate-wide ETL solution using Talend Open Source for Data Integration 5.6. worked on real time Big Data Integration projects leveraging Talend Data integration components.
• Analyzed and performed data integration using Talend open integration suite.
• Wrote complex SQL queries to inject data from various sources and integrated it with Talend.
• Worked on Talend Administration Console (TAC) for scheduling jobs and adding users.
• Worked on Context variables and defined contexts for database connections, file paths for easily migrating to different environments in a project.
• Developed mappings to extract data from different sources like DB2, XML files are loaded into Data Mart.
• Created complex mappings by using different transformations like Filter, Router, lookups, Stored procedure, Joiner, Update Strategy, Expressions and Aggregator transformations to pipeline data to Data Mart.
• Involved in designing Logical/Physical Data Models, reverse engineering for the entire subject across the schema.
• Developed Talend ESB services and deployed them on ESB servers on different instances.
• Created WSDL data services using Talend ESB.
• Created Rest Services using tRESTRequest and tRESTResponse components.
• Used tESBConsumer component to call a method from invoked Web Service.
• Scheduling and Automation of ETL processes with scheduling tool in Autosys and TAC.
• Scheduled the workflows using Shell script.
• Used Talend most used components (tMap, tDie, tConvertType, tFlowMeter, tLogCatcher, tRowGenerator, tSetGlobalVar, tHashInput & tHashOutput and many more)

Environment: Talend 5.x,5.6, XML files, DB2, Oracle 11g, SQL server 2008, SQL, MS Excel, MS Access, UNIX Shell Scripts, TOAD, Autosys.

Confidential

Program Developer

Responsibilities:

• Monitoring Log shipping/Replication and troubleshooting of errors.
• Created Linked Servers between SQL server 2000 & Oracle 9i.
• Wrote complex stored procedures, User Defined Functions, Triggers using T-SQL.
• Created DTS packages for data transfer between the two environments.
• Security issues related to logins, database users, and application roles and linked servers.
• Performance tuning of SQL queries and stored procedures using SQL profiler and Index tuning advisor.
• Administered of all SQL server database objects, logins, users and permissions in each registered server.
• Resolved any deadlocks issues with Databases/Servers on a real-time basis.
• Wrote scripts for generating Daily Backup Report, verifying completion of all routine backups, log space utilization monitoring etc.
• Backup and Restoration of data/databases using third party tool (SQL LITE Speed).
• Involved in Design and Development of Disaster Recovery Plan.
• Created reports using Crystal Reports.

Environment: SQL Server 2000 Enterprise Edition, Windows 2000/NT, UNIX, Excel, SQL Profile, Replication, DTS, MS Access, T-SQL, Crystal Reports.