SUMMARY

• AWS professional with 7+ years of Industry experience. A full stack AWS Engineer with focus on building Data pipelines and Analytics Applications on AWS platform with a strong background on Platform engineering and DevOps
• Extensive experience in building data lake layers using scalable AWS Object storage and ETL services. Experience in building high performing DWs using AWS Redshift DW service
• Proficient in storage, compute and networking services with implementation experience in data engineering using key AWS services such as VPC, EC2, S3, ELB, AutoScalingGroup(ASG), EBS, RDS, IAM, EFS, CloudFormation, Redshift, DynamoDB, Glue, Lambda, Step Functions, Kinesis, Route 53, CloudWatch, CloudFront, CloudTrail, SQS,SNS, SES, AWS Systems Manager etc.
• Highly skilled in deployment, data security and troubleshooting of the applications in AWS.
• Experienced in working in variety of DevOps tools in mixed environments of Linux and windows servers in Amazon Web Services.
• Proficient in writing Cloud Formation Templates (CFT) in YAML and JSON format to build the AWS Services with the paradigm of Infrastructure as a Code.
• Experienced with event - driven and scheduled AWS Lambda functions to trigger events in variety of AWS resources using boto3 modules
• Developed data ingestion modules using AWS Step Functions, AWS Glue and Python modules
• Experienced in version control and source code management tools like GIT, SVN, and TFS.
• Good knowledge in relational and NoSQL databases like MySQL, DynamoDB, Redshift and AWS RDS.
• Experienced in Continuous Integration/Continuous Delivery tools such as GitBucket, bug-tracking tool JIRA and Jenkins to merge development with testing through pipelines.
• Experience in Shell and Python scripting language with AWS CLI and BOTO3 scripting experience
• Highly result driven, quality and craftmanship focused with good interpersonal skills, linear problem-solving skills, self-motivated, fast learner, good team player.

TECHNICAL SKILLS

Hardware / Server: Redhat Linux / Ubuntu, Windows

RDBMS: MySQL, Oracle, SQL Server, PostgresSQL, AWS Aurora

Cloud: AWS Services EC2, S3, ELB, Glacier, EBS, EFS, ENI, CloudFormation, RDS, VPC, Route53, CloudWatch, CloudTrail, IAM, SNS, SQS, AWS Athena, RedShift, EMR, DynamoDB, Lambda, Step Functions, EKS, ECS, Glue, Kinesis, Snowflake D

CI/CD: Terraform, Jenkins, Git

BIG Data: Hadoop, HIVE, Spark, Cloudera

LANGUAGES/SCRIPTING: Python, bash Scripting, JSON

PROFESSIONAL EXPERIENCE

AWS Engineer

Confidential

Responsibilities:

• Responsible for provisioning key AWS Cloud services and configure them for scalability, flexibility and cost optimization
• Create VPCs, subnets including private and public, NAT gateways in a multi- region, multi-zone infrastructure landscape to manage its worldwide operation
• Manage Amazon Web Services (AWS) infrastructure with orchestration tools such as CFT, Terraform and Jenkins Pipeline
• Create Terraform scripts to automate deployment of EC2 Instance, S3, EFS, EBS, IAM Roles, Snapshots and Jenkins Server
• Build Cloud data stores in S3 storage with logical layers built for Raw, Curated and transformed data management
• Create data ingestion modules using AWS Glue for loading data in various layers in S3 and reporting using Athena and Quicksight
• Create manage bucket policies and lifecycle for S3 storage as per organization’s and compliance guidelines
• Create parameters and SSM documents using AWS Systems Manager
• Established CICD tools such as Jenkins and Git Bucket for code repository, build and deployment of the python code base
• Build Glue Jobs for technical data cleansing such as deduplication, NULL value imputation and other redundant column removal. Also build Glue jobs to build standard data transformations (date/string and Math operations) and Business transformations required by business users
• Create Athena data sources on S3 buckets for adhoc querying and business dashboarding using Quicksight and Tableau reporting tools
• Copy Fact/Dimension and aggregate output from S3 to Redshift for Historical data analysis using Tableau and Quicksight
• Use Lambda functions and Step Functions to trigger Glue Jobs and orchestrate the data pipeline
• Use PyCharm IDE for Python/PySpark development and Git for version control and repository management

Environment: AWS - EC2, VPC, S3, EBS, ELB, CloudWatch, CloudFormation, ASG, Lambda, AWS CLI, GIT, Glue, Aetna and QuickSight, Python and PySpark, Shell scripting, Jenkins, Terraform

Hadoop Engineer

Confidential

Responsibilities:

• Participated on project scoping exercises and creating requirement document and source to target mapping
• Participated in designing the Ingestion framework for History and Incremental load into Hadoop file system and hive
• Built data ingestion modules with Sqoop and shell scripts
• Performed complex business transformations using both spark sql and Spark APIs and saved final dataset in partitioned Hive tables
• Developed ETL data pipeline using Spark API to fetch data from Legacy system (SQL Server) and third-party APIs (External data)
• Migrated SQL Server packages into Spark transformations using Spark RDDs and Data Frames
• Worked on Data lake Staging, conformed and Reporting layers and building the data pipeline from ingest to consumption
• Created Fact and Dimension tables and summary tables for Reporting consumption
• Developed and designed POCs using PySpark and Hive and deployed on the YARN cluster, compared the performance of Spark with SQL Server modules
• Improved runtime performance of Spark applications with YARN queue management and Memory tuning
• Performed Unit testing and end to end application testing with data validation
• Used PyCharm IDE, spark CLI for development and managed code repository in Git
• Performed Hive Query performance tuning and helped end users with Reports
• Used Impala query engine for Reports serving in Tableau

Environment: Languages/Technologies: Hadoop, Data Lake, Python, PySpark, Spark SQL, Hive, Impala, PyCharm, GIT Repository, Cloudera CDH, Maven, UNIX Shell Scripting, SQL server, Sqoop, Autosys, AWS, S3, EMR