SUMMARY:

• Around 8 years of experience spread across Python, Big Data, Apache Spark, Scala, Java, SQL technologies.
• Have a hands - on experience on fetching the live stream data from DB2 to HDFS using SparkStreaming and Apache Kafka.
• Experience in real time data from various data sources through Kafka data pipelines and applied various transformations to normalize the data stored in HDFS Data Lake.
• Expertise with different tools in Hadoop Environment including Pig, Hive, HDFS, MapReduce, Sqoop, Spark, Kafka, Yarn, Oozie, and Zookeeper.
• Extensive work in ETL process consisting of data transformation, data sourcing, mapping, conversion and loading using Informatica.
• Expertise in developing data driven applications using Python 2.7, Python 3.0 on Pycharm and Anaconda Spyder IDE's.
• Hands on experience in configuring and working with Flume to load the data from multiple sources
• Proficient in designing and querying the NoSQL databases like Hbase, Cassandra, MongoDB, Impala.
• Experience with Web Development, Amazon Web Services, Python and the Django framework.
• Experienced in using MVC architecture using RESTful, Soap Web services and SoapUI and high-level Python Web frameworks like Django and Flask. Experience object-oriented programming (OOP) concepts using Python, Django, and Linux.
• Experienced in WAMP (Windows, Apache, MYSQL, and PHP) and LAMP (Linux, Apache, MySQL, and PHP) Architecture.
• Experienced in MVW frameworks like Django, Angular.js,Java Script, backbone.js, JQuery and Node.js.
• Have experience on Kubernetes and Docker for runtime environment of system to build, test & deploy
• Good experience in working with Amazon Web Services like EC2, Virtual private clouds (VPCs), Storage models (EBS, S3, instance storage), Elastic Load Balancers (ELBs)
• Familiar with JSON based REST Web services and Amazon Web services.
• Deeply involved in writing complex Spark-Scala scripts, Spark context, Cassandra SQL context, used multiple API's, methods which support data frames, RDD's, Cassandra table joins and finally write/save the data frames/RDD's to Cassandra database.
• Knowledge on integrating different eco-systems like Kafka - Spark - HDFS.
• Good Knowledge in Apache Spark and SparkSQL.
• Experience in running Spark streaming applications in cluster mode and Spark log debugging.
• Skilled on migrating the data from different databases to Hadoop HDFS and Hive using SQOOP.
• Good Experience in the core concepts of MapReduce Framework and Hadoop ecosystem.
• Experience in optimizing volumes, EC2 instances and created multiple VPC instances and created alarms and notifications for EC2 instances using Cloud Watch
• Extensive knowledge of creating manages tables and external tables in Hive Eco system.
• Worked extensively in design and development of business process using SQOOP, PIG, HIVE, HBASE
• Expertise in Working on Data Encryption (Client-Side and Server-Side) and securing data at rest and in transit for data in S3, EBS, RDS, EMR, Red Shift using Key Management Service (KMS).
• Good Knowledge in Amazon AWSconcepts like EMR and EC2 web services which provides fast and efficient processing of Big Data.
• Having a Complete Understanding on Lambda architectures
• Experienced in developing Web Services with Python programming language and Good working experience in processing large datasets with Spark using Scala and Pyspark.
• Knowledge on Spark framework for batch and real-time data processing.
• Knowledge on Scala Programming Language. Good experience with Talend open studio for designing ETL Jobs for Processing of data.
• Hands on experience in MVC architecture and Java EE frameworks like Struts2, Spring MVC, and Hibernate.
• Experienced in WAMP (Windows, Apache, MYSQL, and Python) and LAMP (Linux, Apache, MySQL, and Python) Architecture and Wrote Automation test cases using Selenium Web Driver, JUnit, Maven, and spring.
• Good knowledge in Software Development Life Cycle (SDLC) and Software Testing Life Cycle (STLC).
• Worked in agile and waterfall methodologies with high quality deliverables delivered on-time.
• Experience with Test Driven Development (TDD), Agile, Scrum and Waterfall methodologies. Used ticketing systems like JIRA, Bugzilla and other proprietary tools.
• Excellent communication and inter-personal skills detail oriented, analytical, time bound, responsible team player and ability to coordinate in a team environment and possesses high degree of self-motivation and a quick learner.

WORK EXPERIENCE:

Senior Python Developer

Confidential, Chicago, IL

Responsibilities:

• Worked extensively on AWS Components such as Airflow, Elastic Map Reduce (EMR), Athena, SnowFlake.
• Developed SQOOP scripts to migrate data from Oracle to Big data Environment.
• Extensively worked with Avro and Parquet files and converted the data from either format Parsed Semi Structured JSON data and converted to Parquet using Data Frames in PySpark.
• Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3buckets, performed folder management in each bucket, managed logs and objects within each bucket.
• Created Hive DDL on Parquet and Avro data files residing in both HDFS and S3 bucket
• Created Airflow Scheduling scripts in Python to automate the process of sqooping wide range of data sets.
• Deployed Scalable Hadoop cluster on AWS using S3 as underlying file system for Hadoop.
• Worked with Amazon Web Services (AWS) using EC2 for hosting and Elastic map reduce (EMR) for data processing with S3 as storage mechanism.
• Worked with various HDFS file formats like Avro, Sequence File and various compression formats like Snappy, bzip2.
• Maintained and developed Docker images for a tech stack including Cassandra, Kafka, Apache, and several in house written Java services running in Google Cloud Platform(GCP) on Kubernetes.
• Created data partitions on large data sets in S3 and DDL on partitioned data.
• Converted all Hadoop jobs to run in EMR by configuring the cluster according to the data size.
• Extensively used Stash Git-Bucket for Code Control
• Monitor and Troubleshoot Hadoop jobs using Yarn Resource Manager and EMR job logs using Genie and kibana.
• Consumed the data from Kafka using Apache spark.
• Worked with SQOOP jobs to import the data from RDBMS and used various optimization techniques to optimize Hive, Pig and SQOOP.
• Developed analytical component using Scala and KAFKA.
• Designed Forms, Modules, Views and Templates using Django and Python.
• Involved in application development for Cloud platforms using technologies like Java/J2EE, Spring Boot, Spring Cloud, Micro Services, REST.
• Implemented RESTful Web-services for sending and receiving the data between multiple systems.
• Rewrite existing Python/Flask module to deliver certain format of data.
• Developed an application in Linux environment and dealt with all its commands.
• Administrate Continuous Integration services (Jenkins, Nexus Artifactory and Repository).
• Designed and Developed DB2 SQL Procedures and UNIX Shell Scripts for Data Import/Export and Conversions.
• Hands on experience Installation, configuration, maintenance, monitoring, performance and tuning, and troubleshooting Hadoop clusters in different environments such as Development Cluster, Test Cluster and Production.
• Performed dynamic UI designing with HTML5, CSS3, less, Bootstrap JS, JavaScript, JQuery, JSON and AJAX.
• Loading, analyzing and extracting data to and from Elastic Search with Python.
• DB2 SQL Procedures and UNIX Shell Scripts were designed and developed for Data Import/Export and Conversions.
• Experience in creating Kafka producer and Kafka consumer for Spark streaming.
• Developed an information pipeline utilizing Kafka and Storm to store data into HDFS.
• Loading spilling data using Kafka, Flume and real time Using Spark and Storm.
• Performed Kafka analysis, feature selection, feature extraction using Apache Spark Machine.
• Used Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames.
• Develop and Execute scripts on AWS Lambda to generate AWS CloudFormation template.
• Microservice architecture development using Python and Docker on an Ubuntu Linux platform using HTTP/REST interfaces with deployment into a multi-node Kubernetes environment.
• Loading spilling data using Kafka, Flume and real time Using Spark and Storm.
• Implemented Spark SQL to access Hive tables into Spark for faster processing of data.
• Used Spark SQL with Python for creating data frames and performed transformations on data frames like adding schema manually, casting, joining data frames before storing them.
• Worked on Spark streaming using Apache Kafka for real time data processing and implemented Oozie job for daily import.
• Used Spark Data Frame Operations to perform required Validations in the data and to perform analytics on the Hive data.

Environment: Scala language with Akka framework, Java, J2EE, Hadoop, HDFS, Pig, Nifi, Hive, MapReduce, Sqoop, Kafka, CDH3, Kubernetes, PHP,, Docker,,Cassandra, Python, Oozie, collection, Scala, AWS cloud, storm, Ab Initio, Apache, SQL, Elastic search, NoSQL, Bitbucket, HBase, Flume, Zookeeper, ETL,, Agile.

Python Backend developer

Confidential, King of Prussia, PA

Responsibilities:

• Responsible for gathering requirements, system analysis, design, development, testing and deployment.
• Participated in the complete SDLC process.
• Developed web-based open stack applications using Python and Django for large dataset analysis.
• Extensively used regular expressions and core features in Python using lambda, map, reduce etc and effectively implemented logging feature using python logging library and profiling using cProfile
• Written many programs to parse excel file and process many user data with data validations.
• Used Numpy for Numerical analysis for Insurance premium.
• Used Subversion version control tool to coordinate team-development.
• Developed tools to automate some base tasks using Shell Scripting, Python.
• Designed and Developed User Interface using front-end technologies like HTML, CSS, JavaScript, JQuery, Angular JS, Bootstrap and JSON.
• Worked with regularexpressions,urllib modules.
• Used PySpark to expose Spark API to Python.
• Developed Spark code using Python for faster processing of data on Hive (Hadoop). Developed
• Map Reduce jobs in Python for data cleaning and data processing.
• Used different type of transformations and actions in apache Spark.
• Experience in writing custom User Defined Functions (UDF) in Python for Hadoop (Hive and Pig).
• Used Spark cluster to manipulate RDD's (resilient distributed datasets). And also used concepts of RDD partitions.
• Connecting my SQL database through Spark driver.
• Having experienced in Agile Methodologies, Scrum stories and sprints experience in a Python based environment, along with data analytics, data wrangling and Excel data extracts.
• Developed views and templates with Python and Django's view controller and templating language to create a user-friendly website interface.
• Worked under DEVOPS environment for continuous integration and continuous deployment using Jenkins and puppet.
• Configured and deployed project using the Amazon EC2 on AWS.
• Designed and developed data management system using MySQL. Involved in Agile Methodologies and SCRUM Process.
• Created unit test/regression test framework for working/new code.
• Using version control tool - Git with Jenkins to accumulate all the work done by team members.
• Using agile methodology - SCRUM, along with JIRA for project..
• Developed entire frontend and backend modules using Python on Django Web Framework.
• Responsible for debugging and troubleshooting the web application.
• Participated in writing scripts for test automation

Environment: Python 2.7, Django, C++, Java, JQuery, MySQL, Oracle 11.2, Linux, Django, Eclipse, Shell Scripting, HTML, XHTML, SVN, CSS, AJAX, Bugzilla, JavaScript, Apache Web Server, Apache spark, Git, Jenkins.

Python Developer

Confidential, Miami Lakes, Florida

Responsibilities:

• Worked with Linux systems and RDBMS database on a regular basis in order to ingest data using Sqoop.
• Experience in Cloud based services(AWS) to retrieve the data.
• Worked and expertise hands on scala programming for processing real time information using Spark API's in the cloud environment.
• Using Kafka and Kafka brokers we initiated spark context and processed live streaming information with the help of RDD as is.
• Worked on Spark using Python and Spark SQL for faster testing and processing of data.
• Involved in enabling Amazon Kinesis firehose to capture streaming data directly on to S3 and also Red Shift. It automatically scales to match the throughput of your data and requires no ongoing administration.
• Developed and maintained the continuous integration and deployment systems using Jenkins, ANT, Akka and MAVEN.
• Used Akka as a framework to create reactive, distributed, parallel and resilient concurrent applications in Scala.
• Installed, Configured Talend ETL on single and multi-server environments.
• Experience in creating tables, dropping and altered at run time without blocking updates and queries using HBase and Hive.
• Developed ETL test scripts based on technical specifications/Data design documents and Source to Target mappings.
• Used Spark API over Hortonwork Hadoop YARN to perform analytics on data in Hive.
• Hands-on experience with Hortonwork tools like Tez and Ambari.
• Worked on Apache Nifi as ETL tool for batch processing and real time processing.
• Extracted files from MongoDB through Sqoop and placed in HDFS and processed.
• Writing user console page in lift along with the snippets in Scala. The product is responsible to give access to the user to all their credentials and privileges within the system
• Used Oozie workflow engine to create the workflows and automate the MapReduce, Hive, Pig jobs.
• Implemented SEO based Drupal modules optimizing the search function all over the site.
• Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop. Cluster co-ordination through Zookeeper.
• Used a LAMBDA EXPRESSION to improve SackEmployees further and avoid the need for a separate class.
• Developed Unix shell scripts to load large number of files into HDFS from Linux File System.
• Experience in creating hive tables HiveQL.
• Using HIVE join queries to join multiple tables of a source system and load them into Elastic Search Tables.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.

Environment: Hadoop ecosystem components, ETL, Spark, Kafka, Python, Shell Scripting, SQL Talend, Elastic search, solr, Linux- Ubuntu, AWS, Hortonworks, MongoDB, VPC, Lambda, Hive, Zookeeper, Pig, Sqoop, Oozie, Solr, Tez, Ambari, YARN, Akka, Jenkins, Kinesis, ANT, Map Reduce.

Python Developer

Confidential, Dallas, TX

Responsibilities:

• Designed and developed Use-Case Diagrams, Class Diagrams, Object Diagrams using UML Rational Rose for OOA/OOD techniques.
• Programmed the components using C/C++.
• Software release to internal clients by compiling the code into C/C++ libraries and implementing the APIs.
• Created UI using HTML CSS and Javascript and monitored and trouble shooted the data loading problems of word Informatica.
• Created Servlets and Beans to implement Business Logic.
• Used SAX/DOM Parser for parsing the data to Oracle Database.
• Developing step definition code in cucumber using jemmy and jUnit
• Used Django framework for application development.
• Created entire application using Python, MySQL and Linux.
• Enhanced existing automated solutions, such as the Editorial Tool for automated request reporting and the Who's Who site, by adding new features and fixing bugs.
• Created database using MySQL, wrote several queries to extract/store data.

Environment: C, C++, HTML, CSS, PL/SQL, Linux and Windows, Python, Linux, HTML, CSS, Django, infomatica, OOA/OOD.

Java Developer

Confidential, Boston, MA

Responsibilities:

• Interacted with the stake-holders and gather requirements and business artifacts based on Agile SCRUM methodology.
• Developed web pages by using HTML, CSS and JavaScript.
• Used Eclipse to develop source code and debug the application.
• Built a custom cross-platform architecture using Java, Spring Core/MVC, and Hibernate.
• Implemented Web-Services to integrate between different applications internal and third party components using SOAP and REST services using Apache-CXF
• Configured Queues and Topics as part of application integration and defined Camel routing between queues/topics.
• Reviewed the XML logical data model and developed XML schema XSD to validate the model and used JAX-B for Xml-Java Mapping and XML-XSLT conversion.
• Developed and published web-services using WSDL,SOAP and XML, JAX-B based on top-down and bottom-up approaches.
• Experience in pre-testing the Web-Service calls with SOAP UI using end-points.
• Coordinated Unit Testing by JUnit, Easy Mock, Integration and User Acceptance Testing and developed test cases for bean components.
• Used Subversion for maintaining the component and for release and version management. And JIRA for defect tracking.
• Maven tool has been used for the build and deploy procedure scripts in UNIX, Linux and Windows environment using Hudson.
• Been though Release process for project execution as part of SCRUM.

Environment: Java, J2EE,SOA, Spring, Web Services, CXF, XML, XSD, Shell Script, UML, HTML, JavaScript, Eclipse, WebLogic, Active MQ, JAX-B, JMS, Axis2, AJAX, Hudson, Maven, Jenkins, Restful, SOAP, JIRA, Oracle, PL/SQL, XPath, XSLT, SOAP UI, JUnit, SVN, Windows, Unix