SUMMARY

• Around 5 years of Expertise in designing, developing and maintaining web based, desktop and interactive applications.
• Responsible for delivering end to end technical solutions as per the architecture and business strategies and have ability to mentor and lead Team.
• In - depth understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming.
• Knowledge on Spark framework for batch and real-time data processing.
• Excellent Programming skills at a higher level of abstraction using Python and Spark.
• Extensively used Spark, Numpy, Pandas, and SQL for Data Analysis and Model building.
• Knowledge on Scala Programming Language.
• Extensive experience in Backend and UI development using Java, Multithreading JEE, Java frameworks like Structs, Spring, Hibernate, JavaScript frameworks and UI technologies. Excellent Experience in Java/JEE Technologies like JSP, Servlet etc.
• Experience in job workflow scheduling and monitoring tools like Airflow and Autosys.
• Excellent knowledge of Python, JavaScript frameworks and UI technologies like AngularJS.
• Experienced in writing SQL queries, Procedures, Functions, Triggers.
• Expertise in using IDE like Eclipse, NetBeans, IntelliJ PyCharm, Jupiter Notebook and Spyder.
• Strong experience in deployment of application on application server like Tomcat, Glassfish, CGI.
• Experienced in working with distributed team in collaborative way to deliver the common solutions.
• Possessing good interpersonal skills which were applied in coordinating with business users, Offshore and on-site development team, QA team and support team as required.
• Involved in designing the backend database for the application, worked on Fusion charts and Tableau for generating the reports.

TECHNICAL SKILLS

Operating Systems: Windows, Unix and Linux.

Technologies: Python, SPARK, AWS, Hive, HDFS, GCP, Core java, Swings, JDBC, JSP, Servlets, EJB 3.0, JPA, JSF, log4j, SQL, PL-SQL, P4Java and SOLR.

Frameworks: Pandas, NumPy, matplotlib, Google app engine

Web Technologies: HTML, JavaScript, XML, CSS, JSON, and Angular JS.

Build Tools: Ant, Maven and sbt.

Utilities, Tools and plug-ins: Airflow, Autosys, Eclipse, and Google cloud pub/sub, IntelliJ PyCharm, Jupiter Notebooks, Jedi’s, Net Beans, putty, SQL Developer, Spyder, Visio.

Data base / Cloud: AWS, Google Cloud, Oracle, MySQL, MS SQL, Google Big Query, Apache Cassandra, Redis, Memcached, PostgreSQL, Teradata, Google Storage.

Version Control: Mercurial, Google cloud source repository, Git, Bit Bucket.

Methodologies: Agile/Scrum and Waterfall model

PROFESSIONAL EXPERIENCE

Python Developer

Confidential, NJ

Responsibilities:

• Involved in identifying the workflows in Information to migrate the historical data from Teradata, SQL Server.
• Implemented workflow sequences using scheduling tools - Apache Airflow and Autosys
• Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and SQL queries.
• Developed JIL scripts to move the historical CSV files to Google Storage.
• Created and configured the bucket structure in Google Cloud Storage to load, process and archive the source CSV files.
• Used Google cloud Pub/Sub service to notify the arrival of files in GS from Source Systems.
• Developed Directed Acyclic Graphs (DAG) using Python to define the dependency between various tasks.
• Loaded the data into Spark RDD and do in memory data computation to generate the output response.
• Experienced in Performing tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Analyzed the SQL scripts and designed the solution to implement using Pyspark
• Developed custom airflow operators using Python to generate and load CSV files into GS from SQL Server and Oracle databases.
• Developed python framework using various Python libraries like Pandas and NumPy for Data validation
• Extensively used Tableau to generate reports on a daily basis.

Environment: Python, Airflow, Autosys, BigQuery, Google Storage, Google Pub/Sub Notifications, Spark, Git

Software Engineer

Confidential

Responsibilities:

• Involved in analyzing the requirement and specification and design technical solution.
• Worked on integrating Real User Monitoring with APM Insight.
• Designed the MySQL database and created dynamic tables to store various performance metrics.
• Developed servlets to handle auto-upgrade requests from java, .Net and Python agents.
• Designed the controller and router modules to implement the MVC framework for the UI using AngularJS.
• Used Memcached and Redis to store the real time data.
• Responsible for code review and application deployment in UAT
• Involved in migrating the MySQL data to Apache Cassandra using a custom framework based on XML.
• Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the anomaly detector model which gets the data from Kafka in near real time and persists into Cassandra.
• Developed Spark scripts by using Scala as per the requirement.
• Performed POC on SPARK using AWS EMR.
• Developed UML diagrams using Visio and other tools.
• Involved in unit testing and bug fixes.

Environment: AngularJS, Cassandra, MySQL, Tomcat, HTML, Java, Python, AWS, EMR, SPARK

Software Engineer

Confidential

Responsibilities:

• Involved in requirements gathering by interacting with the customers (application developers)
• Used a dynamic approach to design the MySQL database tables to store the performance metrics.
• Involved in collecting performance metrics using JAVA Instrumentation framework based on the configured time interval.
• Developed a collector servlet to process the various performance metrics.
• Used Memcached to speed up the D3 chart rendering process
• Involved in unit testing and integration testing of the module with other monitoring tools

Environment: Python, Java, Struts, JSP, AngularJS, Tomcat, MySQL, HTML, MySQL, PostgreSQL, AWS, Amazon Redshift

Associate Engineer

Confidential

Responsibilities:

• Involved in designing the various modules involved in the architecture
• Developed a JS to capture the various performance metrics using Navigation Timing API and aggregate the collected metrics in JSON format.
• Designed a hassle-free approach to inject the script in the website’s index file.
• Implemented a collector module using Restful Web Services and Java Servlets which collects five mins data and pushes to Rabbit MQ message queue service.
• Designed the NoSQL Cassandra database which offers horizontal scaling to handle the huge amount of data.
• Developed a coordinator module which listens to the queuing service and archives the data in Cassandra
• Used Redis caching techniques to display real time data.
• Installed and deployed Apache Cassandra, Redis and Rabbit MQ in AWS EC2 instances
• Develop UI using AngularJS
• Worked on complex reports by using Joins and Unions to render the performance charts using D3 based on browser, device and more for various time windows.

Environment: Java, JSP, Tomcat, Rabbit MQ, Redis, Cassandra, Fusion Charts, AWS