SUMMARY

• Experience in all phases of Data pipeline Administration using Kafka and Python
• Experience in Leading deploying and managing multi - node development, testing and production of Kafka Cluster’s
• Implemented Schema Registry, Rest API and SSL protocols
• Immense Knowledge on Kafka Streams, Kafka SQL, Kafka Connect
• Experience in using load balancer’s, disaster recovery and traffic routings
• Diverse experience utilizing cloud technologies like Amazon, GCP and microservices.
• Proficient in monitoring technologies Kafka Manager, Grafana, Kibana and Kafka tool
• Expert database noledge; Elasticsearch, Prometheus, MySQL and
• Strong automation development experience in using Ansible and Python.
• Have good understanding of performance testing using Locust, Perftest and JMeter
• Created batch jobs using Python celery and Spark.
• Maintained housekeeping using Python and Shell scripting.
• Possess strong working noledge of collecting metrics using JMX, beats and Prometheus agents.
• Strong expertise in handling and solving issues on Consumer groups and certificate managements.
• Good understanding in Deployment of Hadoop Clusters using Automated Puppet scripts.
• Worked wif teh Linux administration team to prepare and configure teh systems to support for installations and deployments.

TECHNICAL SKILLS

Domain: Kafka Administration, System Management, Pipeline Architectures, Data manipulations, Data Processing and Pipeline Managements

Programing Languages: Python, Shell, Go

Routers/Middleware’s: Nginx, Kong, GCP Load balancer

Automation: Ansible and Salt Stack

Databases: Elasticsearch, Prometheus, MySQL

Tools: Kafka Manager, Grafana, Kibana

Testing Frameworks: Locust, JMeter

Operating Systems: Windows, Linux and UNIX

PROFESSIONAL EXPERIENCE

Kafka Admin

Confidential

Responsibilities:

• Maintaining Data pipeline (Kafka) for teh entire traffic
• Deploying and architecture planning’s for Pivotal
• Designed and implemented by configuring Topics in new Kafka cluster in all environment.
• Exposure and Knowledge of managing streaming platform on cloud provider (Azure, AWS & EMC)
• Efficiently Worked wif all of teh following tools/Instances but not limited to including: Kafka, Zookeeper, Console Producer, Console Consumer, Kafka Tool, File Beat, Metric Beat, Elastic Search, Logstash, Kibana, Spring Tool Suite, Apache Tomcat Server etc.
• Used Apache Nifi for ingestion of data from teh IBM MQ's (Messages Queue)
• Implemented Nifi flow topologies to perform cleansing operations before moving data into HDFS.
• Started using Apache NiFi to copy teh data from local file system to HDP
• Worked wif Nifi for managing teh flow of data from source to HDFS.
• Operations - Worked on Enabling JMX metrics.
• Operations - Involved wif data cleanup for JSON and XML responses dat were generated.
• Successfully secured teh Kafka cluster wif Kerberos Implemented Kafka Security Features using SSL and wifout Kerberos. Further wif more grain-fines Security me set up Kerberos to have users and groups this will enable more advanced security features.
• Integrated Apache Kafka for data ingestion
• Successfully Generated consumer group lags from kafka using their API Kafka- Used for building real-time data pipelines between clusters.
• Created POC for multiple use cases related to CBRE’s Homebuilt Application SEQUENTRA and client LEASE ACCELERATOR
• Complete noledge regarding Elasticsearch, Logstash and Kibana.
• Installed Hadoop cluster and worked wif big data analysis tools including hive
• Created and wrote shell scripts (kasha, Bash), Ruby, Python and PowerShell for setting up baselines, branching, merging, and automation processes across teh environments using SCM tools like GIT, Subversion (SVN), Stash and TFS on Linux and windows platforms.
• Design, build and manage teh ELK (ElasticSearch, Logstash Kibana) cluster forcentralized logging and search functionalities for teh App.Responsible to designing and deploying new ELK clusters (Elasticsearch, logstash, Kibana,beats, Kafka, zookeeper etc.Installed Kerberos secured kafka cluster wif no encryption on Dev and Prod. Also set up Kafka ACL's into it
• Successfully did set up a no autantication kafka listener in parallel wif Kerberos (SASL) Listener. Also me tested non autanticated user (Anonymous user) in parallel wif Kerberos user.
• Installed Ranger in all environments for Second Level of security in Kafka Broker.
• Involved in Data Ingestion Process to Production cluster.
• Worked on Oozie Job Scheduler
• Worked on Spark Transformation Process, RDD Operations, Data Frames, Validate Spark Plug-in for Avro Data format (Receiving gzip data compression Data and produce Avro Data into HDFS files).
• Installed Docker for utilizing ELK, Influxdb, and Kerberos.
• Involved in defining test automation strategy and test scenarios, created automated test cases, test plans and executed tests using Selenium WebDriver and JAVA. Architected Selenium framework which TEMPhas integrations for API automation, database automation and mobile automation.
• Executed and maintained Selenium test automation scriptb
• Created Database on InfluxDB also worked on Interface, created for Kafka also checked teh measurements on Databases
• Created a Bash Scripting wif Awk formatted text to send metrics to InfluxDB.
• Enabled influxDB and Configured Influx database source into Grafana interface
• Succeeded in deploying of ElasticSearch 5.3.0, Influx DB 1.2 on teh Prod machine in a Docker container.
• Created a Cron Job those will execute a program dat will start teh ingestion process. Teh Data is read in, converted to Avro, and written to teh HDFS files
• Successfully Upgraded HDP 2.5 to 2.6 in all environment Software patches and upgrades.
• Worked on Kafka Backup Index, Log4j appender minimized logs and Pointed ambari server logs to NAS Storage.
• Deployed Data lake cluster wif Hortonworks Ambari on AWS using EC2 and S3.
• Installed teh Apache Kafka cluster and Confluent Kafka open source in different environments.
• Basically, one can install kafka open source or confluent version on windows and Linux/Unix systems.
• Implemented real time log analytics pipeline using Confluent Kafka, storm, elastic search Logstash kibana, and greenplum.
• We need to install jdk 1.8 or later and make accessible to teh entire box.
• 3Download teh Apache kafka opensource and Apache zookeeper and start configuring in teh box where we want to run teh cluster.
• nce both kafka and zookeeper up and running, we will be able to create teh topics. Later we can produce and consume teh data. To make it secure, plugin teh security configuration wif SSL encryption, SASL Autantication and ACLs.
• Finally, creating teh backup, adding clients, corgis, patch up and monitoring.
• Intial design we can start wif single node or three node cluster and start adding teh nodes wherever requires.
• Teh required features are CPU core:24, RAM memory:32/64 GB and Main Memory:500GB(least case) to 2 TB.
• Basically usuage is for functional flow of data in parallel processing and distribute streaming platform.
• Kafka replaces teh traditional pub-sub model wif ease, fault-tolerant, high thorughtput and low latency.
• Installed and developed different POC's for different application/infrastructure teams both in Apache Kafka and Confluent open source for multiple clients.
• Installing, monitoring and maintenance of teh clusters in all environments.
• Installed single node-single broker and multi-node multi broker clusters and encrypted wif SSL/TLS, autanticate wif SASL/PLAINTEXT, SASL/SCRAM and SASL/GSSAPI (Kerberos).
• Integrated topic-level security and teh cluster is full up and running for 24/7.
• Installed Confluent Enterprise in Docker and kubernetes in a 18-node cluster.
• Installed Confluent Kafka, applied security to it and monitoring wif Confluent control center.
• Involved in clustering wif Cloudera and Hortonworks and not exposing zookeeper, provided teh cluster to end user using teh Kafka-connect to communicate.
• Setup redundancy to teh cluster and using teh monitoring tools like yahoo-Kafka manager and setup performance tuning to get teh data in real time approach wifout any latency.
• Supported and worked for teh Docker team to install Apache Kafka cluster in multimode and enabled security in teh DEV environment.
• Worked on Disk space issues in Production Environment by monitoring how fast dat space is filled, review what is being logged created a long-term fix for this issue (Minimize Info, Debug, Fatal Logs, and Audit Logs).
• Installed Kafka manager for consumer lags and for monitoring Kafka metrics also this TEMPhas been used for adding topics, Partitions etc.
• Successfully Generated consumer group lags from Kafka using their API
• Successfully did set up a no autantication Kafka listener in parallel wif Kerberos (SASL) Listener. In addition, me tested non-autanticated user (Anonymous user) in parallel wif Kerberos user.
• Installed Ranger in all environments for Second Level of security in Kafka Broker.
• Grid planning and monitoring for Pivotal in-mem database
• Tracking and delivering teh issues wif strict SLA’s
• VMware workstation managements
• Planning and assigning sprints (Scrum master)
• POC on data parser’s and suggesting pipeline changes based on customer needs

Technology & Tools: Pivotol, Confluent Kafka, Python, Altoros, Dynatrace and Splunk

Kafka Admin

Confidential, CA

Responsibilities:

• Maintaining Data pipeline (Kafka) for teh billing records
• Managing and assigning team tasks of size 7
• Manage and monitor Kafka cluster (48 nodes)
• Automating installations and deployments using Ansible
• Housekeeping using Python and Shell scripts
• Keeping track of SLA and issues
• Providing noledge to team on security protocols
• Providing Architectural changes for business needs
• Collecting and maintaining performance metrics
• Monitoring performance using charts and fine tuning on necessary

Technology & Tools: Apache Kafka, Confluent Kafka, Grafana, Python, Ansible, Nginx, GCP LB, Shell, Prometheus

Kafka Admin

Confidential, San Francisco

Responsibilities:

• Migrating from Apache Kafka to Confluent Kafka on ease wifout loss of data and zero downtimes
• Merging multiple clusters into a single cluster
• Maintaining multi name spaces in Zookeeper and reducing cluster sizes
• Providing security to data using SSL Protocols and key files
• Estimating teh cluster capacity and teh business needs
• Predicting teh future requirements
• Technology selections based on teh resource availabilities and environment situations

Technology & Tools: Confluent Kafka, Apache Kong, Cassandra

Confidential

Responsibilities:

• Creating instances from teh VM’s
• Enabling auto scaling
• Moving VM’s to pods
• Enabling firewalls between networks
• Providing login’s to users using ssh keys

Technology & Tools: GCP, AWS, IAM