Hadoop Admin Resume
Denver, CO
SUMMARY
- Over 10 years of professional IT experience in Business Analysis, Design, Data Modeling, Development and Implementation of various client server and decision support system environments with focus on Big Data, Data Warehousing, Business Intelligence and Database Applications.
- Over 3 Years of experience in dealing with Apache Hadoop components like HDFS, MapReduce, HIVE, Hbase, PIG, SQOOP, Spark, Impala, OOZIE, Kafka and Flume Big Data and Big Data Analytics.
- Over 1 years of Informatica Power Center Consulting with Data mapping, Transformation and Loading from Source to Target Databases, well versed in various Ab Initio parallelism techniques and implemented Ab Initio Graphs using Data, Component, pipeline parallelism and Multi File System (MFS), techniques in complex, high volume Data Warehousing projects in both UNIX and Windows.
- In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, NameNode, Resource Manager, DataNode, Node Manager and Map Reduce concepts.
- Experience in installation, configuration, support and management of a Hadoop Cluster.
- Experience in task automation using Oozie, cluster co - ordination through Pentaho and MapReduce job scheduling using Fair Scheduler.
- Experience in analyzing data using HiveQL, Pig Latin and custom Map Reduce programs in Java.
- Experience in writing custom UDF’s to extend Hive and Pig core functionality.
- GotexperienceinmanagingandreviewingHadoopLogfiles.
- Worked with Sqoop to move (import/export) data from a relational database into Hadoop and used FLUME to collect data and populate Hadoop.
- Worked with HBase to conduct quick look ups (updates, inserts and deletes) in Hadoop.
- Experience in working with cloud infrastructure like Amazon Web Services (AWS) and Rackspace.
- Experience in Core Java, Hadoop Map Reduce related program. Used Hive to transfer the data from RDBMS to our Hive data warehouse.
- Experience in writing Pig Latin. Use Pig Interpreter to run Map Reduce jobs.
- Experience in storing and managing data on H-catalog data model.
- Experience in writing SQL queries to process some joins on Hive table and No SQL Database.
- Experience in Agile Methodology, Micro Service Management, tracking and bug tracking using JIRA Working experience on designing and implementing complete end-to-end Hadoop Infrastructure including Pig, Hive, Sqoop, Oozie and Zookeeper.
- Experience in DBMS Utilities such as SQL, PL/SQL, TOAD, SQL*Loader, Teradata SQL Assistant.
- Experienced with Teradata utilities Fast Load, Multi Load, BTEQ scripting, Fast Export, OleLoad, SQL Assistant.
- Skillfully exploit OLAP analytical power of Teradata by using OLAP functions such as Rank, Quantile, Csum, MSum, group by grouping set etc to generate detail reports for marketing folks.
- Experienced in all phases of Software Development Life Cycle (SDLC).
- Expert knowledge in using various Transformation components such as Join, Lookup, Update, Router, Normalize, De-normalize, Partitioning and De-partitioning components etc.
- Experience in Data Modeling, Data Extraction, Data Migration, Data Integration, Data Testing and Data Warehousing using Ab Initio.
- Configured Informatica environment to connect to different databases using DB config, Input Table, Output Table, Update table Components.
- Able to interact effectively with other members of the Business Engineering, Quality Assurance, Users and other teams involved with the System Development Life cycle
- Excellent Communication skills in interacting with various people of different levels on all projects and also playing an active role in Business Analysis.
- Experienced in Automation of software testing process using Quick Test Pro, Load Runner and Quality Center), Rational Clear Case and Clear Quest.
- Excellent skills in testing web and Client/Server based applications.
- Experienced in Designing Test plans, Test Cases and Test Scripts.
- Strong skills in performing System, Acceptance, Regression, Stress, Performance, Load, Functionality, Front End and Back End Testing.
- Experienced in Backend Testing of the applications by executing SQL commands.
- Expertise in testing Performance, Load and Stress for Web and Client/Server based Applications.
PROFESSIONAL EXPERIENCE
Confidential, Denver, CO
Hadoop Admin
Responsibilities:
- Tested loading the raw data, populate staging tables and store the refined data in partitioned tables in the EDW.
- Sqoop configuration of JDBC drivers for respective relational databases, controlling parallelism, controlling dist-chache, controlling import process, compression codecs, importing data to hive, hbase, incremental imports, configure saved jobs and passwords, free form query option and trouble shooting.
- Involved in Cluster Level Security, Security of perimeter (Authentication- Cloudera Manager, Active directory and Kerberoes) Access (Authorization and permissions- Sentry) Visibility (Audit and Lineage - Navigator) Data ( Data Encryption Confidential Rest)
- Collection and aggregation of large amounts of streaming data into HDFS using Flume Configuration of Multiple Agents, Flume Sources, Sinks, Channels, Interceptors defined channel selectors to multiplex data into different sinks and log4j properties
- Troubleshooting issues in the execution of Mapreduce jobs by inspecting and reviewing log files
- Worked on setting up high availability for major production cluster and designed automatic failover control using zookeeper and quorum journal nodes.
- Assisted in configuration, development and testing of AutoSys JIL and other scripts
- Involved in development and maintenance of AutoSys and related scheduling solutions.
- Documented the systems processes and procedures for future references
- Worked with systems engineering team to plan and deploy new Hadoop environments and expand existing Hadoop clusters..
- Monitored workload, job performance and capacity planning using Cloudera Manager
- Worked with hadoop tools like Flume, Hive, Sqoop and Oozie on the Hadoop cluster.
- Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.
Environment: Hadoop, HDFS, MapReduce, Yarn, Hive, Pig, Sqoop, Oozie, Flume, Zookeeper, Hbase.
Confidential, Mountain View, CA
Hadoop Admin
Responsibilities:
- Worked on hadoop cluster with 450 nodes on cloudera distribution 5.0.1.
- Delivered end to end project. Form requirement gathering to development and testing.
- Ingested data from different sources into Hadoop
- Created external tables with proper partitions for efficiency and loaded the structured data in HDFS resulted from MR jobs.
- Migrated the existing data to Hadoop from RDBMS (SQL Server and Oracle) using sqoop for processing the data.
- Involved with various teams on and offshore for understanding of the data that is imported from their source.
- Involved in data visualization and provided the files required for the team by analyzing the data in hive and developed Pig scripts for advanced analytics on the data.
- Provided updates in daily SCRUM and Self planning on start of sprint and provided the planned task using JIRA. In sync up with team in order to pick priority task and update necessary documentation in WIKI.
- Worked on Performance tuning on Hive SQLs.
- Did all the documentation of the project.
- Loaded data from hive to netezza and build tableau reports for the end user.
- Weekly meetings with Business partners and active participation in review sessions with other developers and Manager.
Environment: Hadoop, HDFS, MapReduce, Yarn, Hive, Pig, Sqoop, Oozie, Flume, Zookeeper, AutoSys, Teradata
Confidential, Dallas, TX
Hadoop Admin/Tester
Responsibilities:
- Collected the logs data from web servers and integrated into HDFS using Flume.
- Checking the Oozie job logs running multiple Hive Jobs
- Checking the last jobs run and getting more info on job id of the failed jobs
- Tested data pipeline using Flume, Sqoop and map reduce to ingest customer data into HDFS for analysis.
- Used AutoSys to create and run command line, file watcher, box jobs. View processing events and set alarms
- Repomsible for Cluster security like Identity, Authorization and Authentication including Active Directory,Kerberos and Sentry.
- Used Sqoop and Exporting/Importing data into HDFS
- Experienced in loading data from UNIX local file system to HDFS.
- Use of Sqoop to import and export data from HDFS to Relational database and vice-versa.
- Data analysis in running Hive queries.
- Tested the roles, groups and privileges created in Sentry reflected in the Cloudera Navigator logs
- Cloudera Navigator Audtor . Created transaction using hive and Impala query editors and check the entries in the navigator logs.
- Tested Cloudera Navigator masking the sensitive data like the account number, social security numbers in the logs. Created audit logs. Customized the audit logs as requested.
- Created reports in Cloudera Navigator
Environment: Hadoop, HDFS, MapReduce, Yarn, Hive, Pig, Sqoop, Oozie, Flume, Zookeeper, Hbase, Cloudera Navigator
Confidential, Pleasanton, CA
Hadoop Admin
Responsibilities:
- Installing and configuring Hive, Flume, Sentry, Sqoop, Kerberos, Hue, Navigator, generating dashboards and metrics. Managing hadoop cluster, services, activity, events and alerts using cloudera manager.
- Responsible of health checks of cluster, disk mantaintence, adding nodes, decommissioning nodes and troubleshooting job failures, latency and node failure
- Involved in security of perimeter (Authentication- Cloudera Manager and Kerberoes) Access (Authorization and permissions- Sentry) and also worked with IAM team
- Validate the accessibility of the end users. Validation of application startup process Helping new user accessing the Hadoop cluster through hue UI or through edge node.
- Importing into, exporting out and manipulating data in HDFS.
Environment: Hadoop, HDFS, MapReduce, Yarn, Hive, Pig, Sqoop, Oozie, Flume, Zookeeper, AutoSys, Teradata, Sentry, Kerberoes
Confidential, Irving, TX
Responsibilities:
- Deploy, administer and manage Hadoop Software on large cluster implementations
- Importing into, exporting out and manipulating data in HDFS.
- Management of Hadoop log files
- Hadoop cluster maintenance like adding nodes, decommissioning nodes, monitoring health of the cluster and trouble shooting
- Hadoop mapreduce cluster maintenance like monitoring the health of the mapred cluster, maintaining task trackers
- Managing mapreduce jobs like submitting jobs, checking status, and changing status of the jobs
- Managing tasks like listing jobs, killing jobs and failing jobs
- Installation and Configuring schedulers, Flume, Scoop, Oozie
- Experience in day to day production support of Hadoop infrastructure like HDFS maintenance, Backups, manage and review Hadoop log files
- Installation and configuration knowledge of Ganglia tool.
- Tuning the cluster through the configuration of job tracker, tasktracker, memory etc
- Tuned the quotas Confidential directory, user and bandwidth level.
- Balancing clusters
- Monitor system health and logs and respond accordingly to any warning or failure conditions
- Commission/decommission nodes as needed.
- Maintain system integrity of all sub-components (primarily HDFS, MR, HBase, Flume, Oozie, Scoop)
Environment: Linux, Java, CDH4, Apache HDFS, Map Reduce, Pig, Hive, HBase, Flume, Sqoop
Confidential, Plano TX
TESTER
Responsibilities:
- Developed mappings, workflows and sessions and scheduled them into ControlM.
- Involved in ETL test script design for the new requirements.
- Interacted with Business Analysts in regards to clarification requirements
- Created Requirements Test Matrix (RTM) to keep track of the requirements
- Involved in design session meeting to understand how the system is developed.
- Developed test plans and executed test scripts in Quality Center
- Infrastructure people set up test environments.
- Tested the performance of the system when extracting a large amount of data.
- Involved in business analysis and technical design sessions to develop requirements traceability and test plan document, and ETL specifications
- Developed standard and re-usable mappings and mapplets using various transformations like expression, aggregator, joiner, source qualifier, router, lookup, and Router
- Designed ETL process using Informatica Tool to load from Sources to Targets through data Transformations
- Used TOAD for SQL.
- Used Debugger for debugging Mappings.
Environment: Informatica Power center 8.6/9.1, ControlM Scheduler 7.0, ControlM Desktop, ControlM Enterprise Manager, Toad, PL/SQL (Stored Procedure, Trigger, Packages), Force explorer 0.58A Oracle 9i, 10g
Confidential, Mckinney, TX
QA Test Lead
Responsibilities:
- The test cases were documented in Enterprise tester
- The defects and requirements were filed in the Jira
- Base camp was used to stage all the documents and results
- SoapUI and Rest Client were the tools used for the API testing.
- Samsung TV SDK tool was used to down load the app to the Samsung devices
- Used Toad and SQL Developer were the tools use to query of the oracle database.
- Cross device testing and browser compatibility testing were also part of this testing.
- Performed command line driven testing on Confidential POS running VMS or Linux operating systems.
- Used Reflections tool to ftp the mst, dump and data files form the VMS systems to the desktop to analyze data.
- Ran SQL queries against MySQL database to verify data input for the Pos on linux systems.
- Performed functional and regression testing on POS system, On Demand and documented results in Enterprise Tester.
Environment: Java/J2EE WebLogic Oracle 10g, Windows XP and Sun Solaris, Mercury Interactive Performance Center, Mercury Interactive LoadRunner 8.1 FP4, HP Quality Center 9.0.
Confidential, Irving, TX
QA LEAD
Responsibilities:
- Lead a team of 10 members
- Was responsible for the testing the SIT environment.
- Document the test cases of IMG Project North in Quality Center
- The team including me executed the test cases. I was responsible in allocating the modules to the team. We used rational Clear Quest in for logging the defects and the reports.
- Created defect reports sent the development team and testers.
- Conducted defect tracking meetings to get ETA from the developers on the defects.
- These meeting were also to facilitate the developers in understanding the defects and get more information if needed from the testers.
- The Confidential Fios sold the 14 states of its operation to Frontier Communications. I with a team of three was in Fort Wayne. The data centers were migrated from blue hill to Fort Wayne. I was responsible to test the connectivity of the modules. We tested the connectivity to the servers. This was in the month of March.
- The project would go live on may16th for user friendly customers. The team is testing the IMG code, all the modules in the IMG code and all the other modules that need server connectivity. We are here to see the deployment is done successfully.
- For the third time in the month of June I am in Fort Wayne, IN., to see production release was successful with no issues.
Environment: Java/J2EE WebLogic Oracle 10g, Windows XP and Sun Solaris, Mercury Interactive Performance Center, Mercury Interactive LoadRunner 8.1 FP4, HP Quality Center 9.0.
Confidential, Richardson, TX
QA LEAD
Responsibilities:
- Created UML for the business process flows for different scenarios of activation, updates showing the information sent in the request and the information got in response
- Created reusable Business Components in quality center 9.0 and 10.0 versions.
- Worked closely with architects and developers in creating the UML diagrams
- Involved with business analyst team in collecting the requirements. I was involved in the brain storming session. I was also involved in the JAD sessions.
- Create Business Process Test Cases in Quality Center
- Tested activities of POS applications like OPUS (POS of Confidential stores) and PDC2 (POS of Confidential partners BEST BUY, WAL-MART etc)
- The main activities tested through POS new equipment sales and upgrades. Receiving returned equipment, accepting damaged equipment and sending for repair. One other main set of activities is cash register activities.
- Tested the Integration with the Inventory Control Module
- Created usage for all data types like SMS, MMS, VSC, GPRS, VXE, and PTT.
- Coordinated with Enabler team in processing data usage
- Worked with Tech support team in processing billing
- Added Relevant SOCs and validated the usage in all retail application like OPUS, PDC2, OLAM and Telegence
- Validation included the direction of the calls, location origination and terminating, the rating of the call, the allowance on the SOC etc.
- Processed usage in UNIX using Unix Scripts.
- Used Perl for Scripting
- Used Seibel based BASE application create Large, Commercial and Individual foundation accounts
- Lead of team of 8 members
- Gathered requirements for scenarios under automation
- Created automation test plans test cases and defect reports for new product releases
- Created test scripts using QTP in action level and executed these test scripts in Quality Center
- Created test scripts in QTP and enhanced those scripts with VB script (programming logic)
- Maintained function libraries and created new function libraries and created new repositories and maintained object repositories
- Executed automated integration, data driven and regression test scripts
- Evaluated reports and implemented process improvement
Environment: Java/J2EE WebLogic Oracle 10g, Windows XP and Sun Solaris, Mercury Interactive Performance Center, Mercury Interactive LoadRunner 8.1 FP4, HP Quality Center 9.0.
Confidential, Plano, TX
QA LEAD / Configuration Management Engineer
Responsibilities:
- The requirements of the application were not clear. Worked with PMOs, Business Analysts and developers in clarifying the requirements
- Involved with business analyst team in collecting the requirements. I was involved in the brain storming session. I was also involved in the JAD sessions
- Created the Test Plan and Test Scripts for the projects in Quality Center
- Used Clear Quest to log the defects.
- Reported the status the senior management on the project.
- Worked on Siebel, Metsolv TBS, PIMS, Scopus and IPS Applications
- Worked on Creating Accounts through XOG application
- MSAG validated and Vertex validated the Account through Seibel
- Customized sales orders in Siebel which included the add network, handoff methods, rate plans, equipment, rate centers and markets
- Worked on credit evaluation and submission of the sales order to integration.
- Worked building the order submitted in TBS which was submitted through Siebel
- This included building the circuit, adding the right switches (TN Switches, Sonus Switches) to product based on the location and the type of product.
- Allocated the provisioning plan to the product.
- Assigned Task to the order and completed the Tasks in TBS
- Processed the order through Internet Provisioning System
- Retrieve the order in Scopus and validated the data
- Retrieved the orders in PIMS (Clarify application) accept the case from the Queues and checked the provisional parts and change the status to PEND-FOC and LOG PROV task is completed.
- Check if DPS has sent in notes to TBS with the complete information
- Check TBS has generated a XML and sent to ATS
- Checked in ATS to find out if the provisioning was successful without Errors
- Worked with developers and the engineering team for the status of the order.
- Checked that order passed through ALGX SingleView 5.x
- Retrieved Accounts in ALGX SingleView 5.x and validated the Accounts and the bills
- Gathered requirements for scenarios under automation
- Created automation test plans test cases and defect reports for new product releases
- Created test scripts using QTP in action level and executed these test scripts in Quality Center
- Created test scripts in QTP and enhanced those scripts with VB script (programming logic)
- Maintained function libraries and created new function libraries and created new repositories and maintained object repositories
- Executed automated integration, data driven and regression test scripts
- Evaluated reports and implemented process improvement
- Created automation test plans test cases and defect reports for new product releases
- Created test scripts using QTP in action level and executed these test scripts in Quality Center
- Created test scripts in QTP and enhanced those scripts with VB script (programming logic)
- Maintained function libraries and created new function libraries and created new repositories and maintained object repositories
- Executed automated integration, data driven and regression test scripts
- Evaluated reports and implemented process improvement
Environment: Windows XP, QTP, load runner, Unix shell scripting Remedy, XML, Toad, Seibel 7.5, Seibel 8.0, Clarify, SingleView, ClearCase, CearQuest, Quality Center, Change Management Process, Oracle
Confidential, Richardson, TX
Sr. QA Analyst
Responsibilities:
- Wrote the test plan based on the HLD, BRD, TRD and IA and assisted testing team when more in information is needed for the requirements when developing the test plans.
- Point of contact for the clarification of the requirements.
- Involved with business analyst team in collecting the requirements. I was involved in the brain storming session. I was also involved in the JAD sessions
- Involved in the peer reviews of the test plans and the test cases
- Involved in MAF and PIC testing
- Wrote test cases in excel based on the HLD, IA and Test Plan and exported them to the Quality Center and submitted for Test Case Review
- Used Toad to access the databases and query the databases.
- Involved in the reboot process of the web sphere tuxedos and the taps in the Early mornings
- Created XMLs for creating canceling suspending resuming products and ran Unix scripts to creating canceling suspending resuming products to the bans
- Used the AMC tool to bring down and bring up the demons
- Done file level validation based on the IA document and also negative validation (e.g. existence of header, trailer, record count etc.)
- Used scheduled jobs calendar based and file watcher based
- Used BO to retrieve Information from the database (Ad hoc report and canned report validation)
- Created Usage to meet the test requirement.
- Tested pre billing and post billing adjustments. Write offs and write off reversals, payments via CSM online and Lockbox and payment back outs.
- Coordinated with the AR team for testing the above
- Worked with vertex team (tables’ team) regarding tax related issues like verifying taxes and tax exemptions.
Environment: Windows, XP, Unix shell scripting Mercury Quality Center-8.5, Remedy, KEA, Crimson Editor, Toad, Access FTP Commander, XML, Test Director 8.0, Ensemble, Telegence, Quality Center, API Simulator, Usage Creation, Switch/Provisioning testing, A/R, MAF/MPS, Enabler, Change Management Process
Confidential, Alexandria, MN
QA TEST LEAD
Responsibilities:
- Worked on POS and CSM modules. Wrote test cases for CSM module, reviewed the test cases of POS module executes the test cases
- Worked on the interfaces like Lockline, Assurion, Web authorize, Equifax and Lockbox
- Audited all Price plans in the Midwest region in Product Catalog module
- Assisted the Conversion team in checking data sent in the UF file from the legacy system is populated in the Amdocs system
- Queried the data from AS 400 and imported the data as UF file and sent to Amdocs to populate the data in Telegence tables
- Used AS400 SQL for queries to retrieve data from multiple tables.
- Validated data in AS400 with the data in oracle telegence tables.
- Validated data of telegence online with the data of AS400
- Worked on Collections module. After the data conversion is done the Phase in jobs are run. Tested the results of the phase in jobs. Checked if the conversion phase in jobs placed the customers in the right collection steps and right collection path for the customers who are in delinquent status based on the business rules.
- Work on a part of AR module. Checked if the after conversion the phase in jobs placed the right due amounts in the right age buckets.
- Worked on IVR interface. Viecore, vendor, which is interfacing the Amdocs system and the end user. This has speech recognition technology which helps the user access the data of his account like balances, payment, used minutes, left minutes, and few other CSM activities without CSR intervention. The system switches to the available CSR. When the system does not recognize or when the user opts for a CSR
- Assisted the loading the team in loading the numbers through the Resource Management (RM) module and the equipment and accessories through Inventory Module (IC).
- Also involved in test Automation created basic scripts. Created test data and ran data driven testing using Winrunner
- Debugged the test scripts using Break points
- Reporting the overall status of the modules to the management using the test director reports.
- Communicating the change of action to the testing when there is showstopper.
- Assigning the bugs to the right developer
- Escalating the issues of immediate attention in the bug status meeting.
- Discuss with the PMs on the requirements that need clarification and communicating with the testing team.
- Updating the management through Weekly status report on the progress and the issues in that week.
- Mediating between the programmers and the testing team in case of disagreement of functionality.
Environment: AS400 Windows, Winrunner, Unix shell scripting, Remedy, Access, XML, Toad, Test Director 7.6, Ensemble, Telegence, Quality Center, API, Usage Creation, Switch/Provisioning testing, A/R, Change Management Process