Backup Administrator Resume
Backup Technology expert with 7 years of design, administration and implementation experience of EMC products ranging from EMC Networker (Legato), Avamar, DATA DOMAIN, to other metric tools like Data Protector Adviser, Skilled in disaster recovery operations and upgrades using Legato Networker. Expertise in Data Migration from older to latest Backup technologies. I have knowledge of Symantec NetBackup and HP Data protector.
EXPERIENCE SUMMARY
Backup administrator at Confidential, Feb 2010 - onwards
Backup infrastructure auditor at Confidential, May 2007 -Dec 2009
Networker Administrator at Confidential, Jan 2005 - Apr 2007
Technology Management Confidential, July 2004 - Dec 2004
TECHNICAL SKILLS
Backup: EMC Legato networker 7.6.1, 7.5, 7.4, 7.3, 7.2, 7.1, 7.0 on UNIX and Windows platforms, SQL
Deduplication: EMC Avamar, DATA DOMAIN DD690, DD680, DD890
Tape Libraries: L20, L40, L80, L700, SL500, SL700, PV132T
Virtual Tape Libraries: EDL 4200, EDL 4700, HP D2D
O.S: Solaris 2.6, 2.7, 2.8, 2.10, windows
Backup Metric Tools: EMC Data protector Adviser
Backup administrator,Confidential, Feb 2010 - onwards
Project 1
A Solution comprising of three EMC Networker data zones two Data domains (DD890), two Gen4 Avamar grids and DPA server was designed, architected and implemented by me along with EMC hardware team. The Backup solution comprised of Avamar server (for backing up file systems) and EMC Networker. Virtual tape library option was utilized for Data domain. Since whole operations team at Airtel was comfortable with Physical libraries, it was mutually decided to just use Virtual tape library option of data domain configuration. The Avamar grids were configured to replicate (both ways), in the sense that Avamar 1 will replicate to avamar2 and vice versa. But Data domain was configured to replicate just one way. The networker design included all aspects of intra site recovery as well as pure disaster recovery. In case of real disaster & in absence of primary networker server, second replicated networker server which was pre zoned with replicated data domain will own responsibility as the host will have same IP address and same hostname and the recoveries will be easy. This license server was always planned to be down , will remain up only during DR drill, when we manually bring down primary server and divert all backup traffic to DR networker server. In case of disaster recovery pre identified Bootstrap tapes were to be used to recover catalogue and all configuration, backups would proceed now on this new data zone. There was another DR Networker server stood up at a third location which has different IP address and hostname and was utilized for intra site recoveries and database clone operations using replicated copy of oracle database. Main challenge I faced while designing the solution was making customer and his team to understand the new technology and its usage, and documenting the workflow, deduplication concepts. After the solution was implemented I faced major issues on data domain not displaying enough dedupe ratio for Oracle RMAN backups, that was ironed out with lots of RMAN configuration changes like block tracking etc. After a month or so I was able to achieve a 6X dedupe ratio. Avamar dedupe was always good. I was also involved with Avamar configuration once it was stood up by data Avamar Engineers.
Project 2
This was a pure Data migration project, where in my task was to assess the existing backup solutions with the assistance of existing backup team to understand Number of clients, Total data being backed up daily, weekly and annually, find out clients which had more than 500 GB of data, Special Clients with special deep data structures, total number of databases, and variety of operating systems, un supporting applications and operating systems etc. This information was used by me to devise and propose a solution which would make a backup solution which will meet client's objectives. So overall the whole project was divided into three phases.
First phase: This included EMC Avamar, EMC data domain infrastructure which was intended to backup lots of Oracle and MSSQL databases. Big databases were planned to go to datadomain. We configured datadomain with avamar and also configured nfs file systems and were shared with hosts which had large databases. But avamar was configured to receive all smaller application hosts and some of the larger databases were configured to go to data domain via Avamar. I had configured Exchange backups to go to avamar.
Maintenance window (garbage collection, checkpoint and health check validation was) was configured during day time and backup window was just 5 PM to 8 AM.
Second phase: client migration from old TSM solution to newer Networker solution one to five clients per day, take backups, schedule them gather statistics about backup, recover big chunk of file system and gather statistics. Same time I was designing DPA solution so that it can report against all job failures,
Generate and forecast report for Space consumption , Generate switch logs to identify usage against day and time to identify bottlenecks , generate reports for HBA's on networker servers and storage nodes . After DPA was built its reports helped me fine tune networker environment to increase /decreases parallel sessions, multitasking, spreading the Full/VS incremental load throughout the week etc. Main thrust of my design was not to make the solution very complicated in terms of configurations so that new TSM team picks up the Avamar product easily and starts liking it, as this was the pilot product and its success/failure would have impacted eight more datazones to be built.
Third phase: Another solutioncomprising of two Avamar Gen 4 physical grids and 12 Avamar virtual grids were to be stood up, this was a specific design to capture all the data from small remote sites via virtual Avamar grid and then replicate it to a centric location, this saved huge amount of bandwidth. The design also included replicating the central grid again to another grid strictly for disaster recovery purpose. There were issues in this last piece of configuration. EMC engineering was involved and utility nodes were patched, and we failed to achieve the requirement, ultimately it was left on next Avamar code upgrade.
Backup infrastructure auditor,Confidential May 2007 - 2009
This company maintained three Backup infrastructures which were shared between 75 customer's backups. Each of the customers was assigned separate VLANs and networker server and most of the storage nodes were all Linux hosts. The solution was altered to accommodate the needs of the customer each time. I was hired mainly to troubleshoot issues faced by birlasoft in day to day operations like huge amount of failures / multiple failing recoveries. Networker was bad named NOTWORKER at this site.
It took me one month or so to audit the whole backup solution and reached to a decision that whole solution need to be redesigned with additional hardware to do away manual script based backup clones process and selective encryption methodologies. In order to justify my design I had to document the proofs where things are configured wrongly, multiple backup methodologies were tried to implement on a single backup solution. I not only architected the whole backup solution but also helped redesign many of their applications involving huge data from scratch keeping in mind special needs of customer. My proposal included heavy usage of Global and local zones of Sun Solaris servers. Multiple global zones were assigned drives, and multiple local zones on a single global zone were used for applications. So now only networker server was shared but each customer was assigned one or more global zones. So a total of 12 to 15 storage nodes per networker server were used. All the physical libraries were phased out. The EMC disk libraries and data domains were used to share drives and data and then to replicate using library based replication instead of networker based clones. I would gladly say that backup failures were reduced to a very low percentage. I was involved in installation and migration of the solution, and after implementation I worked on this solution for many a months creating documentation, training staff. I also
Coordinated with various application teams to configure database /application backups, restore tests, disaster recovery procedures, and tests. Preparing for annual audits, surprise mini DR test/recoveries and surprise/emergency test of day today activity with 20% of the workforce reduction.
Networker admin ,Confidential,Jan 2005 - Apr 2007
Activities:
The infrastructure at panacea was all physical tape libraries, and LTO2 tape based, we had three backup solutions which were backing up 1400 Clients, most of them were file system backups, few Exchange backups and few MSSQL databases. There were few Cold Oracle RMAN backups configured. Most of the time was utilized in making a report of failed backups, finding out the clients who are failing for more than three days, try to troubleshoot the failure, and take a test backup after issue resolution, or if issue persists, identify the right team who would fix the issue. Once all backup trouble shootings were done and report sent to management, next task was to find out issues in hardware like bad drives , re-scan drives , identify bad tapes struck tapes etc. Here networker server was at 7.1, it had lots of bugs and issues and lots of work was manual and time consuming , new drive scan was little complicated , and was not as easy as today . There were lots of Solaris hosts, which were losing persistent binding all the times so drives fixing was next task undertaken. I had to plan for weekend activities like storage node reboot or maintenance of any HBA's, in that case I had to direct all backup traffic to different storage nodes , and I had to be very careful , when a storage node was overloaded with work as the processes would not functioning at all .
Later on dependence on physical tape libraries was reduced when we introduced EMC disk libraries, but Cloning operations were still networker based, the backup methodologies were revisited where in primary backups going to EDL were retained for just 15 days and then all the backups had to be cloned to physical tapes for longer duration, this added more issues and complexities where in within a time of 24 hours we were not able to finish all backups and clones etc. An EDL based replication was also tried but that failed it had bugs at that time. So overall it was a challenge working as Networker admin, things started easing when we upgraded to networker 7.2 then 7.3.
My daily tasks:
- Maintaining the integrity of critical elements of the operating systems.
- To ensure reliable backups to meet operational and data recovery requirements.
- To allocate and monitor storage to meet backup and recovery requirements.
- Implement and administer a backup and restore system
- Backup schedules, rotation schedules, and a retention scheme.
- Describe your current backup and recovery process and strategy
- Assess the current resources in your environment.
- Determine backup time using throughput rates
- State the purpose of running parallel backup systems
- Read backup logs to verify a successful backup and restore process
- Monitor the backup and restore process.
- Monthly offline backups
- Location of data to be backed up
- Data growth projections
- Determine Backup window availability
- Business expectations for time to restore each data package
Technology Management Analys, Confidential, July 2004 - Dec 2004
Joined as internee for first two months on a management project which included Analysis of business, science and engineering processes and standards, Project planning and coordination, Information system analysis, Business and technology analysis and research, Writing reports and instructional manuals for operations and users. Later I got trained in Networker and worked as an administrator for their daily maintenance. It was a short term project.
QUALIFICATIONS
Bachelor of Engineering
MBA (Technology Management)