Smart Database Replication

We recently received a letter from one of our long standing customers, Jill Salo, Oracle DBA at Westpeak Global Advisors about her experience using Dbvisit due to a real life database failure.

Hi Arjen,
As I mentioned a few weeks ago, we lost our production database and had to failover to our standby using Dbvisit.
I thought you would appreciate some technical details. Feel free to share our experience if you feel it would be helpful to others.

OS version: Linux OEL5
Oracle version: 10.2.0.4 Standard Edition.
Storage: Lefthand Networks (now HP) iscsi SAN, mounted as ext3
Cause of crash: I suspected our mounted volume had a corruption as we were seeing a significant degradation in I/O performance. After many database checks, we decided to run Iometer to check if the disk speeds were below par.

I ran Iometer on the volume that held the database files.

We think Iometer may have hit some corruption because this caused us to lose the whole file partition. Even after rebooting, the system could not find that partition to mount. It had simply disappeared. The SAN partition was still there and available.

Using Dbvisit, we were able to switch over to the standby database in a few minutes. The switch was simple and with no errors. The only problems we encountered after that were network issues on our side.
1. We had tested with one subnet (subnet2) but as an oversight, computers on subnet3 could not see the database. This was rectified by some network configuration changes (by the network admin).
2. The firewall had the default timeout setting of 2 hours. We have some long running jobs that process mostly in Java with occasional Oracle calls. These jobs would fail with the “Connection reset by peer” error message. We set this timeout to the max of 12 hours which seemed to fix the problem.

When we first installed Dbvisit, it did not include the option to create a standby using the Dbvisit software. In a crisis situation, it was a very helpful option and once again it worked without errors. Using the menu I was able to recreate the standby by navigating a few menus. The final switch back to our production server worked seamlessly as well. It did take a couple of hours due to our large and numerous redo logs (write intensive database) but there were no problems.

Thanks again for a superb product. It keeps getting better and has once again saved the day!

Sincerely,

Jill Salo
Oracle DBA
Westpeak Global Advisors

While our development team is hard at work putting the final touches on the NEW Dbvisit Standby web-based GUI, set for release at the end of March, we are pleased to be able to offer you a sneak preview of it here.

We are really excited about this evolution in the Dbvisit Standby product. What this means is that in addition to the command line access already offered, the new version wraps the application’s powerful functionality with a clean and elegant web interface – enabling commands for both the primary and standby databases to be run from the one place. No complex configuration is required as this new release comes with its own web server built in.

Managing your Oracle standby database systems just got a whole lot simpler…

If you have questions or comments then let us know. We’d love to receive your feedback.
Enjoy.

Avisit Solutions Appoints Eric van der Sluis to Lead Sales Growth Worldwide
Rapidly growing NZ Software Company sets stage for growth worldwide

Date: 08 February 2011
For Immediate Release

Avisit Solutions Ltd, creators of Dbvisit the alternative standby database and disaster recovery solution for Oracle® Database Standard Edition & Standard Edition One sites worldwide, today announced the appointment of Eric van der Sluis to drive growth of sales worldwide.

“Eric is a much needed addition to the company and brings extensive international experience as a senior sales executive in the software and IT infrastructure industries,” says Arjen Visser, CTO and founder of Avisit Solutions. “His proven background in developing new markets offshore and track record in building strong relationships with partners makes Eric the ideal person to drive sales in the 54 countries we are currently present in and beyond.”

“We have aggressive growth plans for the business and his international sales experience is a critical part of the equation,” expands Arjen. “Our talented and innovative team all have international experience and this is the key to our continued success.”

Eric has a strong track record in sales growth and brings over 20 years of experience in the information technology space with more than a decade focused specifically in the Software solutions market. He has held senior positions at Orion Health over the past 8 years developing new markets in New Zealand, Australia, Singapore, Japan and Malaysia and was involved in the start up of a Sun /Oracle practice for NSC, Avaya’s largest partner across Australia and New Zealand, in the Northern region of New Zealand.. Eric holds a Bachelor of Science from Auckland University.

(Ends)

For further information please contact:

Libby Russell
Arjen Visser
Tel: +64 (0)9 950 3301
www.dbvisit.com

About Avisit Solutions Ltd

Avisit Solutions enables organisations with Oracle® Database Standard Edition and Standard Edition One software to inexpensively protect their critical data infrastructure. By eliminating the need to upgrade to Oracle® Enterprise Edition to access Data Guard our customers can save up to 85% on their disaster recovery solution.

Designed by Database Administrators (DBAs) for DBAs, our products are designed to be ‘set and forget’. They are simple to install and manage, functionally rich and come with comprehensive documentation and support. We provide peace of mind to DBAs and free up their valuable time to focus on other activities.

Avisit Solutions founded in 2000 and is based in Auckland, New Zealand with customers in more than 50 countries around the world.

www.dbvisit.com

Dbvisit version Dbvisit 5.3.08 has been released. The complete list of new features and fixes are below.

New features:

1. Send an email to ADMINS on successful completion of CSD (with trace) (Creation of Standby Database).
2. Drop tempfiles on standby on completion of graceful switchover for oracle versions 11.2+. As a workaround for Oracle bug: 10292173: “This bug causes rman backups to fail if the tempfiles on the standby are more recent than on the primary”.

Fixes:

1. Use db_unique_name instead of ORACLE_SID to access subdirectories under flash recovery area.
2. Allow for different syntax of the asmcmd command to obtain asmcmd version. Previously caused Dbvisit call to asmcmd to hang.
3. Improved logging for a remote execution of starting the standby database from the primary server.
4. Make sure datafiles are given unique names when copied to/from temp location during CSD (Creation of Standby Database). Previously datafiles with different paths but same basenames overwrote each other.
5. Ensure archive log gap report and heartbeat messages are sent to SUCCESS_MAILTO address if SUCCESS_MAILTO is set, not to ADMINS.
6. Improve the graceful switchover option for RAC. Previously the log gap would not always be 0 to initiate the switchover. This is because the ALTER SYSTEM ARCHIVE LOG was done on the database level and not on the instance. Setting INSTANCE_CLAUSE_FOR_ARCHIVE_LOG_CURRENT = Y will add the optional INSTANCE clause to the ALTER SYSTEM ARCHIVE LOG command.
7. Fix issue in AMM where it did not delete archives when number of archives reached the number specified by NUM_ARCHSOURCE_TO_KEEP.
8. Fix issue in AMM where under some circumstances it deleted newer archives first.

Dbvisit can be downloaded from http://www.dbvisit.com/dbvisit_download.php

A new year and an over-zealous manager gets perhaps a little ambitious with his sweeping changes. Irrespective of this, Dbman still manages to help this new gang of DBAs hit their targets for 2011. Find out more here.

Hopefully you have had some semblance of a relaxing holiday, however fleeting, this Christmas/New Year period. Aside from overeating, having awkward conversations with distant relatives and feigning glee over presents that missed the mark, this time of the year can also present a great opportunity to step back and reevaluate a multitude of areas in your work life.  Here are some suggestions of areas which may benefit from your attention:

1. Look at whether there are any procedures or tasks that could be scripted or automated, especially those manual and repetitive tasks. This only requires a small investment of your time when compared to the savings you’ll make over the course of the year. Perl is an ideal language in this instance.

2. Evaluate how your databases are performing, as this is often one of the things that may be overlooked or cast aside in the course of normal operations. In particular examine whether there is a need to tune them, and explore which tools will be most useful in helping you achieve this.
3. Look around for tools that will help you do your job better. Do you need to speak to your manager to see if they would approve a purchase? Be prepared to harness that post holiday optimism and generosity that he/she is inevitably basking in on their return to the office.
4. Set goals for personal development. Are there certifications that you could aim for in the next year – or books/courses to help you up-skill? We recommend The Oracle Concept 11.2 Guide as a great way to learn about the new features of Oracle.
5. Evaluate your backup/DR strategy. Is it robust? When was the last time you tested it? DR testing should be carried out at least annually, and involve an actual fail over from your production database to the standby.

You may also find the following resources useful:

12 Ways To Think “Big Picture” and Why You Should Bother

Disaster Recovery Testing Best Practises: Test Thoroughly and Often

Dbvisit Versus Automated Oracle Log Shipping

Achieve enterprise class database redundancy, service levels and performance with Cintra’s SuperCharged Oracle Standard Edition solutions.

Wednesday, Jan 19th, 2011
10am EST, 3pm GMT (1 Hour)
Presenter: Abdul Sheikh, CTO

Please join Cintra, a Dbvisit partner, for a live Webinar exploring the significant benefits of SuperCharged Oracle Standard Edition solutions.

While Oracle Standard Edition and Standard Edition One are both powerful and cost effective database solutions, they lack many of the high availability and performance management features and options available with Oracle Enterprise Edition database. More…

Dbvisit version Dbvisit 5.3.06 has been released. The complete list of new features and fixes are below.
New features:

1. Change the name pattern for dbvisit trace files to use DDC rather than ORACLE_SID.
2. Check if primary database is in hot backup mode before starting CSD (Create standby database).
3. Support function to collect and mail standby database trace files.
   Usage: dbv_functions -a DDC get_stdby_traces primary_trace_name:[offset]

4. Improve tracing information for dbv_functions. Now displays the switch in the trace file.
5. Added a global USE_LONG_SERVER_NAME to allow the use of long server names.
   Example: server names db1.atlas.domain.com and db1.zeus.domain.com can now be used. Previously Dbvisit would only use the shortname which would be db1 in both cases.

6. Dbvisit heartbeat file dbv_heartbeat_DDC.lck introduced to improve the logging and tracing of the heartbeat functionality.
7. Set standby_file_management=auto on standby database creation completion.
8. Added global ARCHSOURCE_BACKUP_COUNT for the AMM module. When this is set > 0, Dbvisit will only remove the archive log file on the primary server if it has been backed up by RMAN. The count refers to the number of times it has to be backed up by RMAN before being removed by Dbvisit.
9. Changed default value for JOBHIST and JOBHIST_DR to dbvisit_DDC.hist.
10. Added setting to ignore certain ORA error codes if required.
11. Improvements to the way Oracle databases are started and stopped. A further check is now made after stopping to ensure the correct status of the database.
12. More Oracle database dictionary information is listed in the trace file for the standby database.

Fixes:

1. _INIT_ORA_REMOVE was defined as a list of two strings instead of one – fixed.
2. Dbvisit shuts down primary database using “shutdown normal” – changed to “shutdown immediate”.
3. Globals $SEQ_LENGTH_FORCE and $THREAD_LENGTH_FORCE introduced to overwrite $SEQ_LENGTH and $THREAD_LENGTH values. SEQ_LENGTH is reset to 5 only for oracle version < 11.2 (windows).
4. Removed \ at the end of a certain paths to improve verification.
5. Improved ability to deal with ASM filenames.
6. Replaced checksum verification for datafiles with file size verification for creating standby db when files are copied directly to the standby server.
7. Replaced exit code 768 with 770 to fix child exit code issue.
8. Provide fix for ORA-10878: parallel recovery slave died unexpectedly for Oracle 11 by adding the clause “noparallel”.
9. Bug fixed for Oracle RAC when RAC has more than 2 nodes.
10. Ignore Oracle log_archive which are set to REMOTE destinations.
11. Do not allow the user to change the value of parameter db_name (in CSD).
12. Remove PFILE global from documentation, this is no longer needed.

Dbvisit can be downloaded from www.dbvisit.com/dbvisit_download.php

Ever felt torn between a loved one and your work?

We’ve all felt like this at some point; luckily Dbman is here to help. View his latest escapade here. Plus, email us at dbman@dbvisit.com and tell us about a time when work got in the way and you’ll go into the draw to have flowers sent to your neglected loved one, whether it’s your spouse, mother, housekeeper, or even yourself.

Terms and conditions apply.

We recently carried out some research among some of our clients in order to gain a better insight into the acceptable levels of data loss in a standby environment. We asked two questions:

1. How often is Dbvisit scheduled on your primary server. For example is this every 1, 2, 5, 10, 15 or 20 minutes?
2. What type of database is Dbvisit running on? I.e. OLTP, Data Warehouse  or other?

We got a great response and would like to thank everyone who participated.

The responses (see chart below) showed that Dbvisit is scheduled on average every 12 minutes with the shortest interval reported as 2 minutes

Dbvisit is actually capable of providing a minimum of just 60 seconds data loss although this needs to be considered against how critical the database in question is and the possible subsequent impact on database performance. We would love to hear from any of our customers who schedule Dbvisit every 60 seconds or who have any queries surrounding Dbvisit scheduling, so we better understand your data availability needs.

We also found that OLTP databases are by far the most common, outnumbering all others by 2:1.