Monthly Archives: August 2007

Are Daily Backups Really Sufficent?

Database, Hardware, Work

Monday afternoon we had a critical failure of an Oracle database at work. Within a few minutes of the fault taking place, I started seeing block corruption errors whilst I was reviewing some information in the production environment. At this stage, I was thinking that we might have dropped a disk in the SAN but referred it onto our database administrator to rectify it.

As is quite common, our environment consists of multiple Oracle 10g RAC nodes connected into a shared data source. The shared data source in this instance is a SAN, where we have a whole bunch of disks configured in groups for redundancy and performance. As soon as the database administrator became involved, it became apparent that we didn’t drop a single disk but had in fact lost access to an entire group of disks within the SAN.

Due to the manner in which the SAN and Oracle are configured, we were not in a position where running in a RAID environment was going to help. If we had dropped a single disk or a subset of disks from any group within the SAN, everything would have been fine; unfortunately we dropped an entire disk group. The end result of this was that we were forced to roll back our database to the previous nights backup.

The following days have been spent recovering the lost days data through various checks and balances; but it takes a lot of time and energy from everyone involved to make this happen. We’ve been fortunate enough to trade for several years without ever needing to roll back our production database due to some sort of significant event; which I suppose we should be thankful for.

After three years without performing a production disaster recovery, had we become complacent about data restoration and recovery as haven’t really needed it before? I believe that since we haven’t had a requirement to perform a disaster recovery for some three years, that our previous data recovery guidelines have now become out of date. Whilst a daily backup may have been more than sufficient for this particular database two or three years ago, the business has undergone significant growth since that time. The daily changeset for this database is now significant enough that, whilst having a daily backup is critical – it requires significant amounts of work to recover all of the data in a moderate time frame.

As a direct result of this disaster, we’re going to be reviewing our data recovery policies shortly. The outcome of that discussion will most likely be that we require higher levels of redundancy in our environment to reduce the impact of a failure. Whilst it would be ideal to have an entire copy of our production hardware, it probably isn’t going to be a cost effective solution. I’m open to suggestions about what sort of data recovery we implement, however I think that having some sort of independent warm spare may win out.

What have we learned out of this whole event:

  • daily backup of data is mandatory
  • daily backup of data may not be sufficient
  • verify that your backup sets are valid, invalid backup data isn’t worth the media it is stored on
  • be vigilant about keeping data recovery strategies in step with business growth and expectations

Maybe periodic disasters are actually healthy for a business? Whilst every business strives to avoid any sort of down time, I expect that as a direct result of the typically high availability of certain systems that disaster recovery isn’t put through its paces often or rigorously enough; which may result in longer downtimes or complete loss of data when an actual disaster recovery is required.

Helpful Domain Utilities

Internet, Websites

It used to be quite cumbersome and slow to perform domain research, then the domain registers became a whole lot smarter by providing you with a list of alternative ‘suggested’ domains. While the suggestions that the domain registers were suggesting were at times useful, more often than not they were utterly useless and weren’t worth the money to purchase.

Some time ago I stumbled across Domains Bot, which is an AJAX enabled domain search utility. You enter the name of the domain that you’d like to use and it comes back with an AJAX dropdown list with information on whether the domain is available or if it is for sale on one of the big domain clearing houses.

Yesterday, I followed a link into Bust A Name, which is a similar service with a twist. This time around you enter in a bunch of key words or phrases that you’d like in the domain and it goes away and tries to find all of the domains within your suggested list that are available; drastically cutting down research time.

Next time you’re looking to buy a domain, give one of these utilities a go; I’ve found them very useful.

Winning The Fight Against Spam

Blogging, Software

I hate spam, I hate it in email, I hate it in paper mail, I hate it in instant messaging, I hate it in forums – I just hate it.

For a long time, I have struggled with comment spam on this site up until I installed Akismet, which has been my knight in shining armour to help combat comment spam.

Over the last year, the rate that this site has been getting spammed has increased every single month, without fail – relentlessly. The spam statistics for the Akismet service indicates that the rate they are receiving and filtering spam has also been increasing.

At the moment, the graph doesn’t show any particular sign that the rate of spam has slowed at all. For some reason though, the rate that this site is receiving spam has definitely decreased. Over the last few months, I would delete between 350 and some times over 1000 per day while at the moment it is averaging a number less than 200.

That got me wondering what has changed around these parts, nothing in particular. I’m still running the same blogging software, still using Akismet to filter spam on the site and the search engine rankings of the site haven’t decreased such that it might make it a less likely target for spammers.

Is it possible that enough people are combating spam efficiently these days that at least some portion of the spamming community have called it quits? I don’t know if there is a way for anyone to answer that question with any sort of certainty however I can only hope that it might be the case.

Die filthy spammers, die.

Windows Live ID Web Authentication

Internet, Programming

Today, Microsoft released the Windows Live ID Web Authentication service to the world for public consumption.

The Windows Live ID service is an evolution of the familiar Microsoft Passport system. Over the last few years, Microsoft have been extending the Microsoft Passport system to cater for the constant change and evolution of authentication requirements on the internet.

With the first public release of the Live ID web authentication service, Microsoft have updated all of their documentation and are also providing an SDK for the Live ID services. For the convenience of the developers around the world and to increase the adoption rate, Microsoft have graciously provided implementations for the Live ID web authentication product in .NET, Java, Perl, PHP, Python and Ruby.

Everything I’ve read so far on it makes me think that the new web authentication service from Microsoft might gain some traction; however I think there is a sore point that makes the service less desirable. From a user experience point of view, users expect that when they login to a web site that they are not redirected away from it. The Windows Live ID web authentication service requires that you place an iframe on your site which the actual authentication takes place in; less than perfect. I personally think that it would have been a much nicer product release if they allowed users to authenticate against it using pure web service or remote calls; no iframe and dodgy browser redirection. Unfortunately, I however sympathise with the pressures that Microsoft are under to guarantee that their users privacy is protected and allowing pure web service authentication does compromise that point some what.

If you’re interested: