Tuesday, December 30, 2008

Reader Poll: Where do you put your Domino transaction log files on i5/OS?

When running Domino on i5/OS and using transaction logging, do you put the transaction log files in a folder under the Domino data folder or in a separate folder on the IFS?

There is a bit of a debate going on between us and IBM on this. All of our i5/OS Domino servers have their transaction log files located in a separate subdirectory on the IFS. Such as /TRANSLOG/Server/ versus using /Domino/Server/Data/logdir. Domino doesn't care where they are as long as you point to them. It looks like BRMS does care where they are. If they aren't located under the Data folder then they don't get treated as Domino transaction log files.

We have been running with them outside the data directory ever since we have had the i5/OS systems with no issues. However we are now getting corrupt transaction log files when BRMS runs every since the power issue discussed in my previous post. I also can't find it documented anywhere that they must reside under the data directory on i5.

We use circular logging versus archive so BRMS doesn't need to touch the log files but it still does.

Thanks for the input.

Monday, December 29, 2008

Don't pull that plug

It's never a good thing when any type of server suddenly looses power. I have been told it can be very bad when it happens to an i5/OS box. Up until recently we have been lucky. Our collocation has a bad track record of power outages which takes down one of our biggest i5/OS servers. In the past it has come back up with no issues but the most recent one caused some headaches.

On the 23rd the power went out again. When the Domino partitions came back up we started to see some corruption issues. Since these servers are members of a cluster we just recreated the replicas that were corrupt. Minor pain but no biggie. Then our system engineer started noticing issues with BRMS. Jobs were still showing as active that were not actually running. This started showing up on the Domino servers that were supposedly being backed up by BRMS with the phantom job. One Domino server in particular started crashing and having strange behavior. When I tried to do a manual shutdown and restart the server took a while to come down and then would not come back up.

IBM recommeded an IPL to clear out the phantom BRMS job so we ended all the Domino servers and IPL'd the system. This cleared the BRMS job but the one Domino server still would not start. It eneded up being 2 issues with this server.

1) The Directory= entry in the NOTES.INI was messed up. It had a bunch of symbol charachters after the = sign. We have seen this on occasion with Domino on iSeries. The solution is to put the directory path info back in.

2) The transaction log files were corrupt. The Domino server would stall at the point where it was trying to read the log files when starting. I renamed the current trans log folder and created a new empty one. When Domino restarted it created new log files and ran a consistency check against all dbs. Note: it actually took 2 restarts to get the log files built again. The first start ended with a panic telling you to restart the server again to finish the process. Thanks for the info.

This power outage wasn't as smooth as the previous ones but at least Domino was able to repair itself without a full restore. It also helps that these are clusted servers so any data that was in the corrupt trans log files will replicate back from the other server.

Now to find a new collocation provider. The funny thing is they were upgrading their UPS's when this latest outage happened.