Thursday, March 12, 2009

Domino 8.5 rollout has begun

We started deploying Domino 8.5 to our production mail servers. The upgrade itself has been easy as with previous releases. We run most of our mail servers on IBM i platform so it's a single command to update a Domino partition to 8.5 code. I also run a post upgrade script outlined below. I haven't done any INI tweeking or ODS upgrades yet just laying down the code.

We have 10 production mail servers running 8.5 so far and so far so good except for 2 of them. One is related to a know memory leak, SPR# TLAM7NGJXY, which there is a fix for. We have the fix but haven't deployed it yet. This issue only happened once on one server so far.

The other we saw just this week. The server in question had been upgraded for a week and a half with no issues and then all of the sudden most of the mail files were marked as corrupt. The corruption started right after the UpdAll task started one morning at 2 am. The predominant corruption was Bitmap checksum incorrect. A simple fixup fixed the database and we ran it against a database as the server alerts came in. If I would have know the large number of databases impacted I would have just ran fixup against all mail files on the server when it initially started. We currently have a PMR open on this and IBM is investigating.

One thing I noticed in the PMR updates is there is a known issue with DAOS and Trend ScanMail. I don't know the specifics but a fix is in the works so just be advised if you use the two together. We use ScanMail but haven't turned DAOS on yet except on one proof of concept server. We haven't seen an issue on that server.

Here is the post upgrade script I use on our IBM i Domino servers. This is an IBM i QSH shell script but is easily adpated to Windows. Note I delete the pernames.ntf. This is because we have personal address books replicated to the servers and I don't want them getting the 8.5 design just yet.

domdir='put server data directory here'
domserver='put server name here'

echo "Make sure the Domino server has ENDED before continuing."
echo "Enter 1 to continue or any other key to exit."
read proceed
if test "${proceed}" != "1"; then
echo "Exiting.";

cd $domdir
echo "Making backup of files..."
mv $domdir/log.nsf $domdir/backup/log.R85
mv $domdir/ $domdir/backup/mail1.R85
mv $domdir/ $domdir/backup/mail2.R85
rm $domdir/pernames.ntf

echo "Submitting batch commands..."
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('names.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('admin4.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('names.nsf' '-RX')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('admin4.nsf' '-RX')) BATCH(*YES)"
echo "Jobs have been submitted to the QBATCH job queue. Monitor the queue for job completion before starting the server."


Ed Brill said...

Great to hear you are upgrading to 8.5!

John said...

at lotusphere in all conferenced they said "DAOS will be fully transparent to the third party product"... and now you can see that it does not work well with antivirus.

I've heard that it does not works OK with backup utilities... and that you need to disable the local antivirus (at least for the DAOS repository).

I think the only one that really understand DAOS exceptions is the one who coded it...

Sounds like shared email?

Oliver Regelmann said...

John, this rather sounds like early usage of unsupported configurations. You should always double check any software that comes in touch with Domino before upgrading. AFAIK Trend Micro has announced support for 8.5 for the next version coming midyear. And besides Tivoli Storage Manager I've yet to see a backup utility supporting DAOS/Domino 8.5 right now.

Everything CAN work in these constellations, but I wouldn't blame DAOS if it doesn't.

BTW: Isn't it a best practice to exclude NSFs from local file based antivirus? Same goes for DAOS files.

Jim Casale said...

@John Even if there is a backup utility that can backup Domino with DAOS objects in one job why would you want to do that? That defeats one of the advantages of using DAOS - you won't be backing up static data over and over again everyday. Incremental backups of the DAOS objects will cut down backup time and costs since you won't be backing up the same data everyday.