Wednesday, March 25, 2009

Production mail servers running Domino 8.5

All of our production mail servers are now running Domino 8.5. We ran into some minor issues with database corruption but it seems that newer versions of Domino always pick up corruption that the previous version didn't. We also have very large mail files that have been touched heavily by enterprise archiving lately so I'm sure this exposed some issues. Sometimes the server would catch it and automatically do a consistency check but some we had to manually run a fixup.

We also ran into the SPR# TLAM7NGJXY memory leak issue on some servers as stated in a previous post. We now have HF55 (Windows) / L502113 (i5/OS) running on all mail servers and haven't seen the issue since. We have IF1 running on our administration server for the group update issue. We haven't seen the memory leak issue on that server so no need for a special hot fix on that one.

Next is to upgrade to the 8.5 ODS and enable design and data compression. This will probably happen over Memorial Day weekend when I have a long weekend to run compacts. I'll most likely do our backup cluster servers before then and do the primary servers over the long weekend.

We are holding off on DAOS for now. I expect we will do something with it in the second half of the year.

Thursday, March 12, 2009

Domino 8.5 rollout has begun

We started deploying Domino 8.5 to our production mail servers. The upgrade itself has been easy as with previous releases. We run most of our mail servers on IBM i platform so it's a single command to update a Domino partition to 8.5 code. I also run a post upgrade script outlined below. I haven't done any INI tweeking or ODS upgrades yet just laying down the code.

We have 10 production mail servers running 8.5 so far and so far so good except for 2 of them. One is related to a know memory leak, SPR# TLAM7NGJXY, which there is a fix for. We have the fix but haven't deployed it yet. This issue only happened once on one server so far.

The other we saw just this week. The server in question had been upgraded for a week and a half with no issues and then all of the sudden most of the mail files were marked as corrupt. The corruption started right after the UpdAll task started one morning at 2 am. The predominant corruption was Bitmap checksum incorrect. A simple fixup fixed the database and we ran it against a database as the server alerts came in. If I would have know the large number of databases impacted I would have just ran fixup against all mail files on the server when it initially started. We currently have a PMR open on this and IBM is investigating.

One thing I noticed in the PMR updates is there is a known issue with DAOS and Trend ScanMail. I don't know the specifics but a fix is in the works so just be advised if you use the two together. We use ScanMail but haven't turned DAOS on yet except on one proof of concept server. We haven't seen an issue on that server.

Here is the post upgrade script I use on our IBM i Domino servers. This is an IBM i QSH shell script but is easily adpated to Windows. Note I delete the pernames.ntf. This is because we have personal address books replicated to the servers and I don't want them getting the 8.5 design just yet.

domdir='put server data directory here'
domserver='put server name here'

echo "Make sure the Domino server has ENDED before continuing."
echo "Enter 1 to continue or any other key to exit."
read proceed
if test "${proceed}" != "1"; then
echo "Exiting.";

cd $domdir
echo "Making backup of files..."
mv $domdir/log.nsf $domdir/backup/log.R85
mv $domdir/ $domdir/backup/mail1.R85
mv $domdir/ $domdir/backup/mail2.R85
rm $domdir/pernames.ntf

echo "Submitting batch commands..."
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('names.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('admin4.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('names.nsf' '-RX')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('admin4.nsf' '-RX')) BATCH(*YES)"
echo "Jobs have been submitted to the QBATCH job queue. Monitor the queue for job completion before starting the server."

Sunday, March 1, 2009

Flight Log: March flight to the farm

I have been wanting to fly over my grandparents farm for some time now. I had a plane reserved for today and it looked like the winter weather would actually cooperate. If you call 20 degrees and North winds gusting to 23 knots cooperating. I debated about the winds but they were right down the runway and I needed some more high wind practice.

I originally had the 172SP reserved for today but it ended up going offline for maintenance. The other 172 was available, an R model, so I reserved it. This is the first time I have flown our club’s 172R. I normally fly the SP but I have flown an R model at other places. The R model has 20 fewer horsepower but in this cold weather it isn’t that noticeable.

I planned on flying over my grandparent’s farm West of Pontiac and then go over to Kankakee airport. Well there was lake effect snow coming down along the Illinois - Indiana border making KIKK MVFR. I flew to KVYS instead after the farm. They have a nice long wide runway into the wind to practice some gusty landings. The winds were anywhere from 10 to 40 degrees gusting to 21 knots. It was a little rocky on the approach but I stuck the landings. Landing back at LL10 with its 30 ft wide runway was interesting.

My daughter went with for the ride and took some pictures. She has been learning about weather in school so she was taking pictures of the clouds. Here are some other pictures she took.