Interesting error dialog of the day

Saw this dialog on one of our BES Domino servers yesterday....



I suspect this dialog was from BES since Domino doesn't normally throw up Windows dialogs. The Domino console was in the background with a PANIC message.

Production mail servers running Domino 8.5

All of our production mail servers are now running Domino 8.5. We ran into some minor issues with database corruption but it seems that newer versions of Domino always pick up corruption that the previous version didn't. We also have very large mail files that have been touched heavily by enterprise archiving lately so I'm sure this exposed some issues. Sometimes the server would catch it and automatically do a consistency check but some we had to manually run a fixup.

We also ran into the SPR# TLAM7NGJXY memory leak issue on some servers as stated in a previous post. We now have HF55 (Windows) / L502113 (i5/OS) running on all mail servers and haven't seen the issue since. We have IF1 running on our administration server for the group update issue. We haven't seen the memory leak issue on that server so no need for a special hot fix on that one.

Next is to upgrade to the 8.5 ODS and enable design and data compression. This will probably happen over Memorial Day weekend when I have a long weekend to run compacts. I'll most likely do our backup cluster servers before then and do the primary servers over the long weekend.

We are holding off on DAOS for now. I expect we will do something with it in the second half of the year.

Domino 8.5 rollout has begun

We started deploying Domino 8.5 to our production mail servers. The upgrade itself has been easy as with previous releases. We run most of our mail servers on IBM i platform so it's a single command to update a Domino partition to 8.5 code. I also run a post upgrade script outlined below. I haven't done any INI tweeking or ODS upgrades yet just laying down the code.

We have 10 production mail servers running 8.5 so far and so far so good except for 2 of them. One is related to a know memory leak, SPR# TLAM7NGJXY, which there is a fix for. We have the fix but haven't deployed it yet. This issue only happened once on one server so far.

The other we saw just this week. The server in question had been upgraded for a week and a half with no issues and then all of the sudden most of the mail files were marked as corrupt. The corruption started right after the UpdAll task started one morning at 2 am. The predominant corruption was Bitmap checksum incorrect. A simple fixup fixed the database and we ran it against a database as the server alerts came in. If I would have know the large number of databases impacted I would have just ran fixup against all mail files on the server when it initially started. We currently have a PMR open on this and IBM is investigating.

One thing I noticed in the PMR updates is there is a known issue with DAOS and Trend ScanMail. I don't know the specifics but a fix is in the works so just be advised if you use the two together. We use ScanMail but haven't turned DAOS on yet except on one proof of concept server. We haven't seen an issue on that server.

Here is the post upgrade script I use on our IBM i Domino servers. This is an IBM i QSH shell script but is easily adpated to Windows. Note I delete the pernames.ntf. This is because we have personal address books replicated to the servers and I don't want them getting the 8.5 design just yet.

domdir='put server data directory here'
domserver='put server name here'

echo "Make sure the Domino server has ENDED before continuing."
echo "Enter 1 to continue or any other key to exit."
read proceed
if test "${proceed}" != "1"; then
echo "Exiting.";
exit
fi

cd $domdir
echo "Making backup of files..."
mv $domdir/log.nsf $domdir/backup/log.R85
mv $domdir/mail1.box $domdir/backup/mail1.R85
mv $domdir/mail2.box $domdir/backup/mail2.R85
rm $domdir/pernames.ntf

echo "Submitting batch commands..."
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('names.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('admin4.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/COMPACT) PARM('names.nsf' '-c')) BATCH(*YES) ALWMLTTHD(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/COMPACT) PARM('admin4.nsf' '-c')) BATCH(*YES) ALWMLTTHD(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('names.nsf' '-RX')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('admin4.nsf' '-RX')) BATCH(*YES)"
echo "Jobs have been submitted to the QBATCH job queue. Monitor the queue for job completion before starting the server."

Flight Log: March flight to the farm

I have been wanting to fly over my grandparents farm for some time now. I had a plane reserved for today and it looked like the winter weather would actually cooperate. If you call 20 degrees and North winds gusting to 23 knots cooperating. I debated about the winds but they were right down the runway and I needed some more high wind practice.

I originally had the 172SP reserved for today but it ended up going offline for maintenance. The other 172 was available, an R model, so I reserved it. This is the first time I have flown our club’s 172R. I normally fly the SP but I have flown an R model at other places. The R model has 20 fewer horsepower but in this cold weather it isn’t that noticeable.

I planned on flying over my grandparent’s farm West of Pontiac and then go over to Kankakee airport. Well there was lake effect snow coming down along the Illinois - Indiana border making KIKK MVFR. I flew to KVYS instead after the farm. They have a nice long wide runway into the wind to practice some gusty landings. The winds were anywhere from 10 to 40 degrees gusting to 21 knots. It was a little rocky on the approach but I stuck the landings. Landing back at LL10 with its 30 ft wide runway was interesting.

My daughter went with for the ride and took some pictures. She has been learning about weather in school so she was taking pictures of the clouds. Here are some other pictures she took.

Program Conversion for Domino 8.5 on IBM i V5R4

Be advised of the Program Conversion that will take place if you install Domino 8.5 on V5R4. This takes a long while to process. Info on this step is included in the checklist.

If you have multiple V5R4 systems to install it on you can install it on one, perform a SAVLICPGM of the 8.5 program and then restore it to the other systems. Since the first system did the conversion it wont be needed on the other ones.

The conversion doesn't happen if you are running V6R1.

Change in Compact task for Domino 8.5 on IBM i

I happened to be searching for something related to Domino on IBM i today and ran across this in the ND85 Beta 1 Release Notes...

IBM i: RUNDOMCMD CALL PGM(COMPACT) must run in *BATCH
Starting with Domino 8.5, the COMPACT program is multi-threaded and must be run in *BATCH mode.
Specify BATCH(*YES) ALWMLTTHD(*YES) on the RUNDOMCMD.
Running the COMPACT program can be used to upgrade databases to a new ODS level. For example, the following procedure can be used to upgrade a Domino 8.5 server's databases to ODS level 50.
1. Update notes.ini to contain: Create_R85_Databases=1
2. Run the following command with the server *ENDED to convert all existing server NSFs to ODS 50:
RUNDOMCMD SERVER() CMD(CALL PGM(QDOMINO850/COMPACT) PARM('-C'))
BATCH(*YES) ALWMLTTHD(*YES)

Now I can't find this information in any of the current online documents so I'm not sure if it still holds true. We currently use the Compact task when doing monthly maintenance on our Domino servers and they all use BATCH(*NO). I guess I will be trying BATCH(*YES) next time.

Update: I found where this is documented online here. If you try to run the Compact program with BATCH(*NO) it will return an 'Cannot create thread' error.

New DAOSEST posted - issue on i5/OS

Paul Mooney and Andy Pedisich have posted regarding the new DAOS Estimator tool (v1.1). This new version eliminates the MIME to CD conversion that was seen with the old tool and it also adds a save file for installing on System i. We had already called IBM about the install file for System i so I was glad to see they fixed it.

However, I just installed the tool on our System i and ran it against one of our test Domino servers. The Domino server immediately crashes. I tired it multiple times with different files and it crashes every time. This Domino server is running 7.0.3FP1 and i5/OS V5R4M0.

I tried the new version on Windows and Linux Domino servers and they worked fine.

Run the tool on a test i5/OS Domino server first just in case.

Update: As stated in the comments, you need to have Domino 8.x code installed on i5/OS in order for the tool to work. The Domino server can be running 7.x but the 8.x code must be available.