Thursday, November 5, 2009

Removing an INI entry from a Desktop Settings document doesn't remove it

I ran into an issue recently with the Custom Settings - Notes.ini feature of the Desktop Settings. If you add an INI entry and then later remove it, it is removed from the display field however the Notes clients still apply it. I tracked this down and the underlying fields in the settings document are not updated accordingly. The Notes client still uses these fields to apply the setting.


I had the INI parameter for PromptForLocation set to 0 in our policy. I wanted to remove it because one of our users wanted this enabled. Even after removing the entry from the policy the feature would continue to be disabled. I noticed the $PrefPromptForLocation field was still in the policy settings document and the AlwaysSetItems field still listed $PrefPromptForLocation. This was after I had removed the entry in the form. I created an agent to remove the entry from the AlwaysSetItems field but the Notes client continued to disable it. I ended up creating a brand new Desktop Settings document and applying it to the Policy to get rid of that setting.

It also looks like INI parameters are Enforced whether you check it or not. The INI parameter above wasn't enforced in the settings document but the Notes client continued to reset the value after the user changed it. All INI parameters are added to the AlwaysSetItems field.

Thursday, October 1, 2009

Integrating Notes account managment into an enterprise wide solution

Our IT department is in the early stages of implementing an enterprise wide user account management appliance. One of my tasks was to come up with a way for this appliance to work with Notes. The appliance we are using is IntApp Integration Builder.


On the Notes side we currently use GSX ID Manager to manage Notes accounts and groups. Today admins manually create new user requests via a Notes form and ID Manager processes the request in the background. Since ID Manager is a standard Notes application and the design is not hidden, it is easy to integrate with it.

At first we tried using the NotesSQL connector to connect to the ID Manager application in Notes. This worked very well however it requires a Notes client with the NotesSQL connector installed to act as a gateway between the appliance and Notes. This is a potential breakdown point that we would rather avoid.

The ultimate solution is Web Services. The IntApp appliance integrates with most other applications using web services so why not Notes as well. Since we have full access to the ID Manager database design we can custom code a web service provider and use Lotus Script to create the request for ID Manager to process. This is the first time I have delved into web services but found it fairly easy to setup if you already know Lotus Script or Java.

The one issue I ran into was with web authentication. When a web service is contacted the Domino server presents the standard login web page. I haven't figured out how to make the appliance pass login info to this form in order to have access to the service. If anyone knows how to provide authentication to a Domino web service let me know.

Tuesday, September 29, 2009

Domino on IBM i - Omit /TMP from backups

It has been a while since my last post. I have been deep in ND 8.5 upgrade mode and working on some issues.

If you run Domino on IBM i (i5/OS) you need to omit the /TMP folder from online backups NOW! If you backup the IFS while Domino is running you need to do this or will potentially run into a major database corruption issue. IBM is currently drafting an SPR for this and will add /TMP to the list of items to omit which also include *.id and notes.ini files.

Here is what can potentially happen. There are hourly jobs (tasks) that Domino runs such as Chronos and in our case SMDUPD (ScanMail Update). When these jobs run they look to the /TMP folder to see if Domino is already running. If BRMS has that folder locked the job will think Domino isn’t running and will start a second LOGASIO job. This is the job that controls transaction logging. Running 2 LOGASIO jobs against the same transaction logs is bad and will end up corrupting databases. Most likely the ones that had transactions waiting to be written. The second LOGASIO job tries to perform a recovery since it thinks Domino ended abnormally. So you have one normal running LOGASIO job and another comes along and tries to replay logs.

Running a fixup -F -J on the database will fix the database however you don’t know which ones are corrupt until some action is performed on the corrupt area of the database. The action could be the router delivering a message with an attachment or designer running on the database or some other action. When this issue hit us it would start around 10pm (when backups were running) and we would be up all night and into the next day fixing databases as the server reported them. Auto fixup wouldn’t always catch them.

IBM is also working on changing the behavior so a second LOGASIO job can’t start.

You can see this happening in the OS history job logs. You will see CHRONOS start on an active Domino server and then immediately after a LOGASIO job starts. You may also notice 2 Domino console logs that span the same time range if you have console logging enabled.

Thursday, April 23, 2009

Interesting error dialog of the day

Saw this dialog on one of our BES Domino servers yesterday....



I suspect this dialog was from BES since Domino doesn't normally throw up Windows dialogs. The Domino console was in the background with a PANIC message.

Wednesday, March 25, 2009

Production mail servers running Domino 8.5

All of our production mail servers are now running Domino 8.5. We ran into some minor issues with database corruption but it seems that newer versions of Domino always pick up corruption that the previous version didn't. We also have very large mail files that have been touched heavily by enterprise archiving lately so I'm sure this exposed some issues. Sometimes the server would catch it and automatically do a consistency check but some we had to manually run a fixup.

We also ran into the SPR# TLAM7NGJXY memory leak issue on some servers as stated in a previous post. We now have HF55 (Windows) / L502113 (i5/OS) running on all mail servers and haven't seen the issue since. We have IF1 running on our administration server for the group update issue. We haven't seen the memory leak issue on that server so no need for a special hot fix on that one.

Next is to upgrade to the 8.5 ODS and enable design and data compression. This will probably happen over Memorial Day weekend when I have a long weekend to run compacts. I'll most likely do our backup cluster servers before then and do the primary servers over the long weekend.

We are holding off on DAOS for now. I expect we will do something with it in the second half of the year.

Thursday, March 12, 2009

Domino 8.5 rollout has begun

We started deploying Domino 8.5 to our production mail servers. The upgrade itself has been easy as with previous releases. We run most of our mail servers on IBM i platform so it's a single command to update a Domino partition to 8.5 code. I also run a post upgrade script outlined below. I haven't done any INI tweeking or ODS upgrades yet just laying down the code.

We have 10 production mail servers running 8.5 so far and so far so good except for 2 of them. One is related to a know memory leak, SPR# TLAM7NGJXY, which there is a fix for. We have the fix but haven't deployed it yet. This issue only happened once on one server so far.

The other we saw just this week. The server in question had been upgraded for a week and a half with no issues and then all of the sudden most of the mail files were marked as corrupt. The corruption started right after the UpdAll task started one morning at 2 am. The predominant corruption was Bitmap checksum incorrect. A simple fixup fixed the database and we ran it against a database as the server alerts came in. If I would have know the large number of databases impacted I would have just ran fixup against all mail files on the server when it initially started. We currently have a PMR open on this and IBM is investigating.

One thing I noticed in the PMR updates is there is a known issue with DAOS and Trend ScanMail. I don't know the specifics but a fix is in the works so just be advised if you use the two together. We use ScanMail but haven't turned DAOS on yet except on one proof of concept server. We haven't seen an issue on that server.

Here is the post upgrade script I use on our IBM i Domino servers. This is an IBM i QSH shell script but is easily adpated to Windows. Note I delete the pernames.ntf. This is because we have personal address books replicated to the servers and I don't want them getting the 8.5 design just yet.

domdir='put server data directory here'
domserver='put server name here'

echo "Make sure the Domino server has ENDED before continuing."
echo "Enter 1 to continue or any other key to exit."
read proceed
if test "${proceed}" != "1"; then
echo "Exiting.";
exit
fi

cd $domdir
echo "Making backup of files..."
mv $domdir/log.nsf $domdir/backup/log.R85
mv $domdir/mail1.box $domdir/backup/mail1.R85
mv $domdir/mail2.box $domdir/backup/mail2.R85
rm $domdir/pernames.ntf

echo "Submitting batch commands..."
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('names.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/FIXUP) PARM('admin4.nsf' '-f' '-j' '-v' '-l')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/COMPACT) PARM('names.nsf' '-c')) BATCH(*YES) ALWMLTTHD(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/COMPACT) PARM('admin4.nsf' '-c')) BATCH(*YES) ALWMLTTHD(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('names.nsf' '-RX')) BATCH(*YES)"
system -v "RUNDOMCMD SERVER($domserver) CMD(CALL PGM(QNOTES/UPDALL) PARM('admin4.nsf' '-RX')) BATCH(*YES)"
echo "Jobs have been submitted to the QBATCH job queue. Monitor the queue for job completion before starting the server."


Sunday, March 1, 2009

Flight Log: March flight to the farm

I have been wanting to fly over my grandparents farm for some time now. I had a plane reserved for today and it looked like the winter weather would actually cooperate. If you call 20 degrees and North winds gusting to 23 knots cooperating. I debated about the winds but they were right down the runway and I needed some more high wind practice.

I originally had the 172SP reserved for today but it ended up going offline for maintenance. The other 172 was available, an R model, so I reserved it. This is the first time I have flown our club’s 172R. I normally fly the SP but I have flown an R model at other places. The R model has 20 fewer horsepower but in this cold weather it isn’t that noticeable.

I planned on flying over my grandparent’s farm West of Pontiac and then go over to Kankakee airport. Well there was lake effect snow coming down along the Illinois - Indiana border making KIKK MVFR. I flew to KVYS instead after the farm. They have a nice long wide runway into the wind to practice some gusty landings. The winds were anywhere from 10 to 40 degrees gusting to 21 knots. It was a little rocky on the approach but I stuck the landings. Landing back at LL10 with its 30 ft wide runway was interesting.

My daughter went with for the ride and took some pictures. She has been learning about weather in school so she was taking pictures of the clouds. Here are some other pictures she took.