Thursday, December 27, 2007

Sametime Directory Decision

As I mentioned in my previous post I am in the process of designing a Sametime infrastructure for our environment. I think the biggest and toughest decision when designing Sametime is the directory decision. Based on Chris Miller's repeated statement to use LDAP as the access method this is an easy choice but the question is, which LDAP directory? We have 2 in our environment, AD and Domino, and they are not synchronized.

When deciding on a directory I had a few deciding factors; which password will be used, which groups will be accessible, and ease of directory administration going forward. We don't currently use the Internet Password field in Domino for anything. This means if we use the Domino directory we can populate this field with anything we want. At first I thought we should use AD so we can use the Windows password for authentication but the down sides to AD were; not able to easily manage Sametime information within AD, Sametime would be limited to only seeing AD groups and getting the infrastructure team to extend the LDAP schema. With the Domino directory most of the Sametime info for a user is already there and the e-mail groups would be the most logical groups for Sametime to use. However, there is the issue with what password to use with the Domino directory.

When using the Internet Password field in the Domino directory I have 3 options; enter a completely separate password in the field (last choice), use the built-in functionality in Domino to synch the Notes ID password with the Internet Password field or use a 3rd party product such as Tivoli Directory Integrator to synch the Windows password with the Internet Password field.

I'm currently leaning toward using the Domino directory with the native Notes ID password synch option since it doesn't require any additional cost or infrastructure.

Anyone using the native Notes ID and Internet password synch function in Domino? Is it reliable? How about other password synch options for Sametime use?

Thursday, December 13, 2007

On to the next project...

We are finally starting to plan for a Sametime implementation within our organization. It has been on the back burner for a while waiting on other projects and is now moving forward. I'm still in the research and planning stage but have an initial conceptual diagram done. Hopefully this post and future ones can help people who may be going through the same process in the future.

I have been attending the Sametime sessions given by Chris Miller and Carl Tyler at the ADMIN conference for a few years now as well as pouring over the manuals and the Sametime 7.5.1 BP redbook. This has provided a lot of help getting started but there are still some organization specific questions that need to be answered. Below and in future posts are some decisions we had to make up front. Some have been decided and some are still be thought over.

Overall Sametime network design
Our organization tends to be overly redundant so I set out planning the Sametime network with redundancy in mind. I think it will take off once we implement it and will eventually become a business (if not mission) critical application. We have multiple offices worldwide but have been centralizing most applications in the Chicagoland area with high speed WAN links to the offices.

The conceptual diagram I came up with has 2 Sametime servers clustered in the Chicago area and 2 Sametime servers clustered in London. I also have Multiplexers positioned in each local office because I like the idea of keeping the user connections local and having a single connection to the Sametime servers over the WAN. The Sametime servers will connect to load balancers for connections to multiple LDAP servers for directory lookups.

A single Samtime server can more than handle our entire user base (~3500 users) but as I mentioned earlier we are all about redundancy and response time. This setup still has a single point of failure with the Multiplexers but I thought load balancing multiple MUX's in each office would be a little over the top even for us.

This setup is only for chat (IM) services and not meetings. We will add separate meeting servers in the future for online meetings.

I'll have additional posts regarding directory and OS decisions shortly.

Wednesday, December 5, 2007

Upgraded to Notes 8 client

I went ahead and upgraded to the Notes 8 client a few days ago without upgrading my mail template or the server. I was pleased to discover the things Notes 8 brings to the table just by installing the client itself. Using Notes 8 with the R7 mail template you still get in-line spell checking, drop down type-ahead, and my favorite, the ability to shift-click multiple documents. As an administrator I can't tell you how many times I have longed for the ability to shift-click when selecting hundreds or thousands of documents. I'm sure there are other features there but these are the ones I notice most.

I did have to get used to looking for the term 'Application' versus 'Database'. I also long for the return of the right-click option 'Open in designer'.

The client has been stable except for the one crash I experienced when switching to a different Location/ID and then back to the original Location and ID. It does seem a little slower to load the standard client but once it's up it works well. Our R7 mail template has integration with Interwoven WorkSite which still works with the Notes 8 client.

As for our organization, we are waiting for 8.0.1 to seriously look at upgrading but I wanted to get a jump on the client features. Upgrading our mail template will be interesting as we have integration with multiple 3rd party products.

Tuesday, November 20, 2007

Strange Behavior in Domino Administrator 7.0.3

I have noticed an issue a few times now with the 7.0.3 Domino Administrator client when trying to pull up the Server Tasks. If the server is slow to respond due to a WAN link or high workload, the client will freeze and the nlnotes task uses 100% CPU. The only thing left to do is manually kill the tasks and restart Notes.

This doesn't happen with the 7.0.2 Domino Administrator when accessing the same server. I looked for a possible regression but didn't see any identified yet.

Monday, October 29, 2007

Adding ScanMail to a new Domino partition

As a follow up to my earlier post on Domino upgrade on System i - Copying Trend ScanMail I would like to show how to apply ScanMail to a new Domino partition. This assumes you have a system with at least one existing Domino partition setup and ScanMail for Domino 3.0 installed on that partition.

As stated in the earlier post if you just setup a new Domino partition and then try to run the ScanMail setup to apply it to the new partition, it will give an error if a SMD patch or service pack has been installed. The way around this is to do the following:

  1. Setup the new Domino partition as you would any new partition.
  2. You will then need to get the SMD databases and templates from an existing Domino server in the domain. This can be done one of two ways; do a file system copy of the SMD databases from an existing Domino partition to the new Domino partition (with both Domino partitions shutdown to be safe), or replicate the databases from an existing Domino partition keeping in mind replication is disabled by default on some of the databases.
  3. Add the following entries to the NOTES.INI file on the new partition:
  4. Add the following tasks to the SERVERTASKS line of the NOTES.INI:
    SMDemf,SMDreal,SMDsch,SMDmon (You will also need SMDdbs if you use replication scanning)
When you start the new Domino partition it will load the SMD tasks. You will then need to open the ScanMail Configuration Database in Notes and configure the settings for the new server as usual.

Disclaimer: I'm not responsible for any damage, malfunction, or lost data performing the above procedure may cause.

Monday, October 22, 2007

Comcast filtering Lotus Notes (Update 2)

We started getting reports from some of our users last week that large e-mail transfers via Lotus Notes were once again working on the Comcast network. I did some testing this weekend and every upload was successful.

So it would seem the filtering, at least for Notes in the Chicago area, has been lifted by Comcast. There is no telling if this is permanent since Comcast denies ever doing this in the first place but I will be watching it.

Original post
Updated post

Update 10/24/07:
Here are some links related to the issue being resolved
Associated Press article
Ed Brill bolg posting

Wednesday, October 17, 2007

Domino upgrade on System i - Copying Trend ScanMail

We run Domino on System i (iSeries/AS400) and use Trend ScanMail for virus scanning. When you install a new Domino point release the new Domino library doesn't contain the files needed by ScanMail. When you upgrade a Domino partition to this new release and it tries to load SMD it will fail because it can't find the program files.

The Trend solution for this is to run the ScanMail install and apply it to the new Domino code. However, if you have a service pack or patch applied to ScanMail the install program will error out saying there is a newer version installed than the one you are trying to install. The install program for the service pack is just an update script so you are forced to run the original install program. Trend's answer to this is to completely uninstall ScanMail and re-install from scratch. This is tedious on a system that has multiple Domino partitions running.

Below are the manual steps to add ScanMail to a new Domino library. These steps use the default Domino paths and uses 7.0.2 as the existing Domino code and 7.0.3 as the new Domino code where ScanMail is being copied to.

After installing the latest Domino code do the following (This can be done while the Domino partitions are up):

  1. From a 5250 session use the WRKLNK command and navigate to the /QIBM/ProdData/LOTUS/DOMINO702 folder.
  2. On the bottom command line enter CHGCURDIR ‘/QIBM/ProdData/LOTUS/DOMINO703’
  3. Locate the file SMD.INI and type a 3 next to it and press F4. In the Owner field enter *KEEP and press ENTER.
  4. Locate each of the following files, type a 3 next to each one individually, press F4. In the Symbolic link field enter *YES and in the Owner field enter *KEEP and press ENTER.

Now when you upgrade a Domino partition to the latest version it will be able to find the ScanMail programs. This process works for ScanMail for Domino 3.0 so future versions may change this process.

There may be a more automated way of doing this but my i5/OS knowledge is limited so this is what works for me.

Disclaimer: I'm not responsible for any damage, malfunction, or lost data performing the above procedure may cause.

Tuesday, October 16, 2007

Self-healing Domino

I was on the train yesterday going home when my Blackberry (BB) vibrates followed by the dreaded audible alarm. I have my BB setup to vibrate for most messages but whenever it receives an alert from our GSX Monitor it makes noise. I look at the alert and one of our Domino servers isn't responding. It used to be this would initiate frantic phone calls to find someone near a computer that would know how to look into the problem with the server and get it back online. Not in today's world.

This particular Domino server is running on one of our iSeries (System i) and I have Rove Mobile's (formerly Idokorro) Mobile SSH on my BB. I fired up a 5250 green screen from my BB and get direct access to the Domino server console. This server also has Auto Recovery enabled so I could see from the console and i5OS jobs that the server had initiated an auto restart and was on it's way back up. Within 5 minutes of receiving the server down notification e-mail I received the server up e-mail. All while sitting on a train with just a BB. I didn't even need the BB but it provided peace of mind that I could see the server taking care of itself.

Now the reason the server faulted was due to SPR# MIAS6VALFX which is fixed in 7.0.2 FP2 and 7.0.3. This server is currently running 7.0.2. I'm hoping this was just an isolated incident but as my earlier post stated, I have started the upgrade process to 7.0.3.

This isn't the first time we have had a Domino server fault and I'm sure it won't be the last. It's nice to have a system that will attempt to take care of itself in an emergency. Thanks IBM/Lotus from all the Domino admins and the users that never know what happened behind the scenes.

Domino 7.0.3 Available

Rob Ingram posted the announcement today. I have already downloaded the i5OS, Windows, and Linux server versions and our test Windows server is already upgraded. I had to start testing it since we may need to roll it into production to resolve a server issue that just appeared yesterday. I'm hoping it was just a one time occurrence. More on that later.

Thursday, October 11, 2007

Joined the i revolution

After dealing with the clunky interfaces on other MP3 players I finally broke down and bought an iPod. I gave the Windows Media and less expensive MP3 player combo a try for a while. It just never measured up to the seamless integration you get with iPod and iTunes.

When the new 3rd generation iPod Nano came out I knew it was time to switch. The video wasn't that big of a deal for me although it has a very sharp screen. Even though it is only 2 inches it's easy to view a TV show. I like the interface, form factor, and seamless integration with iTunes. I downloaded 2 episodes of The Office from iTunes just to check out the video and see if I would find it useful. I also downloaded the new David Crowder Band album Remedy while I was in the iTunes store.

I have the 8 GB model and with 3 hour long TV shows, a couple of the latest IdoNotes and Taking Notes podcasts, and 125 songs I still have 5GB free. I don't need my entire music library with me so I don't mind just having a selection available on the iPod. I especially don't need my wife's albums on there so I use the manual sync option in iTunes.

I bought my wife the 1st gen Nano a year ago and after getting used to the iTunes interface decided this combo just works.

A co-worker pointed me to the Best Skins Ever site for a protective skin. I'll be getting this for mine and my wife's device and will post the results.

Wednesday, September 26, 2007

Comcast filtering Lotus Notes (Update)

This is an update to my earlier post. I first want to say that I am pleased to see that IBM/Lotus cares about their customer base and has been working with me and other clients to try and resolve this issue from their end. Even though this is in no way a Lotus Notes issue they still see the need to try and resolve it. Comcast on the other hand is still playing dumb.

I have been in contact with IBM/Lotus and have been testing solutions with them to work around this filtering. So far I have not been successful.

I finally have an end-to-end trace to share which shows that Comcast is filtering the port 1352 traffic. The images below show that Comcast is impersonating and using man-in-the-middle tactics to filter the traffic as stated in the CNet post. The images show a network packet trace from the client side and from the server side during the same session. This was a new memo composed within Notes with a 6 MB attachment and then saved as a draft to the server database. The transfer did not succeed.

Below is a portion of the Notes client trace. The Domino server is .18 and the Notes client PC is .202 (private IP).
The Notes client will tolerate 3 sets of these RST packet streams before it gives up. The one above was that last set before Notes gave the 'Remote server no longer responding' message.

Below is the trace from the Domino server showing what it saw at the same time the Notes client was seeing the above packets. There is about a 5 to 6 second time difference between the server and client clocks so the times don't match up exactly. The Domino server is .18 and the Notes client PC is .19 (public IP).
As you can see from these traces, the Notes client saw the RST packets coming from the Domino server IP and the Domino server saw the RST packets coming from the Notes client PC. However the trace doesn't show either one of them sending the RST packets which means something on the network in between was sending them. The Sandvine appliance (or whatever Comcast is using) sends the RST packets to both systems while imitating the other.

I will continue to work with IBM/Lotus to see if we can come up with a workaround for Notes. Hopefully Comcast will see the error of their ways and limit their filtering to the apps they are actually trying to filter.

Saturday, September 22, 2007

PPG Flying - Polo Field

It was a beautiful evening for a Powered Paragliding flight in the Chicago area. Since this is my first PPG post on this blog and you are wondering what the heck it is you can click here to find out more about it. Or you can click here to see some pictures of it from a couple years ago.

The call went out on our local Yahoo group and we had 9 pilots show up. Three of them came up from the Joliet area. I did some initial kitting since the wind was still a little gusty. Kitting showed the wind to be smooth so I decided to throw on the motor and go flying. I did a reverse launch for the first time in a while but it went without a hitch. The wind was still a little gusty but not bumpy at all. I flew around for a while and watched other pilots arrive. As the sun dipped lower it began to get cooler. I had launched with jeans and just my short sleeved t-shirt so it was time to land and get the long sleeve pull over on. I ended up falling to my knees when I landed but stood up right away. The wing was still flying so I walked it over closer to my car. The wing is what we call the paraglider wing flying over our heads giving us lift. It's like flying a really big kite.

When I relaunched I did a forward launch (reverse and forward are different ways of launching a PPG and each one depends on the strength of the wind). My motor died on me twice before I started the launch. The first time I took it off my back and restarted it. The second time I was already hooked into the wing so I flagged Dave over to help me start it. It's a pull start motor like a lawn mower and is hard to start yourself when it is on your back. After the motor was started the launch was uneventful and off I was flying again.

Shortly after sunset we all landed, packed up, and talked about the evening flying. It was good too see all the pilots again. Some I used to fly a lot with in the past but hadn't flown with this year until this evening.

Flickr photo set

Friday, September 21, 2007

Database Icon Little Secrets

I was working on an issue yesterday where the design of a database was replaced on one server but when it replicated with another server, the design property 'Inherit design from master template' reverted back to the original value. I discovered that when the design task runs on a server (1am by default) it updates the field that holds this property even though it should only be reading it. So the design was replaced on one replica and the database didn't replicate with the other server until after the design task had run on the other server. Domino replication did what it was designed to do and made the latest entry the current one.

So where is the 'Inherit design from master template' property stored you ask.... None other than the Database Icon design element. It's within the $TITLE field. This field is a concatenated field containing the Database Title and Inherit from template property. If you look at it through NotesPeek or the Ytria ScanEZ tool you can see it is modified when the Design task runs.

The Database Icon element also stores the following:
$DefaultFrameset - Database frameset launch parameter
$FormsTemplateFile - iNotes Forms database pathname to use
$LANGUAGE - Language packs available on the server when the database was created
$DefaultLanguage - Will be present if multiple language packs are available
$SoftDeleteExpireHours - Soft delete expire time in hours parameter
$TemplateFileName - Filename of the template last used to update the database
$TemplateModTime - The last time the template specified above was modified
$TemplateServerName - The Domino server where the template was read from
$TITLE - Database Title + (new line character) + #2 + Inherit design from master template name OR if the database is a template then, Database Title + (new line character) + #1 + master template name
$UpdatedByLimit - Limit entries in $UpdatedBy field property
IconBitmap - The actual database icon

I'm sure there are other fields but these are the ones seen in a standard mail file.

Thursday, September 20, 2007

Domino vs Exchange Administrator commercial

A colleague and I were enjoying reading the latest incident of an Exchange shop trying to get their BlackBerries working after fixing a corrupted message store when I thought of a good tag line for Domino Administrators... "Sleep is a wonderful thing".

My colleague
Mark thought it would be a good commercial like the Apple vs. Microsoft ones. The Domino admin could be in a warm bed sound asleep while the Exchange admin is in a cold fluorescent lit server room staring at a monitor trying to shield himself from the air blowing through the raised floor.

Although, Mark was the one on call this week and had to deal with 2 late night issues. Neither of them Domino related. One was a Tivoli Storage Manager issue affecting our archiving system and the other was a network device playing havoc with our Internet connectivity.

Monday, September 17, 2007

Domino SMTP and DNS connectivity

This past weekend was our maintenance window weekend and we ran into an issue with one of our inbound SMTP servers. We didn't have anything specifically planned for this server so it was a surprise when it stopped accepting inbound mail on Sunday morning. We use Postini on the front end of this server and we were seeing connections accepted from Postini but no messages being delivered. We were also seeing the following...

SMTP Server Error: Access to the server is restricted due to maximum number of users.

It was discovered that a firewall change had prevented this server from connecting to the DNS servers so all the SMTP connections were waiting on DNS lookups. Since we use Postini we only allow their servers to relay through by IP address so I wouldn't expect an incoming connection to require a DNS lookup. Evidently Domino SMTP still tries to query DNS during an SMTP connection even if there aren't any hostname/domain name restirctions.

What made this even worse was our redundant SMTP server was also down due to a scheduled building power outage so all inbound Internet traffic was down. Once we resolved the DNS issue the server started delivering mail again.

It also took a while for this issue to appear. The firewall change was made many hours earlier but the server didn't stop accepting messages until the DNS cache timeout expired.

So, if your Domino SMTP server is accepting connections but not messages, check the DNS connectivity.

Thursday, September 13, 2007

That's a big bat

I have an uncle who is a huge baseball fan and today I took him to the Louisville Slugger bat factory and museum in Louisville, KY. This is where they make the wooden bats and give tours of the factory. They have been making bats since the 1800's so there is a lot of memorabilia there. We spent about 2 hours for the tour, a movie, and exploring the museum.

I'm not that big of a baseball fan so my enjoyment came from the mode of transportation. You see Louisville, KY is about 5 hours by car from my house but my uncle lives down state, about 2 hours from my house. So I would have had to drive 2 hours to pick up my uncle then drive another 4 and a half hours to Louisville. This also would have required an overnight stay. So what did we do instead, what anyone with a pilots license would do, I flew. I left my house at 6am and was home by 7pm. A 45 minute flight to pick up my uncle and then an hour and 45 minute flight to Louisville. Not the cheapest way to go but a lot more efficient and fun.

This was my longest cross country thus far and was a perfect flight. Glass smooth on the way there in the morning at 7500 feet. On the return trip the afternoon bumps were active but at 6500 feet it smoothed out.

My uncle got to see the museum and I got some more cross-country time under my belt. All in a day.

Tuesday, September 11, 2007

Comcast filtering Lotus Notes

For the past couple of months our Lotus Notes users (myself included) have experienced dropped connections in Lotus Notes when sending e-mail with attachments from a Comcast Internet connection. Evidently Comcast has implemented a filtering device to try and curtail the use of P2P software on it's network. Unfortunately they didn't do their homework and have applied this filter to port 1352 as well which is a registered Lotus Notes port.

Comcast won't admit to doing this even though there are posts about it all over the Internet.


IBM is writing a technote about it but they currently will not try to contact Comcast.

So if your Notes users are on a Comcast Internet connection and are trying to send an e-mail over 2 Mb thru a pass-thru server, they will get the pop up "Remote system no longer responding". Essentially the filtering device on the Comcast network sends a bunch of TCP RST packets back to the Notes client causing it to drop the connection. This is on uploads only, downloads work fine.

Friday, September 7, 2007

Finally out there

Well here it is folks, another blogger. I finally jumped on the band wagon and got myself a blog. I don't plan on this being a blog about a specific topic but rather a blog regarding all my interests. After all it is my name in the title and labels can be used to differentiate the topics.

Anyway, I'll mostly post topics related to Lotus Notes and Domino (my job), Flying (what my job pays for), Photography (an attempt to document my life), and my family (the most important thing to me on this earth). Although the family posts will be kind of vague since I am still concerned about the whole identity theft thing.

It took some time before I finally decided to use Blogger. I wanted to use the Domino Blog template on my home server but I figured Comcast would eventually block it if traffic spiked. I'm also too cheap to pay for a hosting service. So Blogger was the best bet to get started with.

I'll be tweaking the blog template as I discover what Blogger lets me do. So things may be changing around here.