Wednesday, September 26, 2007

Comcast filtering Lotus Notes (Update)

This is an update to my earlier post. I first want to say that I am pleased to see that IBM/Lotus cares about their customer base and has been working with me and other clients to try and resolve this issue from their end. Even though this is in no way a Lotus Notes issue they still see the need to try and resolve it. Comcast on the other hand is still playing dumb.

I have been in contact with IBM/Lotus and have been testing solutions with them to work around this filtering. So far I have not been successful.

I finally have an end-to-end trace to share which shows that Comcast is filtering the port 1352 traffic. The images below show that Comcast is impersonating and using man-in-the-middle tactics to filter the traffic as stated in the CNet post. The images show a network packet trace from the client side and from the server side during the same session. This was a new memo composed within Notes with a 6 MB attachment and then saved as a draft to the server database. The transfer did not succeed.

Below is a portion of the Notes client trace. The Domino server is .18 and the Notes client PC is .202 (private IP).
The Notes client will tolerate 3 sets of these RST packet streams before it gives up. The one above was that last set before Notes gave the 'Remote server no longer responding' message.

Below is the trace from the Domino server showing what it saw at the same time the Notes client was seeing the above packets. There is about a 5 to 6 second time difference between the server and client clocks so the times don't match up exactly. The Domino server is .18 and the Notes client PC is .19 (public IP).
As you can see from these traces, the Notes client saw the RST packets coming from the Domino server IP and the Domino server saw the RST packets coming from the Notes client PC. However the trace doesn't show either one of them sending the RST packets which means something on the network in between was sending them. The Sandvine appliance (or whatever Comcast is using) sends the RST packets to both systems while imitating the other.

I will continue to work with IBM/Lotus to see if we can come up with a workaround for Notes. Hopefully Comcast will see the error of their ways and limit their filtering to the apps they are actually trying to filter.

Saturday, September 22, 2007

PPG Flying - Polo Field

It was a beautiful evening for a Powered Paragliding flight in the Chicago area. Since this is my first PPG post on this blog and you are wondering what the heck it is you can click here to find out more about it. Or you can click here to see some pictures of it from a couple years ago.

The call went out on our local Yahoo group and we had 9 pilots show up. Three of them came up from the Joliet area. I did some initial kitting since the wind was still a little gusty. Kitting showed the wind to be smooth so I decided to throw on the motor and go flying. I did a reverse launch for the first time in a while but it went without a hitch. The wind was still a little gusty but not bumpy at all. I flew around for a while and watched other pilots arrive. As the sun dipped lower it began to get cooler. I had launched with jeans and just my short sleeved t-shirt so it was time to land and get the long sleeve pull over on. I ended up falling to my knees when I landed but stood up right away. The wing was still flying so I walked it over closer to my car. The wing is what we call the paraglider wing flying over our heads giving us lift. It's like flying a really big kite.

When I relaunched I did a forward launch (reverse and forward are different ways of launching a PPG and each one depends on the strength of the wind). My motor died on me twice before I started the launch. The first time I took it off my back and restarted it. The second time I was already hooked into the wing so I flagged Dave over to help me start it. It's a pull start motor like a lawn mower and is hard to start yourself when it is on your back. After the motor was started the launch was uneventful and off I was flying again.

Shortly after sunset we all landed, packed up, and talked about the evening flying. It was good too see all the pilots again. Some I used to fly a lot with in the past but hadn't flown with this year until this evening.

Flickr photo set

Friday, September 21, 2007

Database Icon Little Secrets

I was working on an issue yesterday where the design of a database was replaced on one server but when it replicated with another server, the design property 'Inherit design from master template' reverted back to the original value. I discovered that when the design task runs on a server (1am by default) it updates the field that holds this property even though it should only be reading it. So the design was replaced on one replica and the database didn't replicate with the other server until after the design task had run on the other server. Domino replication did what it was designed to do and made the latest entry the current one.

So where is the 'Inherit design from master template' property stored you ask.... None other than the Database Icon design element. It's within the $TITLE field. This field is a concatenated field containing the Database Title and Inherit from template property. If you look at it through NotesPeek or the Ytria ScanEZ tool you can see it is modified when the Design task runs.

The Database Icon element also stores the following:
$DefaultFrameset - Database frameset launch parameter
$FormsTemplateFile - iNotes Forms database pathname to use
$LANGUAGE - Language packs available on the server when the database was created
$DefaultLanguage - Will be present if multiple language packs are available
$SoftDeleteExpireHours - Soft delete expire time in hours parameter
$TemplateFileName - Filename of the template last used to update the database
$TemplateModTime - The last time the template specified above was modified
$TemplateServerName - The Domino server where the template was read from
$TITLE - Database Title + (new line character) + #2 + Inherit design from master template name OR if the database is a template then, Database Title + (new line character) + #1 + master template name
$UpdatedByLimit - Limit entries in $UpdatedBy field property
IconBitmap - The actual database icon

I'm sure there are other fields but these are the ones seen in a standard mail file.

Thursday, September 20, 2007

Domino vs Exchange Administrator commercial

A colleague and I were enjoying reading the latest incident of an Exchange shop trying to get their BlackBerries working after fixing a corrupted message store when I thought of a good tag line for Domino Administrators... "Sleep is a wonderful thing".

My colleague
Mark thought it would be a good commercial like the Apple vs. Microsoft ones. The Domino admin could be in a warm bed sound asleep while the Exchange admin is in a cold fluorescent lit server room staring at a monitor trying to shield himself from the air blowing through the raised floor.

Although, Mark was the one on call this week and had to deal with 2 late night issues. Neither of them Domino related. One was a Tivoli Storage Manager issue affecting our archiving system and the other was a network device playing havoc with our Internet connectivity.

Monday, September 17, 2007

Domino SMTP and DNS connectivity

This past weekend was our maintenance window weekend and we ran into an issue with one of our inbound SMTP servers. We didn't have anything specifically planned for this server so it was a surprise when it stopped accepting inbound mail on Sunday morning. We use Postini on the front end of this server and we were seeing connections accepted from Postini but no messages being delivered. We were also seeing the following...

SMTP Server Error: Access to the server is restricted due to maximum number of users.

It was discovered that a firewall change had prevented this server from connecting to the DNS servers so all the SMTP connections were waiting on DNS lookups. Since we use Postini we only allow their servers to relay through by IP address so I wouldn't expect an incoming connection to require a DNS lookup. Evidently Domino SMTP still tries to query DNS during an SMTP connection even if there aren't any hostname/domain name restirctions.

What made this even worse was our redundant SMTP server was also down due to a scheduled building power outage so all inbound Internet traffic was down. Once we resolved the DNS issue the server started delivering mail again.

It also took a while for this issue to appear. The firewall change was made many hours earlier but the server didn't stop accepting messages until the DNS cache timeout expired.

So, if your Domino SMTP server is accepting connections but not messages, check the DNS connectivity.

Thursday, September 13, 2007

That's a big bat

I have an uncle who is a huge baseball fan and today I took him to the Louisville Slugger bat factory and museum in Louisville, KY. This is where they make the wooden bats and give tours of the factory. They have been making bats since the 1800's so there is a lot of memorabilia there. We spent about 2 hours for the tour, a movie, and exploring the museum.

I'm not that big of a baseball fan so my enjoyment came from the mode of transportation. You see Louisville, KY is about 5 hours by car from my house but my uncle lives down state, about 2 hours from my house. So I would have had to drive 2 hours to pick up my uncle then drive another 4 and a half hours to Louisville. This also would have required an overnight stay. So what did we do instead, what anyone with a pilots license would do, I flew. I left my house at 6am and was home by 7pm. A 45 minute flight to pick up my uncle and then an hour and 45 minute flight to Louisville. Not the cheapest way to go but a lot more efficient and fun.

This was my longest cross country thus far and was a perfect flight. Glass smooth on the way there in the morning at 7500 feet. On the return trip the afternoon bumps were active but at 6500 feet it smoothed out.

My uncle got to see the museum and I got some more cross-country time under my belt. All in a day.

Tuesday, September 11, 2007

Comcast filtering Lotus Notes

For the past couple of months our Lotus Notes users (myself included) have experienced dropped connections in Lotus Notes when sending e-mail with attachments from a Comcast Internet connection. Evidently Comcast has implemented a filtering device to try and curtail the use of P2P software on it's network. Unfortunately they didn't do their homework and have applied this filter to port 1352 as well which is a registered Lotus Notes port.

Comcast won't admit to doing this even though there are posts about it all over the Internet.


IBM is writing a technote about it but they currently will not try to contact Comcast.

So if your Notes users are on a Comcast Internet connection and are trying to send an e-mail over 2 Mb thru a pass-thru server, they will get the pop up "Remote system no longer responding". Essentially the filtering device on the Comcast network sends a bunch of TCP RST packets back to the Notes client causing it to drop the connection. This is on uploads only, downloads work fine.

Friday, September 7, 2007

Finally out there

Well here it is folks, another blogger. I finally jumped on the band wagon and got myself a blog. I don't plan on this being a blog about a specific topic but rather a blog regarding all my interests. After all it is my name in the title and labels can be used to differentiate the topics.

Anyway, I'll mostly post topics related to Lotus Notes and Domino (my job), Flying (what my job pays for), Photography (an attempt to document my life), and my family (the most important thing to me on this earth). Although the family posts will be kind of vague since I am still concerned about the whole identity theft thing.

It took some time before I finally decided to use Blogger. I wanted to use the Domino Blog template on my home server but I figured Comcast would eventually block it if traffic spiked. I'm also too cheap to pay for a hosting service. So Blogger was the best bet to get started with.

I'll be tweaking the blog template as I discover what Blogger lets me do. So things may be changing around here.