Wednesday, September 26, 2007

Comcast filtering Lotus Notes (Update)

This is an update to my earlier post. I first want to say that I am pleased to see that IBM/Lotus cares about their customer base and has been working with me and other clients to try and resolve this issue from their end. Even though this is in no way a Lotus Notes issue they still see the need to try and resolve it. Comcast on the other hand is still playing dumb.

I have been in contact with IBM/Lotus and have been testing solutions with them to work around this filtering. So far I have not been successful.

I finally have an end-to-end trace to share which shows that Comcast is filtering the port 1352 traffic. The images below show that Comcast is impersonating and using man-in-the-middle tactics to filter the traffic as stated in the CNet post. The images show a network packet trace from the client side and from the server side during the same session. This was a new memo composed within Notes with a 6 MB attachment and then saved as a draft to the server database. The transfer did not succeed.

Below is a portion of the Notes client trace. The Domino server is .18 and the Notes client PC is .202 (private IP).
The Notes client will tolerate 3 sets of these RST packet streams before it gives up. The one above was that last set before Notes gave the 'Remote server no longer responding' message.

Below is the trace from the Domino server showing what it saw at the same time the Notes client was seeing the above packets. There is about a 5 to 6 second time difference between the server and client clocks so the times don't match up exactly. The Domino server is .18 and the Notes client PC is .19 (public IP).
As you can see from these traces, the Notes client saw the RST packets coming from the Domino server IP and the Domino server saw the RST packets coming from the Notes client PC. However the trace doesn't show either one of them sending the RST packets which means something on the network in between was sending them. The Sandvine appliance (or whatever Comcast is using) sends the RST packets to both systems while imitating the other.

I will continue to work with IBM/Lotus to see if we can come up with a workaround for Notes. Hopefully Comcast will see the error of their ways and limit their filtering to the apps they are actually trying to filter.

16 comments:

Anonymous said...

nice detective work !

Jonathan said...

Suppose encryption would help?

Stanza said...

Is it possible to tunnel this over ssh?

Anonymous said...

Well isn't that nice. To me though, the real question is why in the world are you not using a VPN client to encrypt all your traffic in the first place. Then they would have no way to do a man-in-the-middle attack like this unless they tried it on the entire tunnel.

Shame on ISP, bad, bad ISP.

Shame on users that work without VPN's, bad, bad users.

pepolez said...

If you pay for an internet connection, that's what you should get - a damn internet connection. This filtering crap is ridiculous.

GuyFromOhio said...

"To me though, the real question is why in the world are you not using a VPN client to encrypt all your traffic in the first place."

Because Comcast whacks VPN ports as well. In my area, Comcast was famous for holding customers hostage until a pricier "business plan" was purchased. That would open up other ports beyond just HTTP, POP and SMTP, allowing for usage of VPN. Bad ISP? I suppose - it's their pipes, they can do with them as they wish. Bad for business, definitely - Comcast is an epithet in this part of the world, and I'll share that with all who inquire. I'd go back to dialup or just live without before Comcast got a nickel from me. Obviously not everyone has that choice, unfortunately Comcast knows that and is taking advantage of it.

Also, LN client can encrypt traffic to its server transparently over any IP connection - VPN isn't strictly necessary for secure communications over open internet.

As the author of this article correctly points out, Comcast is making more work for itself and eventually will lose this little contest. So long as it has customers in areas with no competition, it can get away with this indefinitely. But anywhere there is an alternative (WOW, TW, AT&T) they're going to lose customers, particularly as the demand for downloadable rich content and streaming content grows.

Anonymous said...

You can tunnel SSH over HTTP. You can make it indistinguishable from ordinary HTTPS traffic.

http://dag.wieers.com/howto/ssh-http-tunneling/

Kenny said...

this goes beyond nonsense. comcast puts up an insane amount of ads touting their own services - like 3 per hr. whatever. i'm done w/ tyhis comcast shit as soon as i can figure an alternate. in addition gto this, i'll not buy anything from any company that runs ads through comcast. i've alredy changed my oil change service thanks to those douchebags.

Kevin Kanarski said...

We do encrypt the traffic between the Notes client and the Domino server. I don't think the filter actually looks at the data. It just looks at the amount and starts limiting after a certain amount is transferred.

Carlos Martins said...

Unfortunately, it's not only Comcast. My ISP in Portugal (Netcabo) just recently started to use the same tactics (apparently Sandvine too - I suspect)

Just like I had predicted, it also started messing some of my online game authorization packets - meaning me, and several hundred other users are now unable to login to ETQW.

Bit the absolute worst part is that they: categorically deny that they filter out any traffic - even when I sent them a similar wireshark log.

This is getting really crazy!

Anonymous said...

They seem to whack all communications regardless. Battlefield 2 will die for me, Radar (a bug reporting app), iTunes, ssh...

Anonymous said...

For those who haven't seen Kevin's latest... it seems like Notes traffic is passing through Comcast again.

Anonymous said...

Welcome to a Comcastic !!

Anonymous said...

Time for the lawyers to step in. A few sharp whacks on the head in court might clear Comcast's thinking on their service.

Flynn said...
This comment has been removed by the author.
Flynn said...

I've run into a worse issue. It appears they're blocking ALL traffic to my server from my work (who uses Comcast). HTTP, POP/SMTP, you name it. Started at 10am this morning. No clue why. I was lucky enough to have an SSH session going so I could test it on both ends and see that the RST packets weren't coming from my server, and that all of my server's responses were being ignored. At home I'm with AT&T, and other Comcast users I know can reach me, but for some reason, they're blocking me to my work IP.