AJAX :: Intermittent Communication Failures

I have a web app with a game and chat updating with ajax very frequently sending thousands of XHR requests to the server during a user's session. The problem is that most clients experience intermittent connection failures or timeouts. On the server it appears to be fine, all requests are logged to have completed successfully and on time but the client gets either a communication failure or a timeout. This makes the user feel like the game or chat just got stuck.

I use YUI for making the XHR calls but I replaced that with jQuery with exact same results. I make sure that only one query is active at a time, so I am sure that I am not hitting the max 2 connections problem. The failure rate may be 1 in 200 hundred requests or something but it seems that when a client has a failure then it fails at a higher rate, say 1 in 10 until the problem goes away. I have clients send back an error request when such errors happen in order to track the problem. I have tried various things to try to figure out what is wrong but I have not found anything. I don't think that it's the client's connection as it happens to a sizable portion of my users, it doesn't look like it's the server either.

So what I am wondering is whether this failure rate is common or expected when doing heavy ajax staff and I need to find a way to work around it or this must be a problem in my system that I need to figure out. If the latter any pointers where to look at?