Two days spent working onto issues which were side effects of incorrect Wcf configuration for the wsHttpBinding for load balanced environments:
(Thanks to Anthony for digging those out while me and Conor were deeply into debugging :))
It can be really hard though to come to understanding that it is load balancer that might be causing problems. Only one of the intermittent scenarios led to the security exceptions in the service trace log caused by load balancer sending requests to the different server with which security context was initially communicated.
Other scenarios had no security exceptions until the first timeout happened. Then speculatively either load balancer was switching to the second server for the channel initially established via first server. We ran through long and intensive chain of investigation, so I can't remember all of the details now. The whole thing can be very intermittent.
The whole investigation was further complicated by having a client written as C++ application that is calling a WCF proxy replacement for COM+ components, which in turn is calling WCF service which in turn is calling COM+ components. Whole thing was done like this to substitute the Application Center ::).
Feeling something can be bad on a load balancer side, I reconfigured the service/client to use basicHttpBinding with windows integrated authentication. And it worked like a charm. Though the first article mentioned above gives very good idea of what should be done to load balance with wsHttpBinding, but see the note in the msdn article above:
Turn off Security Context Establishment ...
Do not use reliable sessions. This feature is off by default.
So it seems that wsHttpBindings can't be used in the load balanced environment for reliable messaging. It is not obvious to me right now why not, going to investigate further.
Would not be possible to solve without:
Tools: WinDbg, Wcf Service trace viewer, Wcf Configuration Console, Process Explorer
Blogs:http://blogs.msdn.com/tess/ - A must read for anybody doing in the core .net debugging!
Books: John Robbins - Debugging .NET 2.0 Applications - another must.
Great guys: Anthony, Conor