Calling a Web Service in a loop

Wow, does this ever suck.

Ok, so here's the problem. I have a service that takes an XML file, which contains a list of parts from another system, and I need to synchronize the list of parts with our system. In order to do this, the service loops through the list of parts, and for each one, it attempts to look up the part in our system. If it exists, it gets it, and updates its properties with data from the file. If it doesn't, it creates a new one, with properties from the file. It gathers all these creations/changes into a collection, and then saves the batch to the database.

Seems pretty straightforward, right? The catch is, the interface to our system is through an authenticated web service.

Well, gee, that's not so tough, right? After all, in .Net, you just create a reference to the web service, and it does most of the work for you, creating all the wrapper classes and so forth. And, most of the time, it's as simple as that.

The problem, though, is that in this sample data set, we had about 5,000 parts. For each part, a query had to be made to the web service: Give me this part. (It'll return either the part, if it exists, or a null, meaning it doesn't, and I have to create one.) And for some reason, after about 3800 or 3900 calls to the web service in rapid succession, it would just quit. "Unable to connect to web service." The inner exception revealed a little more detail: "Only one usage of each socket address (protocol/network address/port) is normally permitted."


After considerable digging and googling, I finally unearthed this blog post by Durgaprasad Gorti, which reveals the problem. An authenticated call closes the connection, but the Windows TCP stack holds the socket in a "TIME_WAIT" state for four minutes by default before it can be reused. While he does offer a registry hack to tell Windows to cut that time shorter, I wanted to find a way to do it in code, so it's one less variable to keep track of on a client's machine.

Unfortunately, all of my experimentation proved fruitless. No matter how I played with the ServicePoint, trying to forcibly close it, setting its timeouts to minimum values, whatever, the sockets stayed open too long.

So much for trying to out-think the Microsoft guy.

His code-based solution, therefore, is the one I'm using. Unfortunately, it's not great in that it basically just delays the problem — by expanding the range of sockets it can use, instead of crashing in under 4,000 calls, the limit is raised to 60,000.

He gives the basics of how to implement it, but unfortunately he doesn't indicate where the code needed to go. Fortunately, I found another blog post, by Kamil Pakur, that gave me just the clue I needed. (Incidentally, he's trying to solve the same problem, but his solution — forcing the KeepAlive to false and the HTTP protocol version to 1.0 — didn't change anything in my scenario; it still crashed in under 4,000 calls. In fact, it would seem that KeepAlive=false, which is automatic in an authenticated scenario, is the source of the problem.)

So, here's what I did:

  1. Copied the "namespace" and "public partial class" lines from the auto-generated Reference.cs file representing my web service into a new code file.
  2. In that file, copied Gorti's public static IPEndPoint BindIPEndPointCallback method.
  3. Added to that file a protected static int m_LastBindPortUsed = 5001; line (which is used in the "BindIPEndPointCallback" method).
  4. Added a method to override the service's GetWebRequest event that set the ServicePoint.BindIPEndPointDelegate to the BindIPEndPointCallback method (the first line on Gorti's code block).

My entire class file looks a lot like this:

namespace ProjectName.ServiceName

 public partial class Service : System.Web.Services.Protocols.SoapHttpClientProtocol
  protected override System.Net.WebRequest GetWebRequest(Uri uri) {
   System.Net.HttpWebRequest webRequest = (System.Net.HttpWebRequest)base.GetWebRequest(uri);
   webRequest.ServicePoint.BindIPEndPointDelegate = new System.Net.BindIPEndPoint(BindIPEndpointCallback);
   return webRequest;
  //protected override System.Net.WebResponse GetWebResponse(System.Net.WebRequest request) {
  //    if (request is System.Net.HttpWebRequest) {
  //        System.Net.HttpWebRequest httpRequest = (System.Net.HttpWebRequest)request;
  //        System.Net.WebResponse response = base.GetWebResponse(httpRequest);
  //        httpRequest.ServicePoint.MaxIdleTime = 1;
  //        httpRequest.ServicePoint.ConnectionLeaseTimeout = 1;
  //        httpRequest.ServicePoint.CloseConnectionGroup(httpRequest.ServicePoint.ConnectionName);
  //        return response;
  //    } else
  //        return base.GetWebResponse(request);
  protected static int m_LastBindPortUsed = 5001;
  public static System.Net.IPEndPoint BindIPEndpointCallback(
   System.Net.ServicePoint servicePoint,
   System.Net.IPEndPoint remoteEndPoint,
   int retryCount) {
   int port = System.Threading.Interlocked.Increment(ref m_LastBindPortUsed);
   System.Threading.Interlocked.CompareExchange(ref m_LastBindPortUsed, 5001, 65534);
   if (remoteEndPoint.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork) {
    return new System.Net.IPEndPoint(System.Net.IPAddress.Any, port);
   } else {
    return new System.Net.IPEndPoint(System.Net.IPAddress.IPv6Any, port);

(I left in the code for the GetWebResponse override, just so you can see some of the things I tried to clear up the sockets. It's all commented out now, of course, because it, quite simply, just doesn't do a blasted thing.)

The service now completes the run on our test data. However, I'm still not comfortable with the solution. There is an upper limit to the amount of data that it can process at a time.

Maybe a better solution is to ship the file across to the web service and do the processing there. (It would certainly make for a cleaner interface, an actual web service that does tasks, instead of the glorified and bloated data access layer we have now.) But that, too, is a double-edged sword. Web services have limits on how much data can be shipped, plus timeouts on how long the client will wait for a response.

What really surprised me in researching this is how little information there was on this problem. I guess calling an authenticated web service in a loop isn't a common scenario. It was only when I googled the text of the inner exception that I found it, amongst a lot of results pointing to people actually opening a lot of TCP connections manually.

I did come across a few posts of people complaining about web services failing in a loop, but the responses (when there were any given) were nowhere near the actual solution (suggesting a timeout issue with the session or authentication cookies). Maybe this post will be of more help, since I tried to bring the problem and solution together.


cwam said...

I have same problem of web service slowdowns:
web service in gSOAP 2.7.10, called by a .NET clients in loop.

I turned gsoap KeepAlive to false. This helps but doesn't solve.

From there http://www.isi.edu/atomic2/pcbswap.html
it could be a solution to manually send RST order. But how (in C#) ?
Any ideas ?

Yakko Warner said...

Huh. That's an interesting idea. According to that link, it does seem that, if one can get the client to send an RST, it just might ditch the TIME_WAIT loop.

A quick googling, though, seems to indicate that RST is something that is sent automatically (ironically, the posts I'm finding are of people complaining about RSTs going out when people don't want them). I'll have to poke around a bit and see if I can find a way to force an RST and see what happens. With any luck, I'll be able to post an update here.

Thanks for the lead!

Yakko Warner said...

I got this email from Gareth (don't know if you want me using your full name, so I won't), which brings up a good point:

While implementing this though I did notice a small issue that can occur. If you use some of the ports then close the app and re-open, the static counter starts again from 5001, which if within the time frame (240 seconds from memory), you will again recieve the same error.

My solution was to save the current port (as they were being used) to a user setting and then use the "las used port" setting when the static is initilized - effectively picking up from where is left off...

Incidentally, I did look into the RST possibility. It may not be possible to do if this is considered "low-level socket access", which if I was reading right is disallowed as of Windows XP without a special hack.

In other words, "You're not allowed to fix our broken implementation."