GWT to lighttpd/apache to glassfish 502 proxy or 500 internal errors fix

I’ve been dealing with this for a while now trying to figure out why, when using my online accounting software, users sporadically get a StatusCodeException when sending requests to the server.  I finally this week figured out what was going on; glassfish was dropping the connection or sending bad responses occasionally because it doesn’t behave in the was that the mod_proxy modules of these webservers expect it to.

Originally I was running lighttpd and I was thinking this might be a bug in lighttpd, so I eventually switched to apache.  Once I was running apache I got a much more verbose error – instead of just a plan 500 or 502 status code I got a message.  I googled that error message plus glassfish and found the solution.

I thought I’d share it here so that future searchers who are using lighttpd or apache will have more places to find the answer.

To fix the issue, add:

SetEnv force-proxy-request-1.0 1

SetEnv proxy-nokeepalive 1

To your apache httpd.conf.  I don’t know what the equivalent fix for lighttpd is, if there is any.

From this fix, it appears that glassfish is misbehaving in some way in relation to being behind a proxy, but I don’t what way that is and I’m just glad I fixed this mysterious problem!

If any of you readers have more information about this issue, please comment!

About these ads

14 Responses to GWT to lighttpd/apache to glassfish 502 proxy or 500 internal errors fix

  1. dobes says:

    For those having this problem, it looks like this isn’t the only fix required, since I’m still getting those errors. Another avenue I’m trying is changing the HTTP listener settings in glassfish so it won’t timeout apache’s connections.

  2. liechtyman says:

    Did changing the listener settings help or are you still getting proxy errors.

  3. dobes says:

    I am still getting proxy errors, occasionally. :-(

  4. Lerch says:

    We were getting those errors and were able to recreate them reliably which lead us to the issue with the Reciever Buffer on the Http service for glassfish. We bumped it up to 8192 bytes and now all the errors are gone. Hope this helps.

  5. dobes says:

    Thanks, I’ll give that a try!

  6. dobes says:

    Hmm, it looks like my receiver buffer was already set to 8192. Maybe I’ll bump it up a bit more.

  7. dobes says:

    Oh wait – the receive buffer is also set in the connection pool, got it. Hopefully the problem will go away now … thanks for the tip!

  8. Since the above fixes didn’t really do it, I’ve also been adding parameters to my ProxyPass directive; currently it looks like this:

    ProxyPass “http://aux:8080/app” min=1 max=250 ttl=300 smax=32 retry=0

    Someone somewhere had suggested that retry=0 helps … we’ll see. It seems like all this tweaking has greatly reduced the number of errors I’m seeing, but not eliminated them (my app emails me and dumps an event into our analytics whenever that error occurs).

  9. Olivier says:

    I have exactly the same symptoms – is there a definitive medicine finally ?

  10. dobes says:

    I wish there was.

    My current workaround is to automatically retry the requests. I have made a GWT generator which generates a wrapper for a service; the wrapper automatically retries the call when it gets certain status codes. You can get this from the Kiyaa! project:

    http://code.google.com/p/kiyaa

  11. dobes says:

    I wish there was.

    My current workaround is to automatically retry the requests. I have made a GWT generator which generates a wrapper for a service; the wrapper automatically retries the call when it gets certain status codes. You can get this from the Kiyaa! project:

    http://code.google.com/p/kiyaa

    However, the changes to the buffer size and keepalive settings above are helpful also.

  12. bronius says:

    I have read a bit about setting timeout and keepalive on both apache proxy front and glassfish back but have not yet implemented it (am just about to).

    What do these errors really mean to the client? Do they get a 500 message from either gf of apache? Do they know that a request was not fulfilled? I imagine one innocuous side effect is that they might miss only, say, a single image out of all the http requests on a given page… but does it (the proxy or the browser) automatically request it and seamlessly fulfill the request?

  13. Yes, it would pass back the 500 or 502 error sometimes, other times it might be a weirder IE-specific error in the case of IE, I don’t remember the code.

    This would effect images much, it was a problem for the AJAX calls re-using the same connection after a potentially lengthy pause. The connection would be dead and the various bits along the way handled it in different ways.

  14. marcgenou says:

    ProxyPass / http://10.10.10.X:8080/ disablereuse=On retry=0

    something like that in apache will do the trick

    PD: this is for the thread pool problem

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: