We do this by creating a new requests.Session object for each request,
sending a "Connection: close" request header, and then explicitly
marking the connection for close after the response is read.
This is to avoid a longstanding race condition with HTTP keepalive
and server timeouts. Due to data processing, capture, etc, requests
may be separated by an arbitrary delay. If this delay is shorter
than the server's KeepAliveTimeout, the same connection is used.
If the delay is longer, a new connection is used. If the delay is
the same, however, the request may be sent on the old connection at
the exact same time that the server closes it. Typically, the
client sees the connection as closing between the request and the
response, which leads to "httplib.BadStatusLine" errors.
This patch avoids the race condition entirely by not using persistent
connections.
Another solution may be to detect those errors and retry the
connection, resending the request. However, the race condition could
potentially show up in other places, like a closed connection during
the request body, not after. Such an error could also be a legitimate
network condition or problem. This solution should be more reliable,
and the overhead of each new connection will hopefully be minimal for
typical workloads.
Jim Paris
da72fc9777