This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
PUTs generate a "HTTP/1.1 100 Continue" response before the
"HTTP/1.1 200 OK" response, and so we were mistakenly picking up
the 100 status code and not returning any data. Improve the
header callback to correctly process any number of status codes.
Various places suggest that this is needed for better thread-safety,
and the only drawback is that some systems cannot timeout properly on
DNS lookups.
Now nilmdb.client, nilmdb.server, nilmdb.cmdline, and nilmdb.utils
are each their own modules, and there is a little bit more of a
logical separation between them. Various changes scattered throughout
to fix naming (for example, nilmdb.nilmdb.NilmDBError is now
nilmdb.server.errors.NilmDBError).
Reduced usage of "from __future__ import absolute_import" as much
as possible. It's still needed for the functions in the nilmdb/server
directory to be able to import the nilmdb module rather than the
nilmdb.py script.
This should hopefully ease future packaging a bit.
Inside the pycurl callback, we can't raise exceptions, because the
pycurl extension module will unconditionally print the exception
itself, and not pass it up to the caller. Instead, we have the
callback return a value that tells curl to abort. (-1 would be best,
in case we were given 0 bytes, but the extension doesn't support
that either).
This resolves the 'Exception("should die")' problem when interrupting
a streaming generator like stream_extract.
Curl will give an error if we call .setopt() while a .perform() is
in progress, for example if we try to do a stream_insert() while
in the middle of a stream_extract(). Move the setopt() to the
beginning of the get/put functions to ensure that we hit this
error before we mess with the URLs or anything else.
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug. Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not. fun!
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
that. This requires a bit of restructuring of server.py:intervals()
to allow us to properly report errors before beginning the stream.
Make the nilmdb.httpclient save a copy of HTTP header responses, and
add a test that the saved responses to verify that the
transfer-encoding is Chunked for the /stream/interval request. This
should check that the above bug is fixed and doesn't show up again
if we switch to a different WSGI server, etc.
Tweak size estimates in nilmdb for /stream/interval
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10884 ddd99763-3ecb-0310-9145-efcb8ce7c51f
to the database, not the client. The server now maintains the open
HTTP connection and sends a continuous streaming reply to the GET
request.
HTTP client side uses an Iteratorizer to turn the curl.perform()
callback into an interator, and returns an iterator that yields
individual lines to the caller rather than buffering up all the data
at once. Should still be able to handle errors etc.
Server changed to return a "streaming JSON" instance for the
/stream/interval requests. This is just a series of independent
JSON documents (one per interval), separated by newlines.
Adjust nilmdb's max_results a bit. Now, multiple requests only exist
between the server <-> nilmdb threads, and they exist just to avoid
blocking the nilmdb thread by any one server thread for too long.
So adjust the size accordingly to match the fact that this is non-json
encoded data.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10881 ddd99763-3ecb-0310-9145-efcb8ce7c51f
