When enabled, lines like "# interval-start 1234567890123456" and "#
interval-end 1234567890123456" will be added to the data output. Note
that there may be an "interval-end" timestamp followed by an identical
"interval-start" timestamp, if the response at the nilmdb level was
split up into multiple chunks.
In general, assume contiguous data if previous_interval_end ==
new_interval_start.
Added new flag "-R" to command line to perform an automatic removal.
This should be the last of the ways in which a single command could
block the nilmdb thread for a long time.
Previously, we ignored problems when sending intermediate blocks,
since getting more data might make the next attempt succeed.
But in practice, malformed data would just build up, causing
problems. Raise an exception if there's too much data remaining
after trying to send an intermediate block.
Things are now block-focused, rather than line-focused. This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
Previously, we could get empty intervals anyway by having a non-empty
interval and removing a smaller interval around each piece of data.
Turns out that empty intervals are OK and needed in some situations,
so explicitly allow and test for it.
Now supports these operations:
ctx.insert_line()
ctx.insert_iter()
ctx.finalize() (end the current contiguous interval, so a new one
can be started with a gap)
ctx.update_end() (update ending timestamp before finalizing interval)
ctx.update_start() (update starting timestamp for new interval)
block_data += "string" is fast with local variables, but slow with
variables inside some namespace. Instead, build a list of strings and
join them once at the end. This fixes the slowdown that resulted from
the stream_insert_context cleanup.
It still should be closed, but warning each time was mostly for
debugging and it's kind of annoying when writing one-off programs
where it's OK to just let things get torn down as they're completed.
Not closing is not fatal in terms of data integrity etc.
Now nilmdb.client, nilmdb.server, nilmdb.cmdline, and nilmdb.utils
are each their own modules, and there is a little bit more of a
logical separation between them. Various changes scattered throughout
to fix naming (for example, nilmdb.nilmdb.NilmDBError is now
nilmdb.server.errors.NilmDBError).
Reduced usage of "from __future__ import absolute_import" as much
as possible. It's still needed for the functions in the nilmdb/server
directory to be able to import the nilmdb module rather than the
nilmdb.py script.
This should hopefully ease future packaging a bit.
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
Doesn't actually merge them yet; need to change Interval
implementation to allow deletes.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11354 ddd99763-3ecb-0310-9145-efcb8ce7c51f
This lets us quickly count the number of matching rows, rather than
returning them.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10909 ddd99763-3ecb-0310-9145-efcb8ce7c51f
- Flesh out tests for the new nilmdb.layout.Formatter
Coverage doesn't handle the cython module, so this is just
functional stuff, not necessarily complete.
Still need to finish each Layout.format()
- Split out test_client_5_chunked from test_client_4_misc
so it's easier to skip while debugging. Turning off streaming
lets us see tracebacks from within the server's content()
functions.
- More work on stream/extract in cmdline, client, server, nilmdb.
Still needs work on server side, but should be complete in nilmdb.
- Start nilmdb.layout.Formatter class
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10888 ddd99763-3ecb-0310-9145-efcb8ce7c51f
to the database, not the client. The server now maintains the open
HTTP connection and sends a continuous streaming reply to the GET
request.
HTTP client side uses an Iteratorizer to turn the curl.perform()
callback into an interator, and returns an iterator that yields
individual lines to the caller rather than buffering up all the data
at once. Should still be able to handle errors etc.
Server changed to return a "streaming JSON" instance for the
/stream/interval requests. This is just a series of independent
JSON documents (one per interval), separated by newlines.
Adjust nilmdb's max_results a bit. Now, multiple requests only exist
between the server <-> nilmdb threads, and they exist just to avoid
blocking the nilmdb thread by any one server thread for too long.
So adjust the size accordingly to match the fact that this is non-json
encoded data.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10881 ddd99763-3ecb-0310-9145-efcb8ce7c51f