This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
These functions can now take an object or a type (class).
If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
Occasional segfaults may be the result of performing thread-unsafe
operations. This class decorator verifies that all of its methods
are called in a thread-safe manner.
It can separately warn about:
- two threads calling methods in a function (the kind of thing sqlite
doesn't like)
- recursion
- concurrency (two different threads functions at the same time)
This decorator makes a class always be serialized, including its
instantiation, in a separate thread. This is an improvement over
the old Serializer() object wrapper, which didn't put the
instantiation into the new thread.
Previously, we could get empty intervals anyway by having a non-empty
interval and removing a smaller interval around each piece of data.
Turns out that empty intervals are OK and needed in some situations,
so explicitly allow and test for it.
Now nilmdb.client, nilmdb.server, nilmdb.cmdline, and nilmdb.utils
are each their own modules, and there is a little bit more of a
logical separation between them. Various changes scattered throughout
to fix naming (for example, nilmdb.nilmdb.NilmDBError is now
nilmdb.server.errors.NilmDBError).
Reduced usage of "from __future__ import absolute_import" as much
as possible. It's still needed for the functions in the nilmdb/server
directory to be able to import the nilmdb module rather than the
nilmdb.py script.
This should hopefully ease future packaging a bit.
Inside the pycurl callback, we can't raise exceptions, because the
pycurl extension module will unconditionally print the exception
itself, and not pass it up to the caller. Instead, we have the
callback return a value that tells curl to abort. (-1 would be best,
in case we were given 0 bytes, but the extension doesn't support
that either).
This resolves the 'Exception("should die")' problem when interrupting
a streaming generator like stream_extract.
Use numbered subdirectories to avoid having too many files in one dir.
Add appropriate tests.
Also fix an issue where the mmap_open LRU cache could inappropriately
open a file twice because it was using the optional "newsize"
parameter as a key -- now lrucache can be given a slice object that
describes which arguments are important.
We replace cStringIO with StringIO subclass that forces UTF-8
encoding, and explicitly convert commandlines to UTF-8 before
shlex. These changes will only affect tests, not normal commandline
operation.
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug. Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not. fun!
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
test_interval_speed is about O(n * log n), which is good -- but the
constants are high and it hits swap on a 4G machine for the 2**21
test. Hopefully cython helps!
tree.
This is currently hitting an issue where it's considering the
intersection of [0,1] and [1,2] to be [1,1]. It matches the
fully-closed definition of intervals, unlike before -- but might
cause issues. Need to consider whether test case is correct.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11355 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Doesn't actually merge them yet; need to change Interval
implementation to allow deletes.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11354 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Current/old design has specific layouts: RawData, PrepData,
RawNotchedData.
Let's get rid of this entirely and switch to simpler data types that
are
just collections and counts of a single type. We'll still use strings
to describe them, with format:
type_count
where type is "uint16", "float32", or "float64", and count is an
integer.
nilmdb.layout.named() will parse these strings into the appropriate
handlers. For compatibility:
"RawData" == "uint16_6"
"RawNotchedData" == "uint16_9"
"PrepData" == "float32_8"
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10981 ddd99763-3ecb-0310-9145-efcb8ce7c51f
This lets us quickly count the number of matching rows, rather than
returning them.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10909 ddd99763-3ecb-0310-9145-efcb8ce7c51f
row individually, when extracting data.
Switch to using bisect module when doing the bisection, to lessen the
chance of errors.
Added syslog ability for timer module, for timing stuff deep inside
the server.
Make the chunked/non-chunked test just give a warning, rather than
failing the tests, for debugging purposes. Alternate approach would
be to disable "die on error" for the tests.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10896 ddd99763-3ecb-0310-9145-efcb8ce7c51f
though; need to figure out where the slowdown lies.
Add stream existence check to server's /intervals and /extract paths,
add tests for it.
Make start and end arguments optional for /extract, like /intervals
Move --quiet command line option to just the insert subcommand.
It's the only one that uses it right now, and otherwise it doesn't
show up in after a "nilmtool.py intervals --help". Might revisit this
later if more commands start supporting --quiet.
Change cmdline/extract's write into a print, to keep the trailing
newline.
Fix lingering uses of Interval in nilmdb and change to DBInterval
instead.
Fix nilmdb interval bisection:
- handle common case optimization correctly
- db_endpos is always one after the last row, so use hi=db_endpos-1
Finish nlimdb stream_extract
Add a bunch of cmdline tests for extract, particularly testing border
cases around start/end. Compares output to a set of files stored in
the tests/data dir.
Some more tests in test_client to get better coverage.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10893 ddd99763-3ecb-0310-9145-efcb8ce7c51f
- Flesh out tests for the new nilmdb.layout.Formatter
Coverage doesn't handle the cython module, so this is just
functional stuff, not necessarily complete.
Still need to finish each Layout.format()
- Split out test_client_5_chunked from test_client_4_misc
so it's easier to skip while debugging. Turning off streaming
lets us see tracebacks from within the server's content()
functions.
- More work on stream/extract in cmdline, client, server, nilmdb.
Still needs work on server side, but should be complete in nilmdb.
- Start nilmdb.layout.Formatter class
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10888 ddd99763-3ecb-0310-9145-efcb8ce7c51f
that. This requires a bit of restructuring of server.py:intervals()
to allow us to properly report errors before beginning the stream.
Make the nilmdb.httpclient save a copy of HTTP header responses, and
add a test that the saved responses to verify that the
transfer-encoding is Chunked for the /stream/interval request. This
should check that the above bug is fixed and doesn't show up again
if we switch to a different WSGI server, etc.
Tweak size estimates in nilmdb for /stream/interval
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10884 ddd99763-3ecb-0310-9145-efcb8ce7c51f
into a generator. Needed to get pycurl's WRITEFUNCTION to give us a
generator instead. It's a bit tricky and uses threads and queues, but
seems to work OK.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10880 ddd99763-3ecb-0310-9145-efcb8ce7c51f
us to look at just some of the intervals without having to reconstruct
an entire IntervalSet class -- which greatly reduces server load when
handling requests that cover large interval ranges.
Add Client.get and Client.put, analogous to getjson and putjson but
without parsing the result as json.
Add Client.stream_extract. Still needs server side love.
Allow Cmdline subcommands to provide a return value that turns into
the exit code.
More work on cmdline.extract.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10851 ddd99763-3ecb-0310-9145-efcb8ce7c51f
On the big database, the server takes a few seconds to figure out the
interval intersections. Need to think about how to improve that --
the real key might be to start reducing the number of intervals we're
storing by combining them, potentially as they're inserted.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10838 ddd99763-3ecb-0310-9145-efcb8ce7c51f
variable instead now. Adjust tests accordingly.
Start list --detail option, using stream/intervals request.
Frontend should be ready, backend needs implementation.
Put interval adding back into nilmdb:_add_interval so things work.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10833 ddd99763-3ecb-0310-9145-efcb8ce7c51f
intervals are accessed, so it doesn't need to keep rebuilding them as
long as it's running.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10800 ddd99763-3ecb-0310-9145-efcb8ce7c51f
the TestIntervalSpeed versus the previous bisect version, but should
be better in general.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10748 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Make nilmdb.cmdline a proper class
Fix various stuff
Add /dbpath command to get DB path
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10677 ddd99763-3ecb-0310-9145-efcb8ce7c51f
cover overlapping data, missing data, etc.
Consolidate nilmdb.client.MyCurl.getjson() and putjson().
Add __str__ for more compact representation of IntervalSet
Make stream_id not NULL in SQL database.
Format NilmDBError with name of exception for more clarity
Work around CherryPy issue 1138 by removing HTML escaping before
returning JSON error responses from the server.
Reached 100% test coverage again (woohoo)
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10665 ddd99763-3ecb-0310-9145-efcb8ce7c51f
More work with nilmdb.stream_insert. Checks for overlap between
parser output and database intervals, and actually inserts the
data into pytables. Need to benchmark/figure out whether we can
use table.append rather than the row.append() nonsense -- it could
simplify things quite a bit.
Improve layout class and add tests to get more coverage. Better
error handling on invalid inputs (reports the reason for the error)
Replace layout.fillrow with layout.filltable, but turns out that we
can probably just remove it anyway.
Add nilmdb.Timer for simple timing tests
Move some duplicated test suite helper functions into a new file,
test_helper.py
Add class to test_interval.py to match the others
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10661 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Change expected_daily_rows into rate_hz just for simplicity
nilmdb.layout.Parser now validates that timestamps are monotonically
increasing, and remembers the min and max seen.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10660 ddd99763-3ecb-0310-9145-efcb8ce7c51f
tolerance in equality comparisions, but it's a real pain to get all
the edge cases right, and it's not clear we actually need that
functionality at the moment. Skip it for now.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10659 ddd99763-3ecb-0310-9145-efcb8ce7c51f
