Commit Graph

118 Commits

Author SHA1 Message Date
10d58f6a47 More test coverage 2013-01-02 00:00:05 -05:00
c7c65b6542 Work around CherryPy bug #1200; related cleanups
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug.  Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not.  fun!
2013-01-01 23:03:53 -05:00
83b937c720 More Pytables -> bulkdata conversion 2012-12-31 17:22:30 -05:00
6f1e6fe232 Isolate all PyTables stuff to a single file.
This will make migrating to my own data storage engine easier.
2012-12-31 17:22:29 -05:00
b0d76312d1 Add must_close() decorator, use it in nilmdb
Warns at runtime if a class's close() method wasn't called before the
object was destroyed.
2012-12-31 17:21:19 -05:00
f355c73209 Refactor utility classes into nilmdb.utils subdir/namespace
There's some bug with the testing harness where placing e.g.
  from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du".  Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
24d4752bc3 Add LRU cache memoizing decorator for functions 2012-12-31 14:39:16 -05:00
9082cc9f44 Merging adjacent intervals is working now!
Adjust test expectations accordingly, since the number of intervals
they print out will now be smaller.
2012-12-12 19:25:27 -05:00
bf64a40472 Some misc test additions, interval optimizations. Still need adjacency test 2012-12-11 23:31:55 -05:00
32dbeebc09 More insertion checks. Need to get interval concatenation working. 2012-12-11 18:08:00 -05:00
66ddc79b15 Inserting works again, with proper end/start for paired blocks.
timeit.sh script works too!
2012-12-07 20:30:39 -05:00
5fecec2a4c Support deleting streams with new 'destroy' command 2012-12-04 22:15:00 -05:00
17c329fd6d Start to be a little more strict about how intervals are half-open. 2012-11-29 15:35:11 -05:00
c0f87db3c1 Converted rbtree, interval to Cython. Serious speedups! 2012-11-29 15:13:09 -05:00
f39567b2bc Speed updates 2012-11-29 01:35:01 -05:00
99ec0f4946 Converted rbtree.py to Cython
About 3x faster
2012-11-29 01:25:51 -05:00
f5c60f68dc Speed tests.
test_interval_speed is about O(n * log n), which is good -- but the
constants are high and it hits swap on a 4G machine for the 2**21
test.  Hopefully cython helps!
2012-11-29 01:00:54 -05:00
bdef0986d6 rbtree and interval tests fully pass now.
On to benchmarking...
2012-11-29 00:42:50 -05:00
c396c4dac8 rbtree tests complete 2012-11-29 00:07:49 -05:00
0b443f510b Filling out rbtree tests, search routines 2012-11-28 20:57:23 -05:00
66fa6f3824 Add rendering test 2012-11-28 18:34:51 -05:00
7211217f40 Working on getting the RBTree working. Intersections are busted.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11380 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-15 18:55:56 +00:00
d34b980516 RBTree seems generally OK now
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11379 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 20:10:43 +00:00
6aee52d980 Deletion is still broken. F.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11378 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 04:23:53 +00:00
090c8d5315 More progress
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11377 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 04:12:15 +00:00
1042ff9f4b add RBtree C++ example that I based this on; update tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11376 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 03:55:37 +00:00
f9dea53c24 Randomize order for the insertion test
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11358 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 23:50:23 +00:00
6278d32f7d Passes tests, but is slow
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11356 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 23:08:01 +00:00
991039903c Partial implementation of Interval and IntervalSet with a red-black
tree.

This is currently hitting an issue where it's considering the
intersection of [0,1] and [1,2] to be [1,1].  It matches the 
fully-closed definition of intervals, unlike before -- but might
cause issues.  Need to consider whether test case is correct.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11355 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 22:56:05 +00:00
ea3e92be3f Some prep work for merging adjacent insertions.
Doesn't actually merge them yet; need to change Interval
implementation to allow deletes.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11354 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 21:27:52 +00:00
407aedcd20 Can't represent 2112 on 32-bit systems
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10997 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-09 18:04:47 +00:00
bf8ff66c77 Fix timezone issues in cmdline test
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10996 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-09 18:03:18 +00:00
82f753efb1 disable TestLayoutSpeed for now
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10995 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-09 17:53:01 +00:00
e950794866 more info
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10991 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-08 22:00:54 +00:00
cb7c0cf83e Test roundtrip times for parser and formatter. Can optimize now
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10990 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-08 21:58:09 +00:00
33cb7031a3 Test float32_8 type names in addition to PrepData aliases
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10989 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-08 19:23:28 +00:00
5101522025 Tests pass with 100% coverage on non-cython modules
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10982 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-06 22:05:09 +00:00
5130ab7e6a Start reworking the layout types.
Current/old design has specific layouts: RawData, PrepData,
RawNotchedData.
Let's get rid of this entirely and switch to simpler data types that
are
just collections and counts of a single type.  We'll still use strings
to describe them, with format:

    type_count
  
where type is "uint16", "float32", or "float64", and count is an
integer.

nilmdb.layout.named() will parse these strings into the appropriate
handlers.  For compatibility:
  
    "RawData" == "uint16_6"
    "RawNotchedData" == "uint16_9"
    "PrepData" == "float32_8"


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10981 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-06 21:46:09 +00:00
ff4e934bef Add database size to "nilmtool.py info" output.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10968 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-07-24 21:52:38 +00:00
aefaac2405 Add "--count" option to extract
This lets us quickly count the number of matching rows, rather than
returning them.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10909 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-06-25 18:52:50 +00:00
97bec3b1ee Optimization that uses slices on the table rather than checking each
row individually, when extracting data.

Switch to using bisect module when doing the bisection, to lessen the
chance of errors.

Added syslog ability for timer module, for timing stuff deep inside
the server.

Make the chunked/non-chunked test just give a warning, rather than
failing the tests, for debugging purposes.  Alternate approach would
be to disable "die on error" for the tests.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10896 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-06-04 23:46:33 +00:00
27f8dcf06d Extract works now and we're back to 100% test coverage! It's slow,
though; need to figure out where the slowdown lies.

Add stream existence check to server's /intervals and /extract paths,
add tests for it.

Make start and end arguments optional for /extract, like /intervals

Move --quiet command line option to just the insert subcommand.
It's the only one that uses it right now, and otherwise it doesn't
show up in after a "nilmtool.py intervals --help".  Might revisit this
later if more commands start supporting --quiet.

Change cmdline/extract's write into a print, to keep the trailing
newline.

Fix lingering uses of Interval in nilmdb and change to DBInterval
instead.

Fix nilmdb interval bisection:
  - handle common case optimization correctly
  - db_endpos is always one after the last row, so use hi=db_endpos-1

Finish nlimdb stream_extract

Add a bunch of cmdline tests for extract, particularly testing border
cases around start/end.  Compares output to a set of files stored in
the tests/data dir.

Some more tests in test_client to get better coverage.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10893 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-30 18:24:36 +00:00
cb97ad3d2c Interval bisection to find the correct database row seems to work in
nilmdb now.  Needs testing, and nilmdb's stream_extract needs to be
finished


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10891 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-29 23:49:08 +00:00
8a7b3b5f95 Add DBInterval and associated tests. Clean up other interval tests as
well, and fix missing coverage on IntervalSet.str()


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10890 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 21:45:05 +00:00
11cc124019 Formatters done, not necessarily fast enough though
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10889 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 19:02:58 +00:00
e2daeb5e54 - Some updates to max_results handling on server side
- Flesh out tests for the new nilmdb.layout.Formatter
  Coverage doesn't handle the cython module, so this is just
  functional stuff, not necessarily complete.
  Still need to finish each Layout.format()

- Split out test_client_5_chunked from test_client_4_misc
  so it's easier to skip while debugging.  Turning off streaming
  lets us see tracebacks from within the server's content()
  functions.

- More work on stream/extract in cmdline, client, server, nilmdb.
  Still needs work on server side, but should be complete in nilmdb.

- Start nilmdb.layout.Formatter class


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10888 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 16:44:24 +00:00
cbc7c5125d WIP on getting stream extract to work.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10886 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-24 21:05:38 +00:00
27fd9d54f9 Streaming responses from the server weren't actually streaming. Fix
that.  This requires a bit of restructuring of server.py:intervals()
to allow us to properly report errors before beginning the stream.

Make the nilmdb.httpclient save a copy of HTTP header responses, and
add a test that the saved responses to verify that the
transfer-encoding is Chunked for the /stream/interval request.  This
should check that the above bug is fixed and doesn't show up again
if we switch to a different WSGI server, etc.

Tweak size estimates in nilmdb for /stream/interval


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10884 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 23:18:24 +00:00
e5e7ae9eda Fix bug that caused responses to always be truncated early
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10883 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:19:06 +00:00
315bc57ac3 More complete test coverage of nilmdb.httpclient, yay
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10882 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:00:01 +00:00