Commit Graph

39 Commits

Author SHA1 Message Date
c7c65b6542 Work around CherryPy bug #1200; related cleanups
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug.  Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not.  fun!
2013-01-01 23:03:53 -05:00
83b937c720 More Pytables -> bulkdata conversion 2012-12-31 17:22:30 -05:00
f355c73209 Refactor utility classes into nilmdb.utils subdir/namespace
There's some bug with the testing harness where placing e.g.
  from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du".  Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
9082cc9f44 Merging adjacent intervals is working now!
Adjust test expectations accordingly, since the number of intervals
they print out will now be smaller.
2012-12-12 19:25:27 -05:00
bf64a40472 Some misc test additions, interval optimizations. Still need adjacency test 2012-12-11 23:31:55 -05:00
5fecec2a4c Support deleting streams with new 'destroy' command 2012-12-04 22:15:00 -05:00
407aedcd20 Can't represent 2112 on 32-bit systems
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10997 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-09 18:04:47 +00:00
bf8ff66c77 Fix timezone issues in cmdline test
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10996 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-09 18:03:18 +00:00
5101522025 Tests pass with 100% coverage on non-cython modules
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10982 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-06 22:05:09 +00:00
5130ab7e6a Start reworking the layout types.
Current/old design has specific layouts: RawData, PrepData,
RawNotchedData.
Let's get rid of this entirely and switch to simpler data types that
are
just collections and counts of a single type.  We'll still use strings
to describe them, with format:

    type_count
  
where type is "uint16", "float32", or "float64", and count is an
integer.

nilmdb.layout.named() will parse these strings into the appropriate
handlers.  For compatibility:
  
    "RawData" == "uint16_6"
    "RawNotchedData" == "uint16_9"
    "PrepData" == "float32_8"


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10981 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-06 21:46:09 +00:00
ff4e934bef Add database size to "nilmtool.py info" output.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10968 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-07-24 21:52:38 +00:00
aefaac2405 Add "--count" option to extract
This lets us quickly count the number of matching rows, rather than
returning them.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10909 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-06-25 18:52:50 +00:00
97bec3b1ee Optimization that uses slices on the table rather than checking each
row individually, when extracting data.

Switch to using bisect module when doing the bisection, to lessen the
chance of errors.

Added syslog ability for timer module, for timing stuff deep inside
the server.

Make the chunked/non-chunked test just give a warning, rather than
failing the tests, for debugging purposes.  Alternate approach would
be to disable "die on error" for the tests.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10896 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-06-04 23:46:33 +00:00
27f8dcf06d Extract works now and we're back to 100% test coverage! It's slow,
though; need to figure out where the slowdown lies.

Add stream existence check to server's /intervals and /extract paths,
add tests for it.

Make start and end arguments optional for /extract, like /intervals

Move --quiet command line option to just the insert subcommand.
It's the only one that uses it right now, and otherwise it doesn't
show up in after a "nilmtool.py intervals --help".  Might revisit this
later if more commands start supporting --quiet.

Change cmdline/extract's write into a print, to keep the trailing
newline.

Fix lingering uses of Interval in nilmdb and change to DBInterval
instead.

Fix nilmdb interval bisection:
  - handle common case optimization correctly
  - db_endpos is always one after the last row, so use hi=db_endpos-1

Finish nlimdb stream_extract

Add a bunch of cmdline tests for extract, particularly testing border
cases around start/end.  Compares output to a set of files stored in
the tests/data dir.

Some more tests in test_client to get better coverage.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10893 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-30 18:24:36 +00:00
cb97ad3d2c Interval bisection to find the correct database row seems to work in
nilmdb now.  Needs testing, and nilmdb's stream_extract needs to be
finished


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10891 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-29 23:49:08 +00:00
e2daeb5e54 - Some updates to max_results handling on server side
- Flesh out tests for the new nilmdb.layout.Formatter
  Coverage doesn't handle the cython module, so this is just
  functional stuff, not necessarily complete.
  Still need to finish each Layout.format()

- Split out test_client_5_chunked from test_client_4_misc
  so it's easier to skip while debugging.  Turning off streaming
  lets us see tracebacks from within the server's content()
  functions.

- More work on stream/extract in cmdline, client, server, nilmdb.
  Still needs work on server side, but should be complete in nilmdb.

- Start nilmdb.layout.Formatter class


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10888 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 16:44:24 +00:00
cbc7c5125d WIP on getting stream extract to work.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10886 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-24 21:05:38 +00:00
e5e7ae9eda Fix bug that caused responses to always be truncated early
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10883 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:19:06 +00:00
315bc57ac3 More complete test coverage of nilmdb.httpclient, yay
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10882 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:00:01 +00:00
3f436e6dfd Use simplejson instead of json for performance
Misc cleanups

More work on tests to get more coverage


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10869 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-16 22:19:00 +00:00
d647ea7eee Cleanup handling of truncated intervals, more work on extract
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10853 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-11 18:31:11 +00:00
e3be1a1d8a Add IntervalSet.intersection(). This returns a generator that allows
us to look at just some of the intervals without having to reconstruct
an entire IntervalSet class -- which greatly reduces server load when
handling requests that cover large interval ranges.

Add Client.get and Client.put, analogous to getjson and putjson but
without parsing the result as json.

Add Client.stream_extract.  Still needs server side love.

Allow Cmdline subcommands to provide a return value that turns into
the exit code.

More work on cmdline.extract.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10851 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 21:54:29 +00:00
a05a026bc7 Make stream_intervals use a generator, so the client library handles
re-issuing the request when the output data gets truncated.  Works well.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10849 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 18:14:57 +00:00
4501da6edc Can query intervals now.
On the big database, the server takes a few seconds to figure out the
interval intersections.  Need to think about how to improve that --
the real key might be to start reducing the number of intervals we're
storing by combining them, potentially as they're inserted.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10838 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-07 22:32:02 +00:00
36045fe53b Remove --utc option from command line tool -- set TZ environment
variable instead now.  Adjust tests accordingly.

Start list --detail option, using stream/intervals request.
Frontend should be ready, backend needs implementation.

Put interval adding back into nilmdb:_add_interval so things work.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10833 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-04 22:36:27 +00:00
47245df9bd nilmdb now caches the intervals the first time a particular stream's
intervals are accessed, so it doesn't need to keep rebuilding them as
long as it's running. 


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10800 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-24 21:59:33 +00:00
7dfa288270 Can now insert data using command line tool. Time to benchmark etc.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10709 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 21:31:39 +00:00
e85acdd20c Fill out and fix parse_Time
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10705 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 19:35:14 +00:00
d725ed1771 date/time parsing, needs work
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10685 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 20:54:48 +00:00
d58a27e2bf Better coverage
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10684 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:37:05 +00:00
f3b0dfabea More work on commandline and commandline tests.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10682 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:25:09 +00:00
a3f444eb25 A lot more command line testing.
There'es some issue with tons of requests getting slowly blocked,
though...



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10678 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 22:34:01 +00:00
277b0c1d00 More command line tests.
Make nilmdb.cmdline a proper class
Fix various stuff
Add /dbpath command to get DB path


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10677 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 18:54:24 +00:00
7bba4a80d9 Work on command line client, and some improvements to server
handling of bad URLs


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10676 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-03 22:21:42 +00:00
cc4e3bdb76 Rename some nilmdb methods to be private
More work with nilmdb.stream_insert.  Checks for overlap between
parser output and database intervals, and actually inserts the
data into pytables.  Need to benchmark/figure out whether we can
use table.append rather than the row.append() nonsense -- it could
simplify things quite a bit.

Improve layout class and add tests to get more coverage.  Better
error handling on invalid inputs (reports the reason for the error)

Replace layout.fillrow with layout.filltable, but turns out that we
can probably just remove it anyway.

Add nilmdb.Timer for simple timing tests

Move some duplicated test suite helper functions into a new file,
test_helper.py

Add class to test_interval.py to match the others


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10661 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-29 21:43:05 +00:00
09340d543d Add some test prep data
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10641 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-23 14:56:52 +00:00
8cc1dff0b8 Cleanup
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10637 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-22 21:00:01 +00:00
1c4efb92c6 rename command line client nilmdb.py to nilmtool.py
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10622 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 21:46:18 +00:00
3e5d0ef87d rename
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10609 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 14:21:55 +00:00