Commit Graph

45 Commits

Author SHA1 Message Date
e2464efc12 Test everything; remove debugging 2013-01-01 23:46:54 -05:00
f41ff0a6e8 Inserting bulk data is essentially done, not tested 2013-01-01 21:04:35 -05:00
487298986e More work towards bulkdata 2012-12-31 18:44:57 -05:00
b0d76312d1 Add must_close() decorator, use it in nilmdb
Warns at runtime if a class's close() method wasn't called before the
object was destroyed.
2012-12-31 17:21:19 -05:00
f355c73209 Refactor utility classes into nilmdb.utils subdir/namespace
There's some bug with the testing harness where placing e.g.
  from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du".  Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
32dbeebc09 More insertion checks. Need to get interval concatenation working. 2012-12-11 18:08:00 -05:00
66ddc79b15 Inserting works again, with proper end/start for paired blocks.
timeit.sh script works too!
2012-12-07 20:30:39 -05:00
17c329fd6d Start to be a little more strict about how intervals are half-open. 2012-11-29 15:35:11 -05:00
99ec0f4946 Converted rbtree.py to Cython
About 3x faster
2012-11-29 01:25:51 -05:00
bdef0986d6 rbtree and interval tests fully pass now.
On to benchmarking...
2012-11-29 00:42:50 -05:00
0b443f510b Filling out rbtree tests, search routines 2012-11-28 20:57:23 -05:00
7211217f40 Working on getting the RBTree working. Intersections are busted.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11380 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-15 18:55:56 +00:00
6aee52d980 Deletion is still broken. F.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11378 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 04:23:53 +00:00
090c8d5315 More progress
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11377 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 04:12:15 +00:00
1042ff9f4b add RBtree C++ example that I based this on; update tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11376 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 03:55:37 +00:00
ea3e92be3f Some prep work for merging adjacent insertions.
Doesn't actually merge them yet; need to change Interval
implementation to allow deletes.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11354 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 21:27:52 +00:00
82f753efb1 disable TestLayoutSpeed for now
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10995 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-09 17:53:01 +00:00
cb7c0cf83e Test roundtrip times for parser and formatter. Can optimize now
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10990 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-08-08 21:58:09 +00:00
8a7b3b5f95 Add DBInterval and associated tests. Clean up other interval tests as
well, and fix missing coverage on IntervalSet.str()


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10890 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 21:45:05 +00:00
e2daeb5e54 - Some updates to max_results handling on server side
- Flesh out tests for the new nilmdb.layout.Formatter
  Coverage doesn't handle the cython module, so this is just
  functional stuff, not necessarily complete.
  Still need to finish each Layout.format()

- Split out test_client_5_chunked from test_client_4_misc
  so it's easier to skip while debugging.  Turning off streaming
  lets us see tracebacks from within the server's content()
  functions.

- More work on stream/extract in cmdline, client, server, nilmdb.
  Still needs work on server side, but should be complete in nilmdb.

- Start nilmdb.layout.Formatter class


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10888 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 16:44:24 +00:00
3b0b9175d6 For stream_interval, make the server handle sending multiple requests
to the database, not the client.  The server now maintains the open
HTTP connection and sends a continuous streaming reply to the GET
request.

HTTP client side uses an Iteratorizer to turn the curl.perform()
callback into an interator, and returns an iterator that yields
individual lines to the caller rather than buffering up all the data
at once.  Should still be able to handle errors etc.

Server changed to return a "streaming JSON" instance for the 
/stream/interval requests.  This is just a series of independent
JSON documents (one per interval), separated by newlines.

Adjust nilmdb's max_results a bit.  Now, multiple requests only exist
between the server <-> nilmdb threads, and they exist just to avoid 
blocking the nilmdb thread by any one server thread for too long.
So adjust the size accordingly to match the fact that this is non-json
encoded data.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10881 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 19:08:59 +00:00
4501da6edc Can query intervals now.
On the big database, the server takes a few seconds to figure out the
interval intersections.  Need to think about how to improve that --
the real key might be to start reducing the number of intervals we're
storing by combining them, potentially as they're inserted.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10838 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-07 22:32:02 +00:00
36045fe53b Remove --utc option from command line tool -- set TZ environment
variable instead now.  Adjust tests accordingly.

Start list --detail option, using stream/intervals request.
Frontend should be ready, backend needs implementation.

Put interval adding back into nilmdb:_add_interval so things work.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10833 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-04 22:36:27 +00:00
ac32647fac Remove PyTables indexing support; it's not necessary since we're just
using pytables as a row-indexed data store.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10754 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 20:51:18 +00:00
055cfa12b2 Various work on improving intervals...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10738 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-11 22:05:27 +00:00
7dfa288270 Can now insert data using command line tool. Time to benchmark etc.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10709 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 21:31:39 +00:00
d725ed1771 date/time parsing, needs work
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10685 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 20:54:48 +00:00
f3b0dfabea More work on commandline and commandline tests.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10682 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:25:09 +00:00
277b0c1d00 More command line tests.
Make nilmdb.cmdline a proper class
Fix various stuff
Add /dbpath command to get DB path


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10677 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 18:54:24 +00:00
8c619cfde5 Minor cleanups
Remove nilmdb.layout.filltable


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10662 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-29 21:44:57 +00:00
cc4e3bdb76 Rename some nilmdb methods to be private
More work with nilmdb.stream_insert.  Checks for overlap between
parser output and database intervals, and actually inserts the
data into pytables.  Need to benchmark/figure out whether we can
use table.append rather than the row.append() nonsense -- it could
simplify things quite a bit.

Improve layout class and add tests to get more coverage.  Better
error handling on invalid inputs (reports the reason for the error)

Replace layout.fillrow with layout.filltable, but turns out that we
can probably just remove it anyway.

Add nilmdb.Timer for simple timing tests

Move some duplicated test suite helper functions into a new file,
test_helper.py

Add class to test_interval.py to match the others


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10661 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-29 21:43:05 +00:00
cf66eca42c Spent time trying to get Intervals and IntervalSets to handle a gap
tolerance in equality comparisions, but it's a real pain to get all
the edge cases right, and it's not clear we actually need that
functionality at the moment.  Skip it for now.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10659 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-28 19:25:02 +00:00
d940aabf66 Incomplete, need to finish interval/intervalset tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10656 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-27 23:19:08 +00:00
e1bf680d13 pass data to server
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10655 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 22:28:33 +00:00
a672119dd0 Minor cleanups for test passage
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10652 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 21:31:37 +00:00
dd5658f987 Better handling on server side -- insert now reads the body.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10646 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 16:47:11 +00:00
b72e276eb8 More timestamper tests.
Add "end" parameter to TimestamperRate, to force the file to end
before a specific time.  Will be good for loading in old files and
knowing that things don't overlap.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10644 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-24 21:15:24 +00:00
0808ed5bd8 Add datetime_tz module from python-datetime-tz repository. This could
go away if it gets packaged in Debian etc.

Add nilmdb.timestamper classes

Start test_timestamper (needs work)

Number the functions in test_client so they run in the right order to
leave the database in a good state.

Fix some brokenness with module importing and namespaces



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10643 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-24 17:32:11 +00:00
913883be3a Fix slow database stuff in testing by setting PRAGMA synchronous=OFF. Default is still FULL for production use, though
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10639 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-22 22:06:08 +00:00
8cc1dff0b8 Cleanup
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10637 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-22 21:00:01 +00:00
22ef82b59c More work on client tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10625 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-21 16:28:51 +00:00
1c4efb92c6 rename command line client nilmdb.py to nilmtool.py
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10622 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 21:46:18 +00:00
0099a41fd8 Work on a client implementation that can survive testing
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10604 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 01:18:30 +00:00
ba66668fff playing with coverage a bit
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10444 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-02-20 06:09:42 +00:00
22c8b08523 Misc
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@9297 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2011-02-24 22:48:09 +00:00