Commit Graph

121 Commits

Author SHA1 Message Date
27fd9d54f9 Streaming responses from the server weren't actually streaming. Fix
that.  This requires a bit of restructuring of server.py:intervals()
to allow us to properly report errors before beginning the stream.

Make the nilmdb.httpclient save a copy of HTTP header responses, and
add a test that the saved responses to verify that the
transfer-encoding is Chunked for the /stream/interval request.  This
should check that the above bug is fixed and doesn't show up again
if we switch to a different WSGI server, etc.

Tweak size estimates in nilmdb for /stream/interval


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10884 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 23:18:24 +00:00
e5e7ae9eda Fix bug that caused responses to always be truncated early
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10883 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:19:06 +00:00
315bc57ac3 More complete test coverage of nilmdb.httpclient, yay
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10882 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:00:01 +00:00
e570800396 Add "iteratorizor", which transforms a function that uses callbacks
into a generator.  Needed to get pycurl's WRITEFUNCTION to give us a
generator instead.  It's a bit tricky and uses threads and queues, but
seems to work OK.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10880 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 01:00:12 +00:00
c327378373 Split MyCurl into a HTTPClient class instead
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10879 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-21 21:50:01 +00:00
aaffd61e4e rework MyCurl stuff
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10878 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-21 21:43:30 +00:00
3f436e6dfd Use simplejson instead of json for performance
Misc cleanups

More work on tests to get more coverage


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10869 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-16 22:19:00 +00:00
d647ea7eee Cleanup handling of truncated intervals, more work on extract
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10853 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-11 18:31:11 +00:00
e3be1a1d8a Add IntervalSet.intersection(). This returns a generator that allows
us to look at just some of the intervals without having to reconstruct
an entire IntervalSet class -- which greatly reduces server load when
handling requests that cover large interval ranges.

Add Client.get and Client.put, analogous to getjson and putjson but
without parsing the result as json.

Add Client.stream_extract.  Still needs server side love.

Allow Cmdline subcommands to provide a return value that turns into
the exit code.

More work on cmdline.extract.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10851 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 21:54:29 +00:00
a05a026bc7 Make stream_intervals use a generator, so the client library handles
re-issuing the request when the output data gets truncated.  Works well.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10849 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 18:14:57 +00:00
4501da6edc Can query intervals now.
On the big database, the server takes a few seconds to figure out the
interval intersections.  Need to think about how to improve that --
the real key might be to start reducing the number of intervals we're
storing by combining them, potentially as they're inserted.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10838 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-07 22:32:02 +00:00
36045fe53b Remove --utc option from command line tool -- set TZ environment
variable instead now.  Adjust tests accordingly.

Start list --detail option, using stream/intervals request.
Frontend should be ready, backend needs implementation.

Put interval adding back into nilmdb:_add_interval so things work.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10833 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-04 22:36:27 +00:00
7eef39d5fd notes
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10832 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-04 16:08:32 +00:00
47245df9bd nilmdb now caches the intervals the first time a particular stream's
intervals are accessed, so it doesn't need to keep rebuilding them as
long as it's running. 


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10800 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-24 21:59:33 +00:00
c07670ac3e Make the whole nilmdb.layout parser Cython, and rewrite the parsing
bits so it's hopefully quite a bit faster now.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10798 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-24 21:00:26 +00:00
ac32647fac Remove PyTables indexing support; it's not necessary since we're just
using pytables as a row-indexed data store.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10754 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 20:51:18 +00:00
b84ffddd50 New bxinterval implementation is at least linear -- not much better in
the TestIntervalSpeed versus the previous bisect version, but should
be better in general.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10748 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-12 22:39:16 +00:00
1531114677 fix typos in test_interval; test big numbers and floats
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10745 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-12 21:05:54 +00:00
055cfa12b2 Various work on improving intervals...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10738 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-11 22:05:27 +00:00
7dfa288270 Can now insert data using command line tool. Time to benchmark etc.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10709 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 21:31:39 +00:00
881b9a7bd1 add timestamped version
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10708 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 20:49:08 +00:00
e85acdd20c Fill out and fix parse_Time
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10705 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 19:35:14 +00:00
d725ed1771 date/time parsing, needs work
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10685 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 20:54:48 +00:00
d58a27e2bf Better coverage
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10684 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:37:05 +00:00
f3b0dfabea More work on commandline and commandline tests.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10682 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:25:09 +00:00
a3f444eb25 A lot more command line testing.
There'es some issue with tons of requests getting slowly blocked,
though...



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10678 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 22:34:01 +00:00
277b0c1d00 More command line tests.
Make nilmdb.cmdline a proper class
Fix various stuff
Add /dbpath command to get DB path


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10677 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 18:54:24 +00:00
7bba4a80d9 Work on command line client, and some improvements to server
handling of bad URLs


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10676 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-03 22:21:42 +00:00
e63ab23d20 stream_insert puts data into the database now. Fill out tests to
cover overlapping data, missing data, etc.

Consolidate nilmdb.client.MyCurl.getjson() and putjson().

Add __str__ for more compact representation of IntervalSet

Make stream_id not NULL in SQL database.

Format NilmDBError with name of exception for more clarity

Work around CherryPy issue 1138 by removing HTML escaping before
returning JSON error responses from the server.

Reached 100% test coverage again (woohoo)



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10665 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-31 04:15:29 +00:00
776279a4e6 Go back to generic parsing for most stuff now. Need to rework what's
most efficient here.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10664 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-30 22:15:04 +00:00
7a9012c3e9 work on improving layout parsing
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10663 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-30 21:29:22 +00:00
8c619cfde5 Minor cleanups
Remove nilmdb.layout.filltable


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10662 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-29 21:44:57 +00:00
cc4e3bdb76 Rename some nilmdb methods to be private
More work with nilmdb.stream_insert.  Checks for overlap between
parser output and database intervals, and actually inserts the
data into pytables.  Need to benchmark/figure out whether we can
use table.append rather than the row.append() nonsense -- it could
simplify things quite a bit.

Improve layout class and add tests to get more coverage.  Better
error handling on invalid inputs (reports the reason for the error)

Replace layout.fillrow with layout.filltable, but turns out that we
can probably just remove it anyway.

Add nilmdb.Timer for simple timing tests

Move some duplicated test suite helper functions into a new file,
test_helper.py

Add class to test_interval.py to match the others


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10661 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-29 21:43:05 +00:00
3dfd187710 Use instantiated classes for layouts.
Change expected_daily_rows into rate_hz just for simplicity

nilmdb.layout.Parser now validates that timestamps are monotonically
increasing, and remembers the min and max seen.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10660 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-29 16:46:34 +00:00
cf66eca42c Spent time trying to get Intervals and IntervalSets to handle a gap
tolerance in equality comparisions, but it's a real pain to get all
the edge cases right, and it's not clear we actually need that
functionality at the moment.  Skip it for now.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10659 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-28 19:25:02 +00:00
1046d0c47b more tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10657 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-28 00:22:44 +00:00
d940aabf66 Incomplete, need to finish interval/intervalset tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10656 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-27 23:19:08 +00:00
e1bf680d13 pass data to server
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10655 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 22:28:33 +00:00
a672119dd0 Minor cleanups for test passage
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10652 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 21:31:37 +00:00
17073905f7 Working on data parsing...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10648 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 19:54:04 +00:00
3b696e2a03 rework errors
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10647 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 18:06:15 +00:00
dd5658f987 Better handling on server side -- insert now reads the body.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10646 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-26 16:47:11 +00:00
54847b0710 Upload the data in the POST. Client side should be mostly good, for now
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10645 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-25 19:12:49 +00:00
b72e276eb8 More timestamper tests.
Add "end" parameter to TimestamperRate, to force the file to end
before a specific time.  Will be good for loading in old files and
knowing that things don't overlap.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10644 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-24 21:15:24 +00:00
0808ed5bd8 Add datetime_tz module from python-datetime-tz repository. This could
go away if it gets packaged in Debian etc.

Add nilmdb.timestamper classes

Start test_timestamper (needs work)

Number the functions in test_client so they run in the right order to
leave the database in a good state.

Fix some brokenness with module importing and namespaces



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10643 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-24 17:32:11 +00:00
ec25eac697 Misc cleanups, fix some pylint issues
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10642 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-23 15:50:33 +00:00
09340d543d Add some test prep data
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10641 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-23 14:56:52 +00:00
913883be3a Fix slow database stuff in testing by setting PRAGMA synchronous=OFF. Default is still FULL for production use, though
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10639 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-22 22:06:08 +00:00
8cc1dff0b8 Cleanup
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10637 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-22 21:00:01 +00:00
5b2d52b8bc Big rework of how errors are handled and propagated to the HTTP
response.  Now everything (status code, message, possibly tracebacks)
are passed to the client as JSON, so the client can display correct errors.
Nice!


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10630 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-22 00:14:13 +00:00
22ef82b59c More work on client tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10625 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-21 16:28:51 +00:00
a235c94c02 More work towards filling out the client...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10624 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 22:21:16 +00:00
1c4efb92c6 rename command line client nilmdb.py to nilmtool.py
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10622 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 21:46:18 +00:00
56a1770f45 client tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10611 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 14:26:29 +00:00
3e5d0ef87d rename
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10609 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 14:21:55 +00:00
0099a41fd8 Work on a client implementation that can survive testing
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10604 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-20 01:18:30 +00:00
b1baacf272 cleanup
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10600 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-19 21:36:24 +00:00
8afaf8f329 Improve insert strutcure
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10562 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-14 20:29:30 +00:00
610860c379 Update
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10556 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-13 22:12:30 +00:00
1e1c7fa9c4 Create StreamException class in NilmDB, which can get caught and
handled by the HTTP server.

Add /stream/get_metadata HTTP request.

Fill out test_stream_metadata

Add runserver.py script to just run the server from the command line,
outside of the testing environment.  Add necessary hooks within
nilmdb/server.py to allow this to happen.

Metadata operations get stream_id first and report error on missing
stream, instead of returning an empty dict.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10443 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-02-20 05:52:33 +00:00
398d382dac Stuff goes into the SQL database now.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10427 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-02-17 00:32:36 +00:00
c5ec07a661 Start working towards mixed sql/pytables db
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10373 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-02-08 19:28:16 +00:00
6812a28131 Make stream_list return number of rows.
Update layouts to include expected number of rows per day -- this is
used (as strongly suggested by the PyTables docs) to form a rough
estimate of the number of rows in a table during CreateTable.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10340 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-30 20:25:11 +00:00
3b9d84d689 Adjust how layouts are managed: now we pass strings representing the
layouts, rather than direct dict objects.  This allows us to go
backwards and match the description of tables in the database with the
layouts defined within nilmdb.layout.

Fix some big issues with the nilmdb.server and startup errors like an
invalid port number.  We have to catch os._exit and report the error
ourselves.

Add some timeouts and otherwise clean up tests.

Make NilmDB track and complain about files that weren't closed before
the object is destroyed.

Update stream_list and /stream/list to return the layout corresponding
to each table.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10338 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-28 00:36:23 +00:00
b5e3b22558 Fixups, and add serializer to NilmDB access from CherryPy
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10336 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-27 19:17:09 +00:00
9f4d88cc8f Added serialization wrapper. This will let us wrap the NilmDB object
and ensure that all function calls happen from within the context of a
single thread.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10334 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-27 01:27:21 +00:00
acdea5f00b Integrated CherryPy server into tree
Added testing framework for cherrypy server.
Added tests for basic server functionality.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10331 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-26 00:13:44 +00:00
6c9cf198d7 indexing
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10330 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-23 21:07:59 +00:00
35a801d997 Don't include the test database
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10328 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-23 20:38:01 +00:00
a3634c468f Misc
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10327 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-23 00:59:17 +00:00
dfaeddefa2 Mics
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10326 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-01-23 00:40:48 +00:00