that. This requires a bit of restructuring of server.py:intervals()
to allow us to properly report errors before beginning the stream.
Make the nilmdb.httpclient save a copy of HTTP header responses, and
add a test that the saved responses to verify that the
transfer-encoding is Chunked for the /stream/interval request. This
should check that the above bug is fixed and doesn't show up again
if we switch to a different WSGI server, etc.
Tweak size estimates in nilmdb for /stream/interval
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10884 ddd99763-3ecb-0310-9145-efcb8ce7c51f
into a generator. Needed to get pycurl's WRITEFUNCTION to give us a
generator instead. It's a bit tricky and uses threads and queues, but
seems to work OK.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10880 ddd99763-3ecb-0310-9145-efcb8ce7c51f
us to look at just some of the intervals without having to reconstruct
an entire IntervalSet class -- which greatly reduces server load when
handling requests that cover large interval ranges.
Add Client.get and Client.put, analogous to getjson and putjson but
without parsing the result as json.
Add Client.stream_extract. Still needs server side love.
Allow Cmdline subcommands to provide a return value that turns into
the exit code.
More work on cmdline.extract.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10851 ddd99763-3ecb-0310-9145-efcb8ce7c51f
On the big database, the server takes a few seconds to figure out the
interval intersections. Need to think about how to improve that --
the real key might be to start reducing the number of intervals we're
storing by combining them, potentially as they're inserted.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10838 ddd99763-3ecb-0310-9145-efcb8ce7c51f
variable instead now. Adjust tests accordingly.
Start list --detail option, using stream/intervals request.
Frontend should be ready, backend needs implementation.
Put interval adding back into nilmdb:_add_interval so things work.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10833 ddd99763-3ecb-0310-9145-efcb8ce7c51f
intervals are accessed, so it doesn't need to keep rebuilding them as
long as it's running.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10800 ddd99763-3ecb-0310-9145-efcb8ce7c51f
the TestIntervalSpeed versus the previous bisect version, but should
be better in general.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10748 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Make nilmdb.cmdline a proper class
Fix various stuff
Add /dbpath command to get DB path
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10677 ddd99763-3ecb-0310-9145-efcb8ce7c51f
cover overlapping data, missing data, etc.
Consolidate nilmdb.client.MyCurl.getjson() and putjson().
Add __str__ for more compact representation of IntervalSet
Make stream_id not NULL in SQL database.
Format NilmDBError with name of exception for more clarity
Work around CherryPy issue 1138 by removing HTML escaping before
returning JSON error responses from the server.
Reached 100% test coverage again (woohoo)
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10665 ddd99763-3ecb-0310-9145-efcb8ce7c51f
More work with nilmdb.stream_insert. Checks for overlap between
parser output and database intervals, and actually inserts the
data into pytables. Need to benchmark/figure out whether we can
use table.append rather than the row.append() nonsense -- it could
simplify things quite a bit.
Improve layout class and add tests to get more coverage. Better
error handling on invalid inputs (reports the reason for the error)
Replace layout.fillrow with layout.filltable, but turns out that we
can probably just remove it anyway.
Add nilmdb.Timer for simple timing tests
Move some duplicated test suite helper functions into a new file,
test_helper.py
Add class to test_interval.py to match the others
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10661 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Change expected_daily_rows into rate_hz just for simplicity
nilmdb.layout.Parser now validates that timestamps are monotonically
increasing, and remembers the min and max seen.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10660 ddd99763-3ecb-0310-9145-efcb8ce7c51f
tolerance in equality comparisions, but it's a real pain to get all
the edge cases right, and it's not clear we actually need that
functionality at the moment. Skip it for now.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10659 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Add "end" parameter to TimestamperRate, to force the file to end
before a specific time. Will be good for loading in old files and
knowing that things don't overlap.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10644 ddd99763-3ecb-0310-9145-efcb8ce7c51f
go away if it gets packaged in Debian etc.
Add nilmdb.timestamper classes
Start test_timestamper (needs work)
Number the functions in test_client so they run in the right order to
leave the database in a good state.
Fix some brokenness with module importing and namespaces
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10643 ddd99763-3ecb-0310-9145-efcb8ce7c51f
response. Now everything (status code, message, possibly tracebacks)
are passed to the client as JSON, so the client can display correct errors.
Nice!
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10630 ddd99763-3ecb-0310-9145-efcb8ce7c51f
handled by the HTTP server.
Add /stream/get_metadata HTTP request.
Fill out test_stream_metadata
Add runserver.py script to just run the server from the command line,
outside of the testing environment. Add necessary hooks within
nilmdb/server.py to allow this to happen.
Metadata operations get stream_id first and report error on missing
stream, instead of returning an empty dict.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10443 ddd99763-3ecb-0310-9145-efcb8ce7c51f
Update layouts to include expected number of rows per day -- this is
used (as strongly suggested by the PyTables docs) to form a rough
estimate of the number of rows in a table during CreateTable.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10340 ddd99763-3ecb-0310-9145-efcb8ce7c51f
layouts, rather than direct dict objects. This allows us to go
backwards and match the description of tables in the database with the
layouts defined within nilmdb.layout.
Fix some big issues with the nilmdb.server and startup errors like an
invalid port number. We have to catch os._exit and report the error
ourselves.
Add some timeouts and otherwise clean up tests.
Make NilmDB track and complain about files that weren't closed before
the object is destroyed.
Update stream_list and /stream/list to return the layout corresponding
to each table.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb-new@10338 ddd99763-3ecb-0310-9145-efcb8ce7c51f
