Commit Graph

443 Commits

Author SHA1 Message Date
b32edb1ed6 Undo last commit
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10876 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-18 21:49:52 +00:00
624980a47b Started moving to python-requests so we can do a get() with an
interator on the response, but it turns out that it's really crappy
with regards to socket closing, so I'll revert this in a moment.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10875 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-18 21:49:29 +00:00
3f436e6dfd Use simplejson instead of json for performance
Misc cleanups

More work on tests to get more coverage


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10869 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-16 22:19:00 +00:00
d647ea7eee Cleanup handling of truncated intervals, more work on extract
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10853 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-11 18:31:11 +00:00
9ff30f8c1d ignore generated files
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10852 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-11 15:20:52 +00:00
e3be1a1d8a Add IntervalSet.intersection(). This returns a generator that allows
us to look at just some of the intervals without having to reconstruct
an entire IntervalSet class -- which greatly reduces server load when
handling requests that cover large interval ranges.

Add Client.get and Client.put, analogous to getjson and putjson but
without parsing the result as json.

Add Client.stream_extract.  Still needs server side love.

Allow Cmdline subcommands to provide a return value that turns into
the exit code.

More work on cmdline.extract.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10851 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 21:54:29 +00:00
f63e58f2d9 Cleanup cmdline by splitting it into multiple files. Essentially the
individual subcommands are still treated as being part of the same
class (and use "self" to refer to the Cmdline class), but they're
different modules now.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10850 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 20:35:51 +00:00
a05a026bc7 Make stream_intervals use a generator, so the client library handles
re-issuing the request when the output data gets truncated.  Works well.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10849 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-10 18:14:57 +00:00
1d875b1f1f Start extract command in cmdline.py
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10840 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-08 21:27:56 +00:00
f4f2493b59 Some small speed improvements. Intervals are really a pain; need to
rework how this all works together, but will probably move on to
extraction now.

Update runserver.py with some options for profiling, port, etc.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10839 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-08 17:40:11 +00:00
4501da6edc Can query intervals now.
On the big database, the server takes a few seconds to figure out the
interval intersections.  Need to think about how to improve that --
the real key might be to start reducing the number of intervals we're
storing by combining them, potentially as they're inserted.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10838 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-07 22:32:02 +00:00
36045fe53b Remove --utc option from command line tool -- set TZ environment
variable instead now.  Adjust tests accordingly.

Start list --detail option, using stream/intervals request.
Frontend should be ready, backend needs implementation.

Put interval adding back into nilmdb:_add_interval so things work.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10833 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-04 22:36:27 +00:00
7eef39d5fd notes
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10832 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-04 16:08:32 +00:00
de7f78a43b which are not necessary anymore
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10831 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 21:14:58 +00:00
fab3567d74 instructions for blist
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10830 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 21:14:46 +00:00
6d6514d5c3 Go back to bxinterval. The blist version was slower.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10829 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 21:14:24 +00:00
b67fe79e47 Use blist instead of bxtree
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10828 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 21:13:55 +00:00
064b4bf715 notes
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10827 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 19:18:07 +00:00
e08be4c2a8 misc
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10826 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 19:17:48 +00:00
0276810776 More timing tests
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10825 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 19:17:35 +00:00
565d0e98a9 Remove trailing spaces
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10824 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-03 19:17:23 +00:00
47245df9bd nilmdb now caches the intervals the first time a particular stream's
intervals are accessed, so it doesn't need to keep rebuilding them as
long as it's running. 


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10800 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-24 21:59:33 +00:00
c07670ac3e Make the whole nilmdb.layout parser Cython, and rewrite the parsing
bits so it's hopefully quite a bit faster now.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10798 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-24 21:00:26 +00:00
37b4376b4c design notes
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10756 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 21:00:33 +00:00
3b52ecafa3 Clean up unused code
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10755 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 20:53:03 +00:00
ac32647fac Remove PyTables indexing support; it's not necessary since we're just
using pytables as a row-indexed data store.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10754 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 20:51:18 +00:00
4e143dfa18 docs
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10753 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 20:47:56 +00:00
b84ffddd50 New bxinterval implementation is at least linear -- not much better in
the TestIntervalSpeed versus the previous bisect version, but should
be better in general.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10748 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-12 22:39:16 +00:00
1531114677 fix typos in test_interval; test big numbers and floats
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10745 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-12 21:05:54 +00:00
ab4c1f0925 Update bxintersect to use doubles, not ints. Cython is easy!
Haven't updated IntervalSet, so current performance still sucks:
32: 805.000000 μs
64: 1534.000000 μs
128: 3158.000000 μs
256: 6430.000000 μs
512: 12972.000000 μs
1024: 26760.000000 μs
2048: 64091.000000 μs
4096: 108568.000000 μs
8192: 219389.000000 μs
16384: 471753.000000 μs
32768: 923798.000000 μs
65536: 1929795.000000 μs
131072: 3873633.000000 μs
262144: 8639542.000000 μs
524288: 17132838.000000 μs


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10744 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-12 20:26:33 +00:00
f7149e48e8 Add intersection.pyx from bx-python
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10743 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-12 20:16:46 +00:00
055cfa12b2 Various work on improving intervals...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10738 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-11 22:05:27 +00:00
5cb03cd6ef Update design.md. Add commit after each insert
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10729 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-10 20:52:59 +00:00
70bcc6d2b3 design notes
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10728 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-10 19:58:16 +00:00
43d3daa840 Testing
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10712 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 22:46:04 +00:00
7dfa288270 Can now insert data using command line tool. Time to benchmark etc.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10709 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 21:31:39 +00:00
881b9a7bd1 add timestamped version
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10708 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 20:49:08 +00:00
e85acdd20c Fill out and fix parse_Time
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10705 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-09 19:35:14 +00:00
d725ed1771 date/time parsing, needs work
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10685 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 20:54:48 +00:00
d58a27e2bf Better coverage
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10684 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:37:05 +00:00
f3b0dfabea More work on commandline and commandline tests.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10682 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:25:09 +00:00
cccaec326a Need to figure out a better way to handle this
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10679 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 22:57:12 +00:00
a3f444eb25 A lot more command line testing.
There'es some issue with tons of requests getting slowly blocked,
though...



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10678 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 22:34:01 +00:00
277b0c1d00 More command line tests.
Make nilmdb.cmdline a proper class
Fix various stuff
Add /dbpath command to get DB path


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10677 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-04 18:54:24 +00:00
7bba4a80d9 Work on command line client, and some improvements to server
handling of bad URLs


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10676 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-03 22:21:42 +00:00
f29d38d9d9 More work on nilmtool
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10674 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-03 19:43:37 +00:00
b4a0288a39 work on command line
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10673 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-03 18:01:56 +00:00
e63ab23d20 stream_insert puts data into the database now. Fill out tests to
cover overlapping data, missing data, etc.

Consolidate nilmdb.client.MyCurl.getjson() and putjson().

Add __str__ for more compact representation of IntervalSet

Make stream_id not NULL in SQL database.

Format NilmDBError with name of exception for more clarity

Work around CherryPy issue 1138 by removing HTML escaping before
returning JSON error responses from the server.

Reached 100% test coverage again (woohoo)



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10665 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-31 04:15:29 +00:00
776279a4e6 Go back to generic parsing for most stuff now. Need to rework what's
most efficient here.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10664 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-30 22:15:04 +00:00
7a9012c3e9 work on improving layout parsing
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10663 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-03-30 21:29:22 +00:00