Commit Graph

243 Commits

Author SHA1 Message Date
9ce0f69dff Add "--delete" option to "nilmtool metadata" tool
This is the same as "--update" with an empty string as the value.
2013-04-02 16:07:28 -04:00
ebccfb3531 Fix stream renaming when the new path is a parent of the old 2013-04-01 19:25:17 -04:00
173121ca87 Switch URL to one that should definitely not resolve 2013-03-30 17:31:35 -04:00
26bab031bd Add StreamInserter.send() to trigger intermediate block send 2013-03-30 17:30:43 -04:00
f8ca8d31e6 Remove Iteratorizer, as it's no longer needed 2013-03-24 21:31:03 -04:00
ed89d803f0 Remove aplotter code 2013-03-24 21:29:09 -04:00
3d24092cd2 Replace bare 'except:' with 'except: Exception'
Otherwise we might inadvertently catch SystemExit or KeyboardExit or
something we don't want to catch.
2013-03-24 21:28:01 -04:00
c0d450d39e Add locking mechanism to avoid multiple servers on one DB 2013-03-24 21:20:20 -04:00
e790bb9e8a Fix test failure when tests are run as root 2013-03-21 14:33:02 -04:00
89be6f5931 Add option to include interval start/end markup on extract
When enabled, lines like "# interval-start 1234567890123456" and "#
interval-end 1234567890123456" will be added to the data output.  Note
that there may be an "interval-end" timestamp followed by an identical
"interval-start" timestamp, if the response at the nilmdb level was
split up into multiple chunks.

In general, assume contiguous data if previous_interval_end ==
new_interval_start.
2013-03-19 14:23:33 -04:00
4cdef3285d Destroy now requires that all data has been previously removed.
Added new flag "-R" to command line to perform an automatic removal.
This should be the last of the ways in which a single command could
block the nilmdb thread for a long time.
2013-03-18 19:39:03 -04:00
bcd82c4d59 Limit the number of rows removed per call to nilmdb.stream_remove
Server class will retry as needed, as with stream_extract and
stream_intervals.
2013-03-18 18:22:45 -04:00
2d72891162 Accept "min" and "max" as timestamps on command line 2013-03-18 18:19:24 -04:00
0ef71c193b Remove layout.pyx, since rocket replaced it 2013-03-15 22:32:40 -04:00
22274550ab Test python version of Interval too 2013-03-15 21:37:03 -04:00
4f06d6ae68 Move Interval set_difference inside nilmdb.utils for clients
Clients might need to to Interval math too, so move a simple Interval
class and start putting helpers in there.
2013-03-15 21:37:03 -04:00
c54d8041c3 Update design docs 2013-03-15 21:07:01 -04:00
049375d30e Fill out test coverage 2013-03-15 18:08:21 -04:00
88eb0123f5 Add test for Table.__getitem__ indexing 2013-03-15 18:08:21 -04:00
28e72fd53e Remove Table.__getitem__; used only by tests 2013-03-15 18:08:21 -04:00
fb4f4519ff Clean up and simplify Table.get_*, including __getitem__ 2013-03-15 18:08:20 -04:00
90b96799ac Bulk of the switch to int64 microsecond timestamps, including test data. 2013-03-15 15:08:58 -04:00
56679ad770 Move more datetime_tz calls into common code 2013-03-15 15:08:58 -04:00
b5541722c2 Continue moving time-handling code into nilmdb.utils.time 2013-03-15 15:08:58 -04:00
c7a712d8d8 Partial test for rounding issues 2013-03-15 15:08:57 -04:00
b9f0b35bbe Stream renaming support, and comprehensive tests
Implemented in command line, client, server, nilmdb, bulkdata
2013-03-14 11:02:30 -04:00
b1b09f8cd0 Strengthen checks when creating paths, fix some bugs, and add tests 2013-03-13 17:45:47 -04:00
d467df7980 Add specific error for creating a path that already exists 2013-03-13 10:14:28 -04:00
b77f07a4cd Fix reporting of parsing errors with malformed data
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
2013-03-12 16:44:36 -04:00
83bc5bc775 Make rocket/bulkdata errors include column number and the bad data
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
2013-03-12 16:10:00 -04:00
7eca587fdf Add 'nilmtool intervals' command, with --diff option
Can show the set-difference between the interval ranges in two
streams.
2013-03-11 17:07:26 -04:00
1d61d61a81 Add interval.set_difference function and associated tests 2013-03-11 15:40:50 -04:00
8e79998e95 Tune sqlite to use write-ahead-logging
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
2013-03-11 15:13:43 -04:00
9f914598c2 Make /stream/list give some more extended info, like row count
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
2013-03-11 15:13:43 -04:00
232a3876c2 Clean up imports to separate client and server more.
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
2013-03-11 15:13:42 -04:00
1c27dd72d6 Fill out client tests and fix various bugs
Fixes various corner cases and other bugs regarding lines with
comments, having data but no endpoints, etc.
2013-03-08 12:36:17 -05:00
0fc092779d Big rework of stream_insert_context and places that use it.
Things are now block-focused, rather than line-focused.  This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
2013-03-07 20:30:11 -05:00
92724d10ba Rework 'nilmtool insert' and some client stuff to speed up inserting data
Still needs work.
2013-03-06 20:49:14 -05:00
1d7acbf916 Remove null timestamper, speed up insert --none a tiny bit 2013-03-06 20:46:51 -05:00
ea3ea487bc Merge branch 'rocket-insert'
Conflicts:
	nilmdb/server/bulkdata.py
	nilmdb/server/server.py
	nilmdb/utils/__init__.py
2013-03-06 20:46:04 -05:00
69ad8c4842 Merge branch 'rocket' 2013-03-06 20:38:02 -05:00
65f09f793c When re-raising exceptions in the server, preserve original tracebacks 2013-03-05 21:48:40 -05:00
84e21ff467 Move ASCII data parsing from the server to the rocket interface.
The server buffers the string and passes it to nilmdb.  Nilmdb passes
the string to bulkdata.  Bulkdata uses the rocket interface to parse
it in chunks, as necessary.  Everything gets passed back up and
everyone is happy.

Currently, only pyrocket implements append_string.
2013-03-05 17:51:17 -05:00
11b228f77a Convert times to microsecond precision strings more consistently.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
2013-03-05 17:07:39 -05:00
454e561d69 Verify that metadata values are numbers or strings 2013-03-05 13:22:17 -05:00
58c0ae72f6 Support application/json POST bodies as well as x-www-form-urlencoded 2013-03-05 11:54:29 -05:00
c6a0e6e96f More complete CORS handling, including preflight requests (hopefully) 2013-03-04 15:40:35 -05:00
79755dc624 Fix Allow: header by switching to cherrypy's built in tools.allow().
Replaces custom tools.allow_methods which didn't return the Allow: header.
2013-03-04 14:08:37 -05:00
0d372fb878 Modify old formatter to match rocket's formatting style 2013-03-03 21:50:29 -05:00
0b75da7a8f Normalize the floating point formats to %.6e and %.16e
This is mostly a matter of taste, but it matches more closely with the
old way that prep did it, and it's more consistent.  It should roughly
match the available precision of floats and doubles.
2013-03-03 21:43:04 -05:00