Commit Graph

97 Commits

Author SHA1 Message Date
f2a48bdb2a Test binary extract; fix bugs 2013-04-11 13:24:11 -04:00
416a499866 Support wildcards for destroy 2013-04-10 16:23:07 -04:00
0104c8edd9 nilmtool remove: allow wildcards and multiple paths 2013-04-10 15:27:46 -04:00
83d022016c nilmtool list: add new --layout option to show layouts 2013-04-10 14:58:44 -04:00
43b740ecaa nilmtool list: remove old -p parameter 2013-04-10 14:48:23 -04:00
230ec72609 Fix timestamp display issues with --annotate 2013-04-09 18:19:32 -04:00
9ce0f69dff Add "--delete" option to "nilmtool metadata" tool
This is the same as "--update" with an empty string as the value.
2013-04-02 16:07:28 -04:00
ebccfb3531 Fix stream renaming when the new path is a parent of the old 2013-04-01 19:25:17 -04:00
89be6f5931 Add option to include interval start/end markup on extract
When enabled, lines like "# interval-start 1234567890123456" and "#
interval-end 1234567890123456" will be added to the data output.  Note
that there may be an "interval-end" timestamp followed by an identical
"interval-start" timestamp, if the response at the nilmdb level was
split up into multiple chunks.

In general, assume contiguous data if previous_interval_end ==
new_interval_start.
2013-03-19 14:23:33 -04:00
4cdef3285d Destroy now requires that all data has been previously removed.
Added new flag "-R" to command line to perform an automatic removal.
This should be the last of the ways in which a single command could
block the nilmdb thread for a long time.
2013-03-18 19:39:03 -04:00
bcd82c4d59 Limit the number of rows removed per call to nilmdb.stream_remove
Server class will retry as needed, as with stream_extract and
stream_intervals.
2013-03-18 18:22:45 -04:00
2d72891162 Accept "min" and "max" as timestamps on command line 2013-03-18 18:19:24 -04:00
049375d30e Fill out test coverage 2013-03-15 18:08:21 -04:00
90b96799ac Bulk of the switch to int64 microsecond timestamps, including test data. 2013-03-15 15:08:58 -04:00
b5541722c2 Continue moving time-handling code into nilmdb.utils.time 2013-03-15 15:08:58 -04:00
b9f0b35bbe Stream renaming support, and comprehensive tests
Implemented in command line, client, server, nilmdb, bulkdata
2013-03-14 11:02:30 -04:00
b1b09f8cd0 Strengthen checks when creating paths, fix some bugs, and add tests 2013-03-13 17:45:47 -04:00
d467df7980 Add specific error for creating a path that already exists 2013-03-13 10:14:28 -04:00
b77f07a4cd Fix reporting of parsing errors with malformed data
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
2013-03-12 16:44:36 -04:00
83bc5bc775 Make rocket/bulkdata errors include column number and the bad data
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
2013-03-12 16:10:00 -04:00
7eca587fdf Add 'nilmtool intervals' command, with --diff option
Can show the set-difference between the interval ranges in two
streams.
2013-03-11 17:07:26 -04:00
8e79998e95 Tune sqlite to use write-ahead-logging
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
2013-03-11 15:13:43 -04:00
9f914598c2 Make /stream/list give some more extended info, like row count
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
2013-03-11 15:13:43 -04:00
232a3876c2 Clean up imports to separate client and server more.
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
2013-03-11 15:13:42 -04:00
92724d10ba Rework 'nilmtool insert' and some client stuff to speed up inserting data
Still needs work.
2013-03-06 20:49:14 -05:00
ea3ea487bc Merge branch 'rocket-insert'
Conflicts:
	nilmdb/server/bulkdata.py
	nilmdb/server/server.py
	nilmdb/utils/__init__.py
2013-03-06 20:46:04 -05:00
69ad8c4842 Merge branch 'rocket' 2013-03-06 20:38:02 -05:00
84e21ff467 Move ASCII data parsing from the server to the rocket interface.
The server buffers the string and passes it to nilmdb.  Nilmdb passes
the string to bulkdata.  Bulkdata uses the rocket interface to parse
it in chunks, as necessary.  Everything gets passed back up and
everyone is happy.

Currently, only pyrocket implements append_string.
2013-03-05 17:51:17 -05:00
11b228f77a Convert times to microsecond precision strings more consistently.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
2013-03-05 17:07:39 -05:00
79755dc624 Fix Allow: header by switching to cherrypy's built in tools.allow().
Replaces custom tools.allow_methods which didn't return the Allow: header.
2013-03-04 14:08:37 -05:00
e318888a06 Finish Rocket.extract_string; clean up code for other functions too
This is maybe 2.5-3 times faster than the list-based code, which
still isn't amazing, but is decent.
2013-03-03 21:25:00 -05:00
9b6de6ecb7 Replace old layout strings everywhere 2013-03-03 13:37:02 -05:00
28310fe886 Add test for extents 2013-03-02 15:19:25 -05:00
521ff88f7c Support 'nilmtool help command' just like 'nilmtool command --help' 2013-03-02 13:56:03 -05:00
64897a1dd1 Change port from 12380 -> 32180 when running tests
This is so tests can be run without interfering with a normal server.
2013-03-02 13:19:44 -05:00
e3f335dfe5 Move time parsing from cmdline into nilmdb.utils.time 2013-02-28 17:09:26 -05:00
50a4a60786 Replace pyCurl with Requests
Only tested with v1.1.0.  It's not clear how well older versions will
work.
2013-02-26 17:45:40 -05:00
317c53ab6f Improve serializer_proxy and verify_thread_proxy
These functions can now take an object or a type (class).

If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
2013-02-23 14:28:37 -05:00
dfbbe23512 Switch to explicitly wrapping nilmdb objects in a serializer_proxy
This is quite a bit simpler than the class decorator method, so it
may be more reliable.
2013-02-23 11:23:54 -05:00
a49c655816 Strictly enforce (start < end) for all intervals.
Previously, we allowed start == end, but this doesn't make sense with
half-open intervals.
2013-02-21 14:06:40 -05:00
841b2dab5c server: Replace /dbpath and /dbsize with a more generic /dbinfo
Update tests accordingly.  This isn't backwards compatible, but
existing clients don't rely on it.
2013-02-14 16:57:33 -05:00
f316026592 Move datetime_tz package under nilmdb.utils
datetime_tz isn't readily available, so it's a lot easier to just
package it within the nilmdb tree.
2013-01-30 19:03:42 -05:00
58c7c8f6ff Support "now" as a timestamp argument 2013-01-28 19:07:45 -05:00
fad23ebb22 Add --timestamp-raw option to extract and list 2013-01-24 16:03:38 -05:00
9bf213707c Properly return an error if two timestamps are equal 2013-01-22 18:35:18 -05:00
87b43e5d04 Command line errors cleaned up and made more consistent 2013-01-16 16:52:43 -05:00
e5d3deb6fe Removal support is complete.
`nrows` may change if you restart the server; documented why this is
the case in the design.md file.  It's not a problem.
2013-01-09 23:26:59 -05:00
7807d6caf0 Progress and tests for bulkdata.remove
Passes tests, but doesn't really handle nrows (and removing partially
full files) correctly, when deleting near the end of the data.
2013-01-09 17:39:29 -05:00
fe3b087435 Remove implemented in nilmdb; still needs bulkdata changes. 2013-01-08 21:07:52 -05:00
f88c148ccc Interval removal work in progress. Needs NilmDB and BulkData work. 2013-01-08 18:37:01 -05:00