Jim Paris
89be6f5931
Add option to include interval start/end markup on extract
When enabled, lines like "# interval-start 1234567890123456" and "#
interval-end 1234567890123456" will be added to the data output. Note
that there may be an "interval-end" timestamp followed by an identical
"interval-start" timestamp, if the response at the nilmdb level was
split up into multiple chunks.
In general, assume contiguous data if previous_interval_end ==
new_interval_start.
11 years ago
Jim Paris
4cdef3285d
Destroy now requires that all data has been previously removed.
Added new flag "-R" to command line to perform an automatic removal.
This should be the last of the ways in which a single command could
block the nilmdb thread for a long time.
11 years ago
Jim Paris
bcd82c4d59
Limit the number of rows removed per call to nilmdb.stream_remove
Server class will retry as needed, as with stream_extract and
stream_intervals.
11 years ago
Jim Paris
caf63ab01f
Fix stream_extract/stream_intervals restart around timestamp == 0.
11 years ago
Jim Paris
2d72891162
Accept "min" and "max" as timestamps on command line
11 years ago
Jim Paris
cda2ac3e77
Don't return a mutable interval from IntervalSet.intersection()
Instead, always take the subset, which creates a new interval.
Also adds a small optimization by moving the 'if orig' check outside the
loop.
11 years ago
Jim Paris
57d3d60f6a
Fix relative import problems
11 years ago
Jim Paris
7429c1788d
Update nilmdb.utils.time
11 years ago
Jim Paris
0ef71c193b
Remove layout.pyx, since rocket replaced it
11 years ago
Jim Paris
4f06d6ae68
Move Interval set_difference inside nilmdb.utils for clients
Clients might need to to Interval math too, so move a simple Interval
class and start putting helpers in there.
11 years ago
Jim Paris
52ae397d7d
Bump database version to 3, reject old version 2 due to timestamp changes
11 years ago
Jim Paris
88eb0123f5
Add test for Table.__getitem__ indexing
11 years ago
Jim Paris
a547ddbbba
Change table.get_timestamp to table.__getitem__
This lets us use simple indexing to get timestamps from the table,
which allows us to use 'bisect' directly without needing a proxy class.
11 years ago
Jim Paris
28e72fd53e
Remove Table.__getitem__; used only by tests
11 years ago
Jim Paris
f63107b334
Add rocket.extract_timestamp to speed up bisections
11 years ago
Jim Paris
955d7aa871
Remove floating port time support from nilmdb.utils.time
11 years ago
Jim Paris
b8d2cf1b78
Consolidate rocket._extract_handle.params with extract_string
11 years ago
Jim Paris
7c465730de
Remove rocket.extract_pyobject
11 years ago
Jim Paris
aca130272d
Remove rocket.extract_list
11 years ago
Jim Paris
76e5e9883f
Remove Table.append, rocket.append_iter
11 years ago
Jim Paris
fb4f4519ff
Clean up and simplify Table.get_*, including __getitem__
11 years ago
Jim Paris
30328714a7
Remove python implementation of rocket
11 years ago
Jim Paris
d3efb829b5
Try to parse timestamps as double, if int64 parse fails
11 years ago
Jim Paris
90b96799ac
Bulk of the switch to int64 microsecond timestamps, including test data.
11 years ago
Jim Paris
b5541722c2
Continue moving time-handling code into nilmdb.utils.time
11 years ago
Jim Paris
aaea105861
Consolidate most timestamp <-> string conversions (outside of rocket)
11 years ago
Jim Paris
e6a081d639
Consolidate timestamp constants into nilmdb.utils.time
11 years ago
Jim Paris
1835d03412
Bump bulkdata database version to 3
11 years ago
Jim Paris
cd1dfe7dcd
Add completion functions to most commandline arguments
11 years ago
Jim Paris
fb35517dfa
Add basic argument completion
11 years ago
Jim Paris
b9f0b35bbe
Stream renaming support, and comprehensive tests
Implemented in command line, client, server, nilmdb, bulkdata
11 years ago
Jim Paris
b1b09f8cd0
Strengthen checks when creating paths, fix some bugs, and add tests
11 years ago
Jim Paris
d467df7980
Add specific error for creating a path that already exists
11 years ago
Jim Paris
09bc7eb48c
Make StreamInserter.insert complain if data remains after send
Previously, we ignored problems when sending intermediate blocks,
since getting more data might make the next attempt succeed.
But in practice, malformed data would just build up, causing
problems. Raise an exception if there's too much data remaining
after trying to send an intermediate block.
11 years ago
Jim Paris
b77f07a4cd
Fix reporting of parsing errors with malformed data
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
11 years ago
Jim Paris
59f0076306
Increase max layout count in rocket
11 years ago
Jim Paris
83bc5bc775
Make rocket/bulkdata errors include column number and the bad data
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
11 years ago
Jim Paris
6b1dfec828
In stream_list, return 0 instead of None for rows and seconds
For rows and seconds only. Extents still give None if they don't
exist.
11 years ago
Jim Paris
7eca587fdf
Add 'nilmtool intervals' command, with --diff option
Can show the set-difference between the interval ranges in two
streams.
11 years ago
Jim Paris
a351bc1b10
Add client, server, nilmdb support for listing interval differences
11 years ago
Jim Paris
1d61d61a81
Add interval.set_difference function and associated tests
11 years ago
Jim Paris
755255030b
Clean up interval __and__ function; we don't need to __and__ sets
11 years ago
Jim Paris
8e79998e95
Tune sqlite to use write-ahead-logging
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
11 years ago
Jim Paris
9f914598c2
Make /stream/list give some more extended info, like row count
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
11 years ago
Jim Paris
0468b04538
Fix pyrocket to handle comments better
11 years ago
Jim Paris
232a3876c2
Clean up imports to separate client and server more.
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
11 years ago
Jim Paris
1c27dd72d6
Fill out client tests and fix various bugs
Fixes various corner cases and other bugs regarding lines with
comments, having data but no endpoints, etc.
11 years ago
Jim Paris
0fc092779d
Big rework of stream_insert_context and places that use it.
Things are now block-focused, rather than line-focused. This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
11 years ago
Jim Paris
7abfdfbf3e
Add const qualifier to strings we get from Python
11 years ago
Jim Paris
92724d10ba
Rework 'nilmtool insert' and some client stuff to speed up inserting data
Still needs work.
11 years ago