Jim Paris
0f5c1c0db6
Run 2to3 automatic fixes over entire tree
4 years ago
Jim Paris
022b50950f
Support using a higher initial nrows in bulkdata, for tests
This gives an easy way to get a large values in the database start_pos
and end_pos fields, which is necessary for testing failure modes when
those get too large (e.g. on 32-bit systems). Adjust tests to make
use of this knob.
9 years ago
Jim Paris
a01cb4132d
Add test for limited interval removal
10 years ago
Jim Paris
f5276e9fc8
Test --no-decim
10 years ago
Jim Paris
a79d6104d5
Documentation fixups
10 years ago
Jim Paris
8e8ec59e30
Support "nilmtool cmd --version"
10 years ago
Jim Paris
bd7bdb2eb8
Add --optimize option to nilmtool intervals
10 years ago
Jim Paris
b5df575c79
Fix tests
11 years ago
Jim Paris
9a2699adfc
Attempt at fixing up more Unicode issues with metadata.
11 years ago
Jim Paris
539c92226c
Add more disk space info
11 years ago
Jim Paris
f2a48bdb2a
Test binary extract; fix bugs
11 years ago
Jim Paris
416a499866
Support wildcards for destroy
11 years ago
Jim Paris
0104c8edd9
nilmtool remove: allow wildcards and multiple paths
11 years ago
Jim Paris
83d022016c
nilmtool list: add new --layout option to show layouts
11 years ago
Jim Paris
43b740ecaa
nilmtool list: remove old -p parameter
11 years ago
Jim Paris
230ec72609
Fix timestamp display issues with --annotate
11 years ago
Jim Paris
9ce0f69dff
Add "--delete" option to "nilmtool metadata" tool
This is the same as "--update" with an empty string as the value.
11 years ago
Jim Paris
ebccfb3531
Fix stream renaming when the new path is a parent of the old
11 years ago
Jim Paris
89be6f5931
Add option to include interval start/end markup on extract
When enabled, lines like "# interval-start 1234567890123456" and "#
interval-end 1234567890123456" will be added to the data output. Note
that there may be an "interval-end" timestamp followed by an identical
"interval-start" timestamp, if the response at the nilmdb level was
split up into multiple chunks.
In general, assume contiguous data if previous_interval_end ==
new_interval_start.
11 years ago
Jim Paris
4cdef3285d
Destroy now requires that all data has been previously removed.
Added new flag "-R" to command line to perform an automatic removal.
This should be the last of the ways in which a single command could
block the nilmdb thread for a long time.
11 years ago
Jim Paris
bcd82c4d59
Limit the number of rows removed per call to nilmdb.stream_remove
Server class will retry as needed, as with stream_extract and
stream_intervals.
11 years ago
Jim Paris
2d72891162
Accept "min" and "max" as timestamps on command line
11 years ago
Jim Paris
049375d30e
Fill out test coverage
11 years ago
Jim Paris
90b96799ac
Bulk of the switch to int64 microsecond timestamps, including test data.
11 years ago
Jim Paris
b5541722c2
Continue moving time-handling code into nilmdb.utils.time
11 years ago
Jim Paris
b9f0b35bbe
Stream renaming support, and comprehensive tests
Implemented in command line, client, server, nilmdb, bulkdata
11 years ago
Jim Paris
b1b09f8cd0
Strengthen checks when creating paths, fix some bugs, and add tests
11 years ago
Jim Paris
d467df7980
Add specific error for creating a path that already exists
11 years ago
Jim Paris
b77f07a4cd
Fix reporting of parsing errors with malformed data
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
11 years ago
Jim Paris
83bc5bc775
Make rocket/bulkdata errors include column number and the bad data
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
11 years ago
Jim Paris
7eca587fdf
Add 'nilmtool intervals' command, with --diff option
Can show the set-difference between the interval ranges in two
streams.
11 years ago
Jim Paris
8e79998e95
Tune sqlite to use write-ahead-logging
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
11 years ago
Jim Paris
9f914598c2
Make /stream/list give some more extended info, like row count
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
11 years ago
Jim Paris
232a3876c2
Clean up imports to separate client and server more.
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
11 years ago
Jim Paris
92724d10ba
Rework 'nilmtool insert' and some client stuff to speed up inserting data
Still needs work.
11 years ago
Jim Paris
84e21ff467
Move ASCII data parsing from the server to the rocket interface.
The server buffers the string and passes it to nilmdb. Nilmdb passes
the string to bulkdata. Bulkdata uses the rocket interface to parse
it in chunks, as necessary. Everything gets passed back up and
everyone is happy.
Currently, only pyrocket implements append_string.
11 years ago
Jim Paris
11b228f77a
Convert times to microsecond precision strings more consistently.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
11 years ago
Jim Paris
79755dc624
Fix Allow: header by switching to cherrypy's built in tools.allow().
Replaces custom tools.allow_methods which didn't return the Allow: header.
11 years ago
Jim Paris
e318888a06
Finish Rocket.extract_string; clean up code for other functions too
This is maybe 2.5-3 times faster than the list-based code, which
still isn't amazing, but is decent.
11 years ago
Jim Paris
9b6de6ecb7
Replace old layout strings everywhere
11 years ago
Jim Paris
28310fe886
Add test for extents
11 years ago
Jim Paris
521ff88f7c
Support 'nilmtool help command' just like 'nilmtool command --help'
11 years ago
Jim Paris
64897a1dd1
Change port from 12380 -> 32180 when running tests
This is so tests can be run without interfering with a normal server.
11 years ago
Jim Paris
e3f335dfe5
Move time parsing from cmdline into nilmdb.utils.time
11 years ago
Jim Paris
50a4a60786
Replace pyCurl with Requests
Only tested with v1.1.0. It's not clear how well older versions will
work.
11 years ago
Jim Paris
317c53ab6f
Improve serializer_proxy and verify_thread_proxy
These functions can now take an object or a type (class).
If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
11 years ago
Jim Paris
dfbbe23512
Switch to explicitly wrapping nilmdb objects in a serializer_proxy
This is quite a bit simpler than the class decorator method, so it
may be more reliable.
11 years ago
Jim Paris
a49c655816
Strictly enforce (start < end) for all intervals.
Previously, we allowed start == end, but this doesn't make sense with
half-open intervals.
11 years ago
Jim Paris
841b2dab5c
server: Replace /dbpath and /dbsize with a more generic /dbinfo
Update tests accordingly. This isn't backwards compatible, but
existing clients don't rely on it.
11 years ago
Jim Paris
f316026592
Move datetime_tz package under nilmdb.utils
datetime_tz isn't readily available, so it's a lot easier to just
package it within the nilmdb tree.
11 years ago