049375d30e
Fill out test coverage
2013-03-15 18:08:21 -04:00
90b96799ac
Bulk of the switch to int64 microsecond timestamps, including test data.
2013-03-15 15:08:58 -04:00
b5541722c2
Continue moving time-handling code into nilmdb.utils.time
2013-03-15 15:08:58 -04:00
b9f0b35bbe
Stream renaming support, and comprehensive tests
...
Implemented in command line, client, server, nilmdb, bulkdata
2013-03-14 11:02:30 -04:00
b1b09f8cd0
Strengthen checks when creating paths, fix some bugs, and add tests
2013-03-13 17:45:47 -04:00
d467df7980
Add specific error for creating a path that already exists
2013-03-13 10:14:28 -04:00
b77f07a4cd
Fix reporting of parsing errors with malformed data
...
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
2013-03-12 16:44:36 -04:00
83bc5bc775
Make rocket/bulkdata errors include column number and the bad data
...
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
2013-03-12 16:10:00 -04:00
7eca587fdf
Add 'nilmtool intervals' command, with --diff option
...
Can show the set-difference between the interval ranges in two
streams.
2013-03-11 17:07:26 -04:00
8e79998e95
Tune sqlite to use write-ahead-logging
...
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
2013-03-11 15:13:43 -04:00
9f914598c2
Make /stream/list give some more extended info, like row count
...
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
2013-03-11 15:13:43 -04:00
232a3876c2
Clean up imports to separate client and server more.
...
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
2013-03-11 15:13:42 -04:00
92724d10ba
Rework 'nilmtool insert' and some client stuff to speed up inserting data
...
Still needs work.
2013-03-06 20:49:14 -05:00
ea3ea487bc
Merge branch 'rocket-insert'
...
Conflicts:
nilmdb/server/bulkdata.py
nilmdb/server/server.py
nilmdb/utils/__init__.py
2013-03-06 20:46:04 -05:00
69ad8c4842
Merge branch 'rocket'
2013-03-06 20:38:02 -05:00
84e21ff467
Move ASCII data parsing from the server to the rocket interface.
...
The server buffers the string and passes it to nilmdb. Nilmdb passes
the string to bulkdata. Bulkdata uses the rocket interface to parse
it in chunks, as necessary. Everything gets passed back up and
everyone is happy.
Currently, only pyrocket implements append_string.
2013-03-05 17:51:17 -05:00
11b228f77a
Convert times to microsecond precision strings more consistently.
...
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
2013-03-05 17:07:39 -05:00
79755dc624
Fix Allow: header by switching to cherrypy's built in tools.allow().
...
Replaces custom tools.allow_methods which didn't return the Allow: header.
2013-03-04 14:08:37 -05:00
e318888a06
Finish Rocket.extract_string; clean up code for other functions too
...
This is maybe 2.5-3 times faster than the list-based code, which
still isn't amazing, but is decent.
2013-03-03 21:25:00 -05:00
9b6de6ecb7
Replace old layout strings everywhere
2013-03-03 13:37:02 -05:00
28310fe886
Add test for extents
2013-03-02 15:19:25 -05:00
521ff88f7c
Support 'nilmtool help command' just like 'nilmtool command --help'
2013-03-02 13:56:03 -05:00
64897a1dd1
Change port from 12380 -> 32180 when running tests
...
This is so tests can be run without interfering with a normal server.
2013-03-02 13:19:44 -05:00
e3f335dfe5
Move time parsing from cmdline into nilmdb.utils.time
2013-02-28 17:09:26 -05:00
50a4a60786
Replace pyCurl with Requests
...
Only tested with v1.1.0. It's not clear how well older versions will
work.
2013-02-26 17:45:40 -05:00
317c53ab6f
Improve serializer_proxy and verify_thread_proxy
...
These functions can now take an object or a type (class).
If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
2013-02-23 14:28:37 -05:00
dfbbe23512
Switch to explicitly wrapping nilmdb objects in a serializer_proxy
...
This is quite a bit simpler than the class decorator method, so it
may be more reliable.
2013-02-23 11:23:54 -05:00
a49c655816
Strictly enforce (start < end) for all intervals.
...
Previously, we allowed start == end, but this doesn't make sense with
half-open intervals.
2013-02-21 14:06:40 -05:00
841b2dab5c
server: Replace /dbpath and /dbsize with a more generic /dbinfo
...
Update tests accordingly. This isn't backwards compatible, but
existing clients don't rely on it.
2013-02-14 16:57:33 -05:00
f316026592
Move datetime_tz package under nilmdb.utils
...
datetime_tz isn't readily available, so it's a lot easier to just
package it within the nilmdb tree.
2013-01-30 19:03:42 -05:00
58c7c8f6ff
Support "now" as a timestamp argument
2013-01-28 19:07:45 -05:00
fad23ebb22
Add --timestamp-raw option to extract and list
2013-01-24 16:03:38 -05:00
9bf213707c
Properly return an error if two timestamps are equal
2013-01-22 18:35:18 -05:00
87b43e5d04
Command line errors cleaned up and made more consistent
2013-01-16 16:52:43 -05:00
e5d3deb6fe
Removal support is complete.
...
`nrows` may change if you restart the server; documented why this is
the case in the design.md file. It's not a problem.
2013-01-09 23:26:59 -05:00
7807d6caf0
Progress and tests for bulkdata.remove
...
Passes tests, but doesn't really handle nrows (and removing partially
full files) correctly, when deleting near the end of the data.
2013-01-09 17:39:29 -05:00
fe3b087435
Remove implemented in nilmdb; still needs bulkdata changes.
2013-01-08 21:07:52 -05:00
f88c148ccc
Interval removal work in progress. Needs NilmDB and BulkData work.
2013-01-08 18:37:01 -05:00
80da937cb7
cmdline: return error when start > end (extract, list, remove)
2013-01-06 20:13:28 -05:00
c81972e66e
Minor testsuite and commandline fixes.
...
Now supports "list /foo/bar" in addition to the older "list --path /foo/bar"
2013-01-06 19:25:07 -05:00
3d212e7592
Move test helpers into subdirectory
2013-01-05 15:00:34 -05:00
07192c6ffb
nilmdb.BulkData: Switch to nested subdir/filename layout
...
Use numbered subdirectories to avoid having too many files in one dir.
Add appropriate tests.
Also fix an issue where the mmap_open LRU cache could inappropriately
open a file twice because it was using the optional "newsize"
parameter as a key -- now lrucache can be given a slice object that
describes which arguments are important.
2013-01-04 16:51:05 -05:00
11b0293d5f
Clean up BulkData file size calculations, test more thoroughly
...
Now the goal is 128 MiB files, rather than a specific length.
2013-01-03 20:19:01 -05:00
c083d63c96
Tests for Unicode compliance
2013-01-03 17:03:52 -05:00
0221e3ea21
Update commandline test helpers to better handle Unicode
...
We replace cStringIO with StringIO subclass that forces UTF-8
encoding, and explicitly convert commandlines to UTF-8 before
shlex. These changes will only affect tests, not normal commandline
operation.
2013-01-03 17:03:52 -05:00
06e91a6a98
Always use function version of print()
2013-01-03 17:02:38 -05:00
c7c65b6542
Work around CherryPy bug #1200 ; related cleanups
...
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug. Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not. fun!
2013-01-01 23:03:53 -05:00
83b937c720
More Pytables -> bulkdata conversion
2012-12-31 17:22:30 -05:00
f355c73209
Refactor utility classes into nilmdb.utils subdir/namespace
...
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
9082cc9f44
Merging adjacent intervals is working now!
...
Adjust test expectations accordingly, since the number of intervals
they print out will now be smaller.
2012-12-12 19:25:27 -05:00
