e5d3deb6fe
Removal support is complete.
...
`nrows` may change if you restart the server; documented why this is
the case in the design.md file. It's not a problem.
2013-01-09 23:26:59 -05:00
d321058b48
Add basic versioning to bulkdata table format file.
2013-01-09 19:26:24 -05:00
cea83140c0
More work towards correctly removing rows.
2013-01-09 19:25:45 -05:00
7807d6caf0
Progress and tests for bulkdata.remove
...
Passes tests, but doesn't really handle nrows (and removing partially
full files) correctly, when deleting near the end of the data.
2013-01-09 17:39:29 -05:00
3d0fad3c2a
Move some helper functions around
2013-01-09 17:39:29 -05:00
fe3b087435
Remove implemented in nilmdb; still needs bulkdata changes.
2013-01-08 21:07:52 -05:00
bcefe52298
nilmdb: Bring out range manipulating SQL so we can reuse it
2013-01-08 18:45:03 -05:00
f88c148ccc
Interval removal work in progress. Needs NilmDB and BulkData work.
2013-01-08 18:37:01 -05:00
4a47b1d04a
remove support: command line, client
2013-01-06 20:13:57 -05:00
80da937cb7
cmdline: return error when start > end (extract, list, remove)
2013-01-06 20:13:28 -05:00
c81972e66e
Minor testsuite and commandline fixes.
...
Now supports "list /foo/bar" in addition to the older "list --path /foo/bar"
2013-01-06 19:25:07 -05:00
b09362fde1
Full coverage of nilmdb.utils.mustclose
2013-01-05 18:02:53 -05:00
b7688844fa
Add a Nosetests plugin that lets me specify a test order within a directory.
2013-01-05 18:02:37 -05:00
3d212e7592
Move test helpers into subdirectory
2013-01-05 15:00:34 -05:00
7aedfdf9c3
Add lower level bulkdata test
2013-01-05 14:55:22 -05:00
ebd4f74959
Remove "pragma: no cover" from things that should get tested
2013-01-05 14:52:06 -05:00
ebe2fbab92
Add wrap_verify option to nilmdb.utils.must_close decorator
2013-01-05 14:51:41 -05:00
4831a0cae1
Small doc updates
2013-01-04 17:27:04 -05:00
07192c6ffb
nilmdb.BulkData: Switch to nested subdir/filename layout
...
Use numbered subdirectories to avoid having too many files in one dir.
Add appropriate tests.
Also fix an issue where the mmap_open LRU cache could inappropriately
open a file twice because it was using the optional "newsize"
parameter as a key -- now lrucache can be given a slice object that
describes which arguments are important.
2013-01-04 16:51:05 -05:00
09d325e8ab
Rename format -> _format in data dirs
2013-01-03 20:46:15 -05:00
11b0293d5f
Clean up BulkData file size calculations, test more thoroughly
...
Now the goal is 128 MiB files, rather than a specific length.
2013-01-03 20:19:01 -05:00
493bbed82c
More coverage and tests
2013-01-03 19:21:12 -05:00
3bc25daaab
Trim urllib to get full coverage of the features in use
2013-01-03 17:10:07 -05:00
40a3bc4bc3
Update README with Python 2.7 requirement
2013-01-03 17:09:51 -05:00
c083d63c96
Tests for Unicode compliance
2013-01-03 17:03:52 -05:00
0221e3ea21
Update commandline test helpers to better handle Unicode
...
We replace cStringIO with StringIO subclass that forces UTF-8
encoding, and explicitly convert commandlines to UTF-8 before
shlex. These changes will only affect tests, not normal commandline
operation.
2013-01-03 17:03:52 -05:00
f5fd2b064e
Replace urllib.encode() with a version that encodes Unicode as UTF-8 instead
2013-01-03 17:02:38 -05:00
06e91a6a98
Always use function version of print()
2013-01-03 17:02:38 -05:00
41b3f3c018
Always use UTF-8 for filenames in nilmdb.bulkdata
2013-01-03 17:02:38 -05:00
842076fef4
Cleanup server error handling with decorator
2013-01-03 17:02:38 -05:00
10d58f6a47
More test coverage
2013-01-02 00:00:05 -05:00
e2464efc12
Test everything; remove debugging
2013-01-01 23:46:54 -05:00
1beae5024e
Bulkdata extract works now.
2013-01-01 23:44:52 -05:00
c7c65b6542
Work around CherryPy bug #1200 ; related cleanups
...
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug. Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not. fun!
2013-01-01 23:03:53 -05:00
f41ff0a6e8
Inserting bulk data is essentially done, not tested
2013-01-01 21:04:35 -05:00
389c1d189f
Make option to turn off chunked encoding for debugging more clear.
2013-01-01 21:03:33 -05:00
487298986e
More work towards bulkdata
2012-12-31 18:44:57 -05:00
d4cd045c48
Fix path stuff, build packer in bulkdata.Table
2012-12-31 17:22:30 -05:00
3816645313
More work on BulkData
2012-12-31 17:22:30 -05:00
83b937c720
More Pytables -> bulkdata conversion
2012-12-31 17:22:30 -05:00
b3e6e8976f
More work towards flat bulk data storage.
...
Cleaned up OS-specific path handling in nilmdb, bulkdata.
2012-12-31 17:22:30 -05:00
c890ea93cb
WIP switching away from PyTables
2012-12-31 17:22:29 -05:00
84c68c6913
Better documentation, cache Tables
2012-12-31 17:22:29 -05:00
6f1e6fe232
Isolate all PyTables stuff to a single file.
...
This will make migrating to my own data storage engine easier.
2012-12-31 17:22:29 -05:00
b0d76312d1
Add must_close() decorator, use it in nilmdb
...
Warns at runtime if a class's close() method wasn't called before the
object was destroyed.
2012-12-31 17:21:19 -05:00
19c846c71c
Remove outdated files
2012-12-31 15:55:43 -05:00
f355c73209
Refactor utility classes into nilmdb.utils subdir/namespace
...
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
173014ba19
Use nilmdb.lrucache for caching interval sets
2012-12-31 14:52:46 -05:00
24d4752bc3
Add LRU cache memoizing decorator for functions
2012-12-31 14:39:16 -05:00
a85b273e2e
Remove compression.
...
Messes up extraction, since we random access for the timestamp binary
search. In the future, maybe switching to multiple tables (one for
timestamp, one for compressed data) would be smart.
2012-12-14 17:19:23 -05:00
