26bab031bd
Add StreamInserter.send() to trigger intermediate block send
2013-03-30 17:30:43 -04:00
f8ca8d31e6
Remove Iteratorizer, as it's no longer needed
2013-03-24 21:31:03 -04:00
ed89d803f0
Remove aplotter code
2013-03-24 21:29:09 -04:00
3d24092cd2
Replace bare 'except:' with 'except: Exception'
...
Otherwise we might inadvertently catch SystemExit or KeyboardExit or
something we don't want to catch.
2013-03-24 21:28:01 -04:00
c0d450d39e
Add locking mechanism to avoid multiple servers on one DB
2013-03-24 21:20:20 -04:00
e790bb9e8a
Fix test failure when tests are run as root
2013-03-21 14:33:02 -04:00
89be6f5931
Add option to include interval start/end markup on extract
...
When enabled, lines like "# interval-start 1234567890123456" and "#
interval-end 1234567890123456" will be added to the data output. Note
that there may be an "interval-end" timestamp followed by an identical
"interval-start" timestamp, if the response at the nilmdb level was
split up into multiple chunks.
In general, assume contiguous data if previous_interval_end ==
new_interval_start.
2013-03-19 14:23:33 -04:00
4cdef3285d
Destroy now requires that all data has been previously removed.
...
Added new flag "-R" to command line to perform an automatic removal.
This should be the last of the ways in which a single command could
block the nilmdb thread for a long time.
2013-03-18 19:39:03 -04:00
bcd82c4d59
Limit the number of rows removed per call to nilmdb.stream_remove
...
Server class will retry as needed, as with stream_extract and
stream_intervals.
2013-03-18 18:22:45 -04:00
2d72891162
Accept "min" and "max" as timestamps on command line
2013-03-18 18:19:24 -04:00
0ef71c193b
Remove layout.pyx, since rocket replaced it
2013-03-15 22:32:40 -04:00
22274550ab
Test python version of Interval too
2013-03-15 21:37:03 -04:00
4f06d6ae68
Move Interval set_difference inside nilmdb.utils for clients
...
Clients might need to to Interval math too, so move a simple Interval
class and start putting helpers in there.
2013-03-15 21:37:03 -04:00
c54d8041c3
Update design docs
2013-03-15 21:07:01 -04:00
049375d30e
Fill out test coverage
2013-03-15 18:08:21 -04:00
88eb0123f5
Add test for Table.__getitem__ indexing
2013-03-15 18:08:21 -04:00
28e72fd53e
Remove Table.__getitem__; used only by tests
2013-03-15 18:08:21 -04:00
fb4f4519ff
Clean up and simplify Table.get_*, including __getitem__
2013-03-15 18:08:20 -04:00
90b96799ac
Bulk of the switch to int64 microsecond timestamps, including test data.
2013-03-15 15:08:58 -04:00
56679ad770
Move more datetime_tz calls into common code
2013-03-15 15:08:58 -04:00
b5541722c2
Continue moving time-handling code into nilmdb.utils.time
2013-03-15 15:08:58 -04:00
c7a712d8d8
Partial test for rounding issues
2013-03-15 15:08:57 -04:00
b9f0b35bbe
Stream renaming support, and comprehensive tests
...
Implemented in command line, client, server, nilmdb, bulkdata
2013-03-14 11:02:30 -04:00
b1b09f8cd0
Strengthen checks when creating paths, fix some bugs, and add tests
2013-03-13 17:45:47 -04:00
d467df7980
Add specific error for creating a path that already exists
2013-03-13 10:14:28 -04:00
b77f07a4cd
Fix reporting of parsing errors with malformed data
...
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
2013-03-12 16:44:36 -04:00
83bc5bc775
Make rocket/bulkdata errors include column number and the bad data
...
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
2013-03-12 16:10:00 -04:00
7eca587fdf
Add 'nilmtool intervals' command, with --diff option
...
Can show the set-difference between the interval ranges in two
streams.
2013-03-11 17:07:26 -04:00
1d61d61a81
Add interval.set_difference function and associated tests
2013-03-11 15:40:50 -04:00
8e79998e95
Tune sqlite to use write-ahead-logging
...
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
2013-03-11 15:13:43 -04:00
9f914598c2
Make /stream/list give some more extended info, like row count
...
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
2013-03-11 15:13:43 -04:00
232a3876c2
Clean up imports to separate client and server more.
...
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
2013-03-11 15:13:42 -04:00
1c27dd72d6
Fill out client tests and fix various bugs
...
Fixes various corner cases and other bugs regarding lines with
comments, having data but no endpoints, etc.
2013-03-08 12:36:17 -05:00
0fc092779d
Big rework of stream_insert_context and places that use it.
...
Things are now block-focused, rather than line-focused. This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
2013-03-07 20:30:11 -05:00
92724d10ba
Rework 'nilmtool insert' and some client stuff to speed up inserting data
...
Still needs work.
2013-03-06 20:49:14 -05:00
1d7acbf916
Remove null timestamper, speed up insert --none a tiny bit
2013-03-06 20:46:51 -05:00
ea3ea487bc
Merge branch 'rocket-insert'
...
Conflicts:
nilmdb/server/bulkdata.py
nilmdb/server/server.py
nilmdb/utils/__init__.py
2013-03-06 20:46:04 -05:00
69ad8c4842
Merge branch 'rocket'
2013-03-06 20:38:02 -05:00
65f09f793c
When re-raising exceptions in the server, preserve original tracebacks
2013-03-05 21:48:40 -05:00
84e21ff467
Move ASCII data parsing from the server to the rocket interface.
...
The server buffers the string and passes it to nilmdb. Nilmdb passes
the string to bulkdata. Bulkdata uses the rocket interface to parse
it in chunks, as necessary. Everything gets passed back up and
everyone is happy.
Currently, only pyrocket implements append_string.
2013-03-05 17:51:17 -05:00
11b228f77a
Convert times to microsecond precision strings more consistently.
...
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
2013-03-05 17:07:39 -05:00
454e561d69
Verify that metadata values are numbers or strings
2013-03-05 13:22:17 -05:00
58c0ae72f6
Support application/json POST bodies as well as x-www-form-urlencoded
2013-03-05 11:54:29 -05:00
c6a0e6e96f
More complete CORS handling, including preflight requests (hopefully)
2013-03-04 15:40:35 -05:00
79755dc624
Fix Allow: header by switching to cherrypy's built in tools.allow().
...
Replaces custom tools.allow_methods which didn't return the Allow: header.
2013-03-04 14:08:37 -05:00
0d372fb878
Modify old formatter to match rocket's formatting style
2013-03-03 21:50:29 -05:00
0b75da7a8f
Normalize the floating point formats to %.6e and %.16e
...
This is mostly a matter of taste, but it matches more closely with the
old way that prep did it, and it's more consistent. It should roughly
match the available precision of floats and doubles.
2013-03-03 21:43:04 -05:00
e318888a06
Finish Rocket.extract_string; clean up code for other functions too
...
This is maybe 2.5-3 times faster than the list-based code, which
still isn't amazing, but is decent.
2013-03-03 21:25:00 -05:00
9d0d2415be
Test bulkdata a little more carefully
2013-03-03 14:00:00 -05:00
4406d51a98
First pass at Python implementation of rocket
2013-03-03 13:37:02 -05:00
9b6de6ecb7
Replace old layout strings everywhere
2013-03-03 13:37:02 -05:00
28310fe886
Add test for extents
2013-03-02 15:19:25 -05:00
521ff88f7c
Support 'nilmtool help command' just like 'nilmtool command --help'
2013-03-02 13:56:03 -05:00
64897a1dd1
Change port from 12380 -> 32180 when running tests
...
This is so tests can be run without interfering with a normal server.
2013-03-02 13:19:44 -05:00
5db3b186a4
Make test_mustclose more complete
2013-03-01 16:30:22 -05:00
ca67c79fe4
Improve test_layout_speed
2013-03-01 16:04:10 -05:00
8917bcd4bf
Fix test case failures due to increased client chunk size
2013-03-01 16:04:00 -05:00
e3f335dfe5
Move time parsing from cmdline into nilmdb.utils.time
2013-02-28 17:09:26 -05:00
3df96fdfdd
Reorder code
2013-02-26 19:41:55 -05:00
740ab76eaf
Re-add persistent connection test for Requests based httpclient
2013-02-26 19:41:27 -05:00
ce13a47fea
Save full response object for tests
2013-02-26 17:45:41 -05:00
50a4a60786
Replace pyCurl with Requests
...
Only tested with v1.1.0. It's not clear how well older versions will
work.
2013-02-26 17:45:40 -05:00
14afa02db6
Temporarily remove curl-specific keepalive tests
2013-02-26 17:45:40 -05:00
cc990d6ce4
Test persistent connections
2013-02-26 13:41:40 -05:00
a4a4bc61ba
Switch to using pycurl.Multi instead of Iteratorizer
2013-02-25 21:05:01 -05:00
3d82888580
Enforce method types, and require POST for actions that change things.
...
This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
2013-02-25 21:05:01 -05:00
749b878904
Add an explicit lock to httpclient's public methods
...
This is to prevent possible reentrancy problems.
2013-02-25 18:06:00 -05:00
317c53ab6f
Improve serializer_proxy and verify_thread_proxy
...
These functions can now take an object or a type (class).
If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
2013-02-23 14:28:37 -05:00
422317850e
Replace threadsafety class decorator version, add explicit proxy version
...
Like the serializer changes, the class decorator was too fragile.
2013-02-23 11:25:40 -05:00
965537d8cb
Implement verify_thread_safety to check for unsafe access patterns
...
Occasional segfaults may be the result of performing thread-unsafe
operations. This class decorator verifies that all of its methods
are called in a thread-safe manner.
It can separately warn about:
- two threads calling methods in a function (the kind of thing sqlite
doesn't like)
- recursion
- concurrency (two different threads functions at the same time)
2013-02-23 11:25:02 -05:00
7fce305a1d
Make server check that the db object has been wrapped in a serializer
...
It's only the server that calls it in multiple threads.
2013-02-23 11:25:01 -05:00
dfbbe23512
Switch to explicitly wrapping nilmdb objects in a serializer_proxy
...
This is quite a bit simpler than the class decorator method, so it
may be more reliable.
2013-02-23 11:23:54 -05:00
7761a91242
Remove class decorator version of the serializer; it's too fragile
2013-02-23 11:23:54 -05:00
9b06e46bf1
Add back a proxy version of the Serializer, which is much simpler.
2013-02-23 11:23:54 -05:00
171e6f1871
Replace "serializer" function with a "serialized" decorator
...
This decorator makes a class always be serialized, including its
instantiation, in a separate thread. This is an improvement over
the old Serializer() object wrapper, which didn't put the
instantiation into the new thread.
2013-02-23 11:23:54 -05:00
1431e41d16
Allow inserting empty intervals in the database, and add tests for it.
...
Previously, we could get empty intervals anyway by having a non-empty
interval and removing a smaller interval around each piece of data.
Turns out that empty intervals are OK and needed in some situations,
so explicitly allow and test for it.
2013-02-21 14:07:35 -05:00
a49c655816
Strictly enforce (start < end) for all intervals.
...
Previously, we allowed start == end, but this doesn't make sense with
half-open intervals.
2013-02-21 14:06:40 -05:00
ca5253ddee
Fix and test stream_count
2013-02-19 18:26:44 -05:00
e19da84b2e
server: always return None instead of sometimes returning "ok"
...
Previously some functions returned the string "ok".
2013-02-19 18:26:44 -05:00
3e8e3542fd
Test for detecting nested HTTP requests
2013-02-19 18:26:44 -05:00
bba9ad131e
Add test for client.stream_insert_context
2013-02-19 17:19:45 -05:00
ee24380d1f
Replace duplicated URL in tests with a variable
2013-02-19 15:27:51 -05:00
bfcd91acf8
client tests: renumber
2013-02-19 15:25:34 -05:00
b53ff31212
client: Add must_close() decorator to nilmdb.Client, and fix tests
...
Test suite wasn't closing connections correctly.
2013-02-16 18:55:23 -05:00
2045e89f24
client: Add context manager functionality, test closing
2013-02-16 18:55:20 -05:00
841b2dab5c
server: Replace /dbpath and /dbsize with a more generic /dbinfo
...
Update tests accordingly. This isn't backwards compatible, but
existing clients don't rely on it.
2013-02-14 16:57:33 -05:00
1593e181a3
Switch to versioneer-provided versions everywhere
2013-02-05 19:07:38 -05:00
b01f23ed99
Move runtests.py script into test directory
2013-02-01 15:47:47 -05:00
f316026592
Move datetime_tz package under nilmdb.utils
...
datetime_tz isn't readily available, so it's a lot easier to just
package it within the nilmdb tree.
2013-01-30 19:03:42 -05:00
58c7c8f6ff
Support "now" as a timestamp argument
2013-01-28 19:07:45 -05:00
225003f412
Huge cleanup of namespaces, modules, packages, imports.
...
Now nilmdb.client, nilmdb.server, nilmdb.cmdline, and nilmdb.utils
are each their own modules, and there is a little bit more of a
logical separation between them. Various changes scattered throughout
to fix naming (for example, nilmdb.nilmdb.NilmDBError is now
nilmdb.server.errors.NilmDBError).
Reduced usage of "from __future__ import absolute_import" as much
as possible. It's still needed for the functions in the nilmdb/server
directory to be able to import the nilmdb module rather than the
nilmdb.py script.
This should hopefully ease future packaging a bit.
2013-01-28 19:04:52 -05:00
40b966aef2
Add pycurl-specific hack to Iteratorizer
...
Inside the pycurl callback, we can't raise exceptions, because the
pycurl extension module will unconditionally print the exception
itself, and not pass it up to the caller. Instead, we have the
callback return a value that tells curl to abort. (-1 would be best,
in case we were given 0 bytes, but the extension doesn't support
that either).
This resolves the 'Exception("should die")' problem when interrupting
a streaming generator like stream_extract.
2013-01-24 19:06:20 -05:00
294ec6988b
Rewrite Iteratorizer as a context manager
...
Relying on __del__ to clean up the thread isn't as reliable.
2013-01-24 19:04:25 -05:00
fad23ebb22
Add --timestamp-raw option to extract and list
2013-01-24 16:03:38 -05:00
b226dc4337
Properly handle test case where server doesn't start
2013-01-24 16:03:38 -05:00
9bf213707c
Properly return an error if two timestamps are equal
2013-01-22 18:35:18 -05:00
5cd7899e98
Send a Access-Control-Allow-Origin (CORS) header with all responses
2013-01-22 14:42:03 -05:00
ceec5fb9b3
Force /stream/interval and /stream/extract responses to be text/plain
2013-01-22 12:47:06 -05:00
2820ff9758
More fixes to mustclose decorator and argspecs
2013-01-18 17:21:30 -05:00
b7f746e66d
Fix lrucache decorator argspecs
2013-01-18 17:13:50 -05:00
40cf4941f0
Test that argspecs are maintained in lrucache
2013-01-18 17:01:46 -05:00
0312b6eb07
Test for issue where mustclose decorator didn't maintain argspec
2013-01-18 16:55:51 -05:00
62354b4dce
Add test for bad-parameters-give-500-error
2013-01-17 19:58:48 -05:00
4f6a742e6c
Fix test failure
2013-01-16 17:31:31 -05:00
87b43e5d04
Command line errors cleaned up and made more consistent
2013-01-16 16:52:43 -05:00
e5d3deb6fe
Removal support is complete.
...
`nrows` may change if you restart the server; documented why this is
the case in the design.md file. It's not a problem.
2013-01-09 23:26:59 -05:00
7807d6caf0
Progress and tests for bulkdata.remove
...
Passes tests, but doesn't really handle nrows (and removing partially
full files) correctly, when deleting near the end of the data.
2013-01-09 17:39:29 -05:00
3d0fad3c2a
Move some helper functions around
2013-01-09 17:39:29 -05:00
fe3b087435
Remove implemented in nilmdb; still needs bulkdata changes.
2013-01-08 21:07:52 -05:00
f88c148ccc
Interval removal work in progress. Needs NilmDB and BulkData work.
2013-01-08 18:37:01 -05:00
80da937cb7
cmdline: return error when start > end (extract, list, remove)
2013-01-06 20:13:28 -05:00
c81972e66e
Minor testsuite and commandline fixes.
...
Now supports "list /foo/bar" in addition to the older "list --path /foo/bar"
2013-01-06 19:25:07 -05:00
b09362fde1
Full coverage of nilmdb.utils.mustclose
2013-01-05 18:02:53 -05:00
b7688844fa
Add a Nosetests plugin that lets me specify a test order within a directory.
2013-01-05 18:02:37 -05:00
3d212e7592
Move test helpers into subdirectory
2013-01-05 15:00:34 -05:00
7aedfdf9c3
Add lower level bulkdata test
2013-01-05 14:55:22 -05:00
07192c6ffb
nilmdb.BulkData: Switch to nested subdir/filename layout
...
Use numbered subdirectories to avoid having too many files in one dir.
Add appropriate tests.
Also fix an issue where the mmap_open LRU cache could inappropriately
open a file twice because it was using the optional "newsize"
parameter as a key -- now lrucache can be given a slice object that
describes which arguments are important.
2013-01-04 16:51:05 -05:00
11b0293d5f
Clean up BulkData file size calculations, test more thoroughly
...
Now the goal is 128 MiB files, rather than a specific length.
2013-01-03 20:19:01 -05:00
493bbed82c
More coverage and tests
2013-01-03 19:21:12 -05:00
c083d63c96
Tests for Unicode compliance
2013-01-03 17:03:52 -05:00
0221e3ea21
Update commandline test helpers to better handle Unicode
...
We replace cStringIO with StringIO subclass that forces UTF-8
encoding, and explicitly convert commandlines to UTF-8 before
shlex. These changes will only affect tests, not normal commandline
operation.
2013-01-03 17:03:52 -05:00
06e91a6a98
Always use function version of print()
2013-01-03 17:02:38 -05:00
10d58f6a47
More test coverage
2013-01-02 00:00:05 -05:00
c7c65b6542
Work around CherryPy bug #1200 ; related cleanups
...
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug. Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not. fun!
2013-01-01 23:03:53 -05:00
83b937c720
More Pytables -> bulkdata conversion
2012-12-31 17:22:30 -05:00
6f1e6fe232
Isolate all PyTables stuff to a single file.
...
This will make migrating to my own data storage engine easier.
2012-12-31 17:22:29 -05:00
b0d76312d1
Add must_close() decorator, use it in nilmdb
...
Warns at runtime if a class's close() method wasn't called before the
object was destroyed.
2012-12-31 17:21:19 -05:00
f355c73209
Refactor utility classes into nilmdb.utils subdir/namespace
...
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
24d4752bc3
Add LRU cache memoizing decorator for functions
2012-12-31 14:39:16 -05:00
9082cc9f44
Merging adjacent intervals is working now!
...
Adjust test expectations accordingly, since the number of intervals
they print out will now be smaller.
2012-12-12 19:25:27 -05:00
bf64a40472
Some misc test additions, interval optimizations. Still need adjacency test
2012-12-11 23:31:55 -05:00
32dbeebc09
More insertion checks. Need to get interval concatenation working.
2012-12-11 18:08:00 -05:00
66ddc79b15
Inserting works again, with proper end/start for paired blocks.
...
timeit.sh script works too!
2012-12-07 20:30:39 -05:00
5fecec2a4c
Support deleting streams with new 'destroy' command
2012-12-04 22:15:00 -05:00
17c329fd6d
Start to be a little more strict about how intervals are half-open.
2012-11-29 15:35:11 -05:00
c0f87db3c1
Converted rbtree, interval to Cython. Serious speedups!
2012-11-29 15:13:09 -05:00
f39567b2bc
Speed updates
2012-11-29 01:35:01 -05:00
99ec0f4946
Converted rbtree.py to Cython
...
About 3x faster
2012-11-29 01:25:51 -05:00
f5c60f68dc
Speed tests.
...
test_interval_speed is about O(n * log n), which is good -- but the
constants are high and it hits swap on a 4G machine for the 2**21
test. Hopefully cython helps!
2012-11-29 01:00:54 -05:00
bdef0986d6
rbtree and interval tests fully pass now.
...
On to benchmarking...
2012-11-29 00:42:50 -05:00
c396c4dac8
rbtree tests complete
2012-11-29 00:07:49 -05:00
0b443f510b
Filling out rbtree tests, search routines
2012-11-28 20:57:23 -05:00
66fa6f3824
Add rendering test
2012-11-28 18:34:51 -05:00
7211217f40
Working on getting the RBTree working. Intersections are busted.
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11380 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-15 18:55:56 +00:00
d34b980516
RBTree seems generally OK now
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11379 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 20:10:43 +00:00
6aee52d980
Deletion is still broken. F.
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11378 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 04:23:53 +00:00
090c8d5315
More progress
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11377 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 04:12:15 +00:00
1042ff9f4b
add RBtree C++ example that I based this on; update tests
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11376 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-14 03:55:37 +00:00
f9dea53c24
Randomize order for the insertion test
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11358 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 23:50:23 +00:00
6278d32f7d
Passes tests, but is slow
...
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11356 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 23:08:01 +00:00
