Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
Things are now block-focused, rather than line-focused. This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
The server buffers the string and passes it to nilmdb. Nilmdb passes
the string to bulkdata. Bulkdata uses the rocket interface to parse
it in chunks, as necessary. Everything gets passed back up and
everyone is happy.
Currently, only pyrocket implements append_string.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
This is mostly a matter of taste, but it matches more closely with the
old way that prep did it, and it's more consistent. It should roughly
match the available precision of floats and doubles.
This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
These functions can now take an object or a type (class).
If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
Occasional segfaults may be the result of performing thread-unsafe
operations. This class decorator verifies that all of its methods
are called in a thread-safe manner.
It can separately warn about:
- two threads calling methods in a function (the kind of thing sqlite
doesn't like)
- recursion
- concurrency (two different threads functions at the same time)
This decorator makes a class always be serialized, including its
instantiation, in a separate thread. This is an improvement over
the old Serializer() object wrapper, which didn't put the
instantiation into the new thread.
Previously, we could get empty intervals anyway by having a non-empty
interval and removing a smaller interval around each piece of data.
Turns out that empty intervals are OK and needed in some situations,
so explicitly allow and test for it.
Now nilmdb.client, nilmdb.server, nilmdb.cmdline, and nilmdb.utils
are each their own modules, and there is a little bit more of a
logical separation between them. Various changes scattered throughout
to fix naming (for example, nilmdb.nilmdb.NilmDBError is now
nilmdb.server.errors.NilmDBError).
Reduced usage of "from __future__ import absolute_import" as much
as possible. It's still needed for the functions in the nilmdb/server
directory to be able to import the nilmdb module rather than the
nilmdb.py script.
This should hopefully ease future packaging a bit.
Inside the pycurl callback, we can't raise exceptions, because the
pycurl extension module will unconditionally print the exception
itself, and not pass it up to the caller. Instead, we have the
callback return a value that tells curl to abort. (-1 would be best,
in case we were given 0 bytes, but the extension doesn't support
that either).
This resolves the 'Exception("should die")' problem when interrupting
a streaming generator like stream_extract.
Use numbered subdirectories to avoid having too many files in one dir.
Add appropriate tests.
Also fix an issue where the mmap_open LRU cache could inappropriately
open a file twice because it was using the optional "newsize"
parameter as a key -- now lrucache can be given a slice object that
describes which arguments are important.
We replace cStringIO with StringIO subclass that forces UTF-8
encoding, and explicitly convert commandlines to UTF-8 before
shlex. These changes will only affect tests, not normal commandline
operation.
Spent way too long trying to track down a cryptic error that turned
out to be a CherryPy bug. Now we catch this using a decorator in the
'extract' and 'intervals' generators that transforms exceptions that
trigger the bugs into one that does not. fun!
There's some bug with the testing harness where placing e.g.
from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du". Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
