Commit Graph

215 Commits

Author SHA1 Message Date
b77f07a4cd Fix reporting of parsing errors with malformed data
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
2013-03-12 16:44:36 -04:00
83bc5bc775 Make rocket/bulkdata errors include column number and the bad data
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
2013-03-12 16:10:00 -04:00
7eca587fdf Add 'nilmtool intervals' command, with --diff option
Can show the set-difference between the interval ranges in two
streams.
2013-03-11 17:07:26 -04:00
1d61d61a81 Add interval.set_difference function and associated tests 2013-03-11 15:40:50 -04:00
8e79998e95 Tune sqlite to use write-ahead-logging
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
2013-03-11 15:13:43 -04:00
9f914598c2 Make /stream/list give some more extended info, like row count
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
2013-03-11 15:13:43 -04:00
232a3876c2 Clean up imports to separate client and server more.
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
2013-03-11 15:13:42 -04:00
1c27dd72d6 Fill out client tests and fix various bugs
Fixes various corner cases and other bugs regarding lines with
comments, having data but no endpoints, etc.
2013-03-08 12:36:17 -05:00
0fc092779d Big rework of stream_insert_context and places that use it.
Things are now block-focused, rather than line-focused.  This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
2013-03-07 20:30:11 -05:00
92724d10ba Rework 'nilmtool insert' and some client stuff to speed up inserting data
Still needs work.
2013-03-06 20:49:14 -05:00
1d7acbf916 Remove null timestamper, speed up insert --none a tiny bit 2013-03-06 20:46:51 -05:00
ea3ea487bc Merge branch 'rocket-insert'
Conflicts:
	nilmdb/server/bulkdata.py
	nilmdb/server/server.py
	nilmdb/utils/__init__.py
2013-03-06 20:46:04 -05:00
69ad8c4842 Merge branch 'rocket' 2013-03-06 20:38:02 -05:00
65f09f793c When re-raising exceptions in the server, preserve original tracebacks 2013-03-05 21:48:40 -05:00
84e21ff467 Move ASCII data parsing from the server to the rocket interface.
The server buffers the string and passes it to nilmdb.  Nilmdb passes
the string to bulkdata.  Bulkdata uses the rocket interface to parse
it in chunks, as necessary.  Everything gets passed back up and
everyone is happy.

Currently, only pyrocket implements append_string.
2013-03-05 17:51:17 -05:00
11b228f77a Convert times to microsecond precision strings more consistently.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
2013-03-05 17:07:39 -05:00
454e561d69 Verify that metadata values are numbers or strings 2013-03-05 13:22:17 -05:00
58c0ae72f6 Support application/json POST bodies as well as x-www-form-urlencoded 2013-03-05 11:54:29 -05:00
c6a0e6e96f More complete CORS handling, including preflight requests (hopefully) 2013-03-04 15:40:35 -05:00
79755dc624 Fix Allow: header by switching to cherrypy's built in tools.allow().
Replaces custom tools.allow_methods which didn't return the Allow: header.
2013-03-04 14:08:37 -05:00
0d372fb878 Modify old formatter to match rocket's formatting style 2013-03-03 21:50:29 -05:00
0b75da7a8f Normalize the floating point formats to %.6e and %.16e
This is mostly a matter of taste, but it matches more closely with the
old way that prep did it, and it's more consistent.  It should roughly
match the available precision of floats and doubles.
2013-03-03 21:43:04 -05:00
e318888a06 Finish Rocket.extract_string; clean up code for other functions too
This is maybe 2.5-3 times faster than the list-based code, which
still isn't amazing, but is decent.
2013-03-03 21:25:00 -05:00
9d0d2415be Test bulkdata a little more carefully 2013-03-03 14:00:00 -05:00
4406d51a98 First pass at Python implementation of rocket 2013-03-03 13:37:02 -05:00
9b6de6ecb7 Replace old layout strings everywhere 2013-03-03 13:37:02 -05:00
28310fe886 Add test for extents 2013-03-02 15:19:25 -05:00
521ff88f7c Support 'nilmtool help command' just like 'nilmtool command --help' 2013-03-02 13:56:03 -05:00
64897a1dd1 Change port from 12380 -> 32180 when running tests
This is so tests can be run without interfering with a normal server.
2013-03-02 13:19:44 -05:00
5db3b186a4 Make test_mustclose more complete 2013-03-01 16:30:22 -05:00
ca67c79fe4 Improve test_layout_speed 2013-03-01 16:04:10 -05:00
8917bcd4bf Fix test case failures due to increased client chunk size 2013-03-01 16:04:00 -05:00
e3f335dfe5 Move time parsing from cmdline into nilmdb.utils.time 2013-02-28 17:09:26 -05:00
3df96fdfdd Reorder code 2013-02-26 19:41:55 -05:00
740ab76eaf Re-add persistent connection test for Requests based httpclient 2013-02-26 19:41:27 -05:00
ce13a47fea Save full response object for tests 2013-02-26 17:45:41 -05:00
50a4a60786 Replace pyCurl with Requests
Only tested with v1.1.0.  It's not clear how well older versions will
work.
2013-02-26 17:45:40 -05:00
14afa02db6 Temporarily remove curl-specific keepalive tests 2013-02-26 17:45:40 -05:00
cc990d6ce4 Test persistent connections 2013-02-26 13:41:40 -05:00
a4a4bc61ba Switch to using pycurl.Multi instead of Iteratorizer 2013-02-25 21:05:01 -05:00
3d82888580 Enforce method types, and require POST for actions that change things.
This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
2013-02-25 21:05:01 -05:00
749b878904 Add an explicit lock to httpclient's public methods
This is to prevent possible reentrancy problems.
2013-02-25 18:06:00 -05:00
317c53ab6f Improve serializer_proxy and verify_thread_proxy
These functions can now take an object or a type (class).

If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
2013-02-23 14:28:37 -05:00
422317850e Replace threadsafety class decorator version, add explicit proxy version
Like the serializer changes, the class decorator was too fragile.
2013-02-23 11:25:40 -05:00
965537d8cb Implement verify_thread_safety to check for unsafe access patterns
Occasional segfaults may be the result of performing thread-unsafe
operations.  This class decorator verifies that all of its methods
are called in a thread-safe manner.

It can separately warn about:
- two threads calling methods in a function (the kind of thing sqlite
  doesn't like)
- recursion
- concurrency (two different threads functions at the same time)
2013-02-23 11:25:02 -05:00
7fce305a1d Make server check that the db object has been wrapped in a serializer
It's only the server that calls it in multiple threads.
2013-02-23 11:25:01 -05:00
dfbbe23512 Switch to explicitly wrapping nilmdb objects in a serializer_proxy
This is quite a bit simpler than the class decorator method, so it
may be more reliable.
2013-02-23 11:23:54 -05:00
7761a91242 Remove class decorator version of the serializer; it's too fragile 2013-02-23 11:23:54 -05:00
9b06e46bf1 Add back a proxy version of the Serializer, which is much simpler. 2013-02-23 11:23:54 -05:00
171e6f1871 Replace "serializer" function with a "serialized" decorator
This decorator makes a class always be serialized, including its
instantiation, in a separate thread.  This is an improvement over
the old Serializer() object wrapper, which didn't put the
instantiation into the new thread.
2013-02-23 11:23:54 -05:00