09bc7eb48c
Make StreamInserter.insert complain if data remains after send
...
Previously, we ignored problems when sending intermediate blocks,
since getting more data might make the next attempt succeed.
But in practice, malformed data would just build up, causing
problems. Raise an exception if there's too much data remaining
after trying to send an intermediate block.
2013-03-12 18:45:56 -04:00
b77f07a4cd
Fix reporting of parsing errors with malformed data
...
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
2013-03-12 16:44:36 -04:00
59f0076306
Increase max layout count in rocket
2013-03-12 16:10:29 -04:00
83bc5bc775
Make rocket/bulkdata errors include column number and the bad data
...
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
2013-03-12 16:10:00 -04:00
6b1dfec828
In stream_list, return 0 instead of None for rows and seconds
...
For rows and seconds only. Extents still give None if they don't
exist.
2013-03-11 19:37:52 -04:00
d827f41fa5
Fix Makefile omission
2013-03-11 17:42:02 -04:00
7eca587fdf
Add 'nilmtool intervals' command, with --diff option
...
Can show the set-difference between the interval ranges in two
streams.
2013-03-11 17:07:26 -04:00
a351bc1b10
Add client, server, nilmdb support for listing interval differences
2013-03-11 17:07:08 -04:00
1d61d61a81
Add interval.set_difference function and associated tests
2013-03-11 15:40:50 -04:00
755255030b
Clean up interval __and__ function; we don't need to __and__ sets
2013-03-11 15:15:43 -04:00
8e79998e95
Tune sqlite to use write-ahead-logging
...
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
2013-03-11 15:13:43 -04:00
9f914598c2
Make /stream/list give some more extended info, like row count
...
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
2013-03-11 15:13:43 -04:00
0468b04538
Fix pyrocket to handle comments better
2013-03-11 15:13:43 -04:00
232a3876c2
Clean up imports to separate client and server more.
...
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
2013-03-11 15:13:42 -04:00
1c27dd72d6
Fill out client tests and fix various bugs
...
Fixes various corner cases and other bugs regarding lines with
comments, having data but no endpoints, etc.
2013-03-08 12:36:17 -05:00
de5e474001
Update benchmarks in design.md
2013-03-07 20:33:30 -05:00
0fc092779d
Big rework of stream_insert_context and places that use it.
...
Things are now block-focused, rather than line-focused. This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
2013-03-07 20:30:11 -05:00
7abfdfbf3e
Add const qualifier to strings we get from Python
2013-03-07 16:27:07 -05:00
92724d10ba
Rework 'nilmtool insert' and some client stuff to speed up inserting data
...
Still needs work.
2013-03-06 20:49:14 -05:00
1d7acbf916
Remove null timestamper, speed up insert --none a tiny bit
2013-03-06 20:46:51 -05:00
ea3ea487bc
Merge branch 'rocket-insert'
...
Conflicts:
nilmdb/server/bulkdata.py
nilmdb/server/server.py
nilmdb/utils/__init__.py
2013-03-06 20:46:04 -05:00
69ad8c4842
Merge branch 'rocket'
2013-03-06 20:38:02 -05:00
0047e0360a
Implement Rocket.append_string() in C; misc cleanups along the way
...
This should more or less complete the rocket interface.
2013-03-06 15:50:00 -05:00
1ac6abdad0
Fix rocket.ParseError exception handling
...
Before, a tuple was crammed into args[0]. Now, the three arguments are
args[0:2].
2013-03-05 22:05:17 -05:00
65f09f793c
When re-raising exceptions in the server, preserve original tracebacks
2013-03-05 21:48:40 -05:00
84e21ff467
Move ASCII data parsing from the server to the rocket interface.
...
The server buffers the string and passes it to nilmdb. Nilmdb passes
the string to bulkdata. Bulkdata uses the rocket interface to parse
it in chunks, as necessary. Everything gets passed back up and
everyone is happy.
Currently, only pyrocket implements append_string.
2013-03-05 17:51:17 -05:00
11b228f77a
Convert times to microsecond precision strings more consistently.
...
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
2013-03-05 17:07:39 -05:00
7860a6aefb
Make helper for removing or truncating a file; use it
2013-03-05 15:27:12 -05:00
454e561d69
Verify that metadata values are numbers or strings
2013-03-05 13:22:17 -05:00
fe91ff59a3
Better handling of JSON requests
2013-03-05 12:38:08 -05:00
64c24a00d6
Add --traceback argument to nilmdb-server script
2013-03-05 12:20:07 -05:00
58c0ae72f6
Support application/json POST bodies as well as x-www-form-urlencoded
2013-03-05 11:54:29 -05:00
c5f079f61f
When removing data from files, try to punch a hole.
...
Requires fallocate(2) support with FALLOC_FL_PUNCH_HOLE, as
well as a filesystem that supports it (in Linux 3.7,
tmpfs, btrfs, xfs, or ext4)
2013-03-04 20:31:14 -05:00
16f23f4a91
Fill out pyrocket.py to fit new interfaces; fix small bugs
2013-03-04 17:01:53 -05:00
b0f12d55dd
Fully replace bulkdata.File with rocket.Rocket
2013-03-04 16:43:26 -05:00
8a648c1b97
Move towards replacing bulkdata.File with rocket.Rocket
...
There isn't much left in File, so let's move as much as possible
over to C.
2013-03-04 16:28:40 -05:00
2d45466f66
Print version at server startup
2013-03-04 15:43:45 -05:00
c6a0e6e96f
More complete CORS handling, including preflight requests (hopefully)
2013-03-04 15:40:35 -05:00
79755dc624
Fix Allow: header by switching to cherrypy's built in tools.allow().
...
Replaces custom tools.allow_methods which didn't return the Allow: header.
2013-03-04 14:08:37 -05:00
f260f2c83d
Remove unnecessary layout argument to nilmdb.stream_extract
2013-03-04 11:09:54 -05:00
14402005bf
Remove extraneous flush
2013-03-03 21:52:45 -05:00
0d372fb878
Modify old formatter to match rocket's formatting style
2013-03-03 21:50:29 -05:00
5eac924118
Ignore built modules
2013-03-03 21:44:08 -05:00
0b75da7a8f
Normalize the floating point formats to %.6e and %.16e
...
This is mostly a matter of taste, but it matches more closely with the
old way that prep did it, and it's more consistent. It should roughly
match the available precision of floats and doubles.
2013-03-03 21:43:04 -05:00
2dfc94b566
Remove old code
2013-03-03 21:40:48 -05:00
e318888a06
Finish Rocket.extract_string; clean up code for other functions too
...
This is maybe 2.5-3 times faster than the list-based code, which
still isn't amazing, but is decent.
2013-03-03 21:25:00 -05:00
7c95934cc2
Add rocket.extract_list; still not as complete as pyrocket
2013-03-03 19:04:26 -05:00
96df9d8323
Starting the C version of rocket
...
Currently, only append_list is written (and hardly tested)
2013-03-03 16:54:11 -05:00
31e2c7c8b4
Add some notes about rocket interface to design.md
2013-03-03 14:43:16 -05:00
2a725ee13f
Add version 1 database format backwards compatibility
2013-03-03 14:37:58 -05:00
