Jim Paris
aaea105861
Consolidate most timestamp <-> string conversions (outside of rocket)
11 years ago
Jim Paris
e6a081d639
Consolidate timestamp constants into nilmdb.utils.time
11 years ago
Jim Paris
1835d03412
Bump bulkdata database version to 3
11 years ago
Jim Paris
c7a712d8d8
Partial test for rounding issues
11 years ago
Jim Paris
20d315b4f7
Add documentation about upcoming timestamp changes
11 years ago
Jim Paris
a44a5e3135
Merge branch 'argcomplete'
11 years ago
Jim Paris
039b2a0557
Include nilmtool-bash-completion.sh script in .tar.gz
11 years ago
Jim Paris
cd1dfe7dcd
Add completion functions to most commandline arguments
11 years ago
Jim Paris
fb35517dfa
Add basic argument completion
11 years ago
Jim Paris
b9f0b35bbe
Stream renaming support, and comprehensive tests
Implemented in command line, client, server, nilmdb, bulkdata
11 years ago
Jim Paris
b1b09f8cd0
Strengthen checks when creating paths, fix some bugs, and add tests
11 years ago
Jim Paris
d467df7980
Add specific error for creating a path that already exists
11 years ago
Jim Paris
09bc7eb48c
Make StreamInserter.insert complain if data remains after send
Previously, we ignored problems when sending intermediate blocks,
since getting more data might make the next attempt succeed.
But in practice, malformed data would just build up, causing
problems. Raise an exception if there's too much data remaining
after trying to send an intermediate block.
11 years ago
Jim Paris
b77f07a4cd
Fix reporting of parsing errors with malformed data
strtod() and friends will happily swallow newlines, so we need to skip
over spaces and look for that sort of thing manually.
11 years ago
Jim Paris
59f0076306
Increase max layout count in rocket
11 years ago
Jim Paris
83bc5bc775
Make rocket/bulkdata errors include column number and the bad data
The bad line is printed out on a new line, and a third line
with a ^ to indicate the position of the error.
11 years ago
Jim Paris
6b1dfec828
In stream_list, return 0 instead of None for rows and seconds
For rows and seconds only. Extents still give None if they don't
exist.
11 years ago
Jim Paris
d827f41fa5
Fix Makefile omission
11 years ago
Jim Paris
7eca587fdf
Add 'nilmtool intervals' command, with --diff option
Can show the set-difference between the interval ranges in two
streams.
11 years ago
Jim Paris
a351bc1b10
Add client, server, nilmdb support for listing interval differences
11 years ago
Jim Paris
1d61d61a81
Add interval.set_difference function and associated tests
11 years ago
Jim Paris
755255030b
Clean up interval __and__ function; we don't need to __and__ sets
11 years ago
Jim Paris
8e79998e95
Tune sqlite to use write-ahead-logging
Enable the following pragmas: synchronous=NORMAL, journal_mode=WAL.
This offers a significant speedup to INSERT times compared to
synchronous=FULL, and is roughly the same as synchronous=OFF
but should be a bit safer.
11 years ago
Jim Paris
9f914598c2
Make /stream/list give some more extended info, like row count
Also changes the HTTP parameter from "extent" to "extended",
and the commandline parameter from "extent" to "ext".
11 years ago
Jim Paris
0468b04538
Fix pyrocket to handle comments better
11 years ago
Jim Paris
232a3876c2
Clean up imports to separate client and server more.
"import nilmdb" doesn't do much; "import nilmdb.client" or "import
nilmdb.server" is now required.
11 years ago
Jim Paris
1c27dd72d6
Fill out client tests and fix various bugs
Fixes various corner cases and other bugs regarding lines with
comments, having data but no endpoints, etc.
11 years ago
Jim Paris
de5e474001
Update benchmarks in design.md
11 years ago
Jim Paris
0fc092779d
Big rework of stream_insert_context and places that use it.
Things are now block-focused, rather than line-focused. This should
give a pretty big speedup to inserting client data, especially when
inserting preformatted data.
11 years ago
Jim Paris
7abfdfbf3e
Add const qualifier to strings we get from Python
11 years ago
Jim Paris
92724d10ba
Rework 'nilmtool insert' and some client stuff to speed up inserting data
Still needs work.
11 years ago
Jim Paris
1d7acbf916
Remove null timestamper, speed up insert --none a tiny bit
11 years ago
Jim Paris
ea3ea487bc
Merge branch 'rocket-insert'
Conflicts:
nilmdb/server/bulkdata.py
nilmdb/server/server.py
nilmdb/utils/__init__.py
11 years ago
Jim Paris
69ad8c4842
Merge branch 'rocket'
11 years ago
Jim Paris
0047e0360a
Implement Rocket.append_string() in C; misc cleanups along the way
This should more or less complete the rocket interface.
11 years ago
Jim Paris
1ac6abdad0
Fix rocket.ParseError exception handling
Before, a tuple was crammed into args[0]. Now, the three arguments are
args[0:2].
11 years ago
Jim Paris
65f09f793c
When re-raising exceptions in the server, preserve original tracebacks
11 years ago
Jim Paris
84e21ff467
Move ASCII data parsing from the server to the rocket interface.
The server buffers the string and passes it to nilmdb. Nilmdb passes
the string to bulkdata. Bulkdata uses the rocket interface to parse
it in chunks, as necessary. Everything gets passed back up and
everyone is happy.
Currently, only pyrocket implements append_string.
11 years ago
Jim Paris
11b228f77a
Convert times to microsecond precision strings more consistently.
Use a new helper, nilmdb.utils.time.float_to_time_string().
This will help if we ever want to change representation (like using
uint64 microseconds since epoch, which saves us from having to
waste bits on the floating-point exponent)
11 years ago
Jim Paris
7860a6aefb
Make helper for removing or truncating a file; use it
11 years ago
Jim Paris
454e561d69
Verify that metadata values are numbers or strings
11 years ago
Jim Paris
fe91ff59a3
Better handling of JSON requests
11 years ago
Jim Paris
64c24a00d6
Add --traceback argument to nilmdb-server script
11 years ago
Jim Paris
58c0ae72f6
Support application/json POST bodies as well as x-www-form-urlencoded
11 years ago
Jim Paris
c5f079f61f
When removing data from files, try to punch a hole.
Requires fallocate(2) support with FALLOC_FL_PUNCH_HOLE, as
well as a filesystem that supports it (in Linux 3.7,
tmpfs, btrfs, xfs, or ext4)
11 years ago
Jim Paris
16f23f4a91
Fill out pyrocket.py to fit new interfaces; fix small bugs
11 years ago
Jim Paris
b0f12d55dd
Fully replace bulkdata.File with rocket.Rocket
11 years ago
Jim Paris
8a648c1b97
Move towards replacing bulkdata.File with rocket.Rocket
There isn't much left in File, so let's move as much as possible
over to C.
11 years ago
Jim Paris
2d45466f66
Print version at server startup
11 years ago
Jim Paris
c6a0e6e96f
More complete CORS handling, including preflight requests (hopefully)
11 years ago