Commit Graph

74 Commits

Author SHA1 Message Date
64897a1dd1 Change port from 12380 -> 32180 when running tests
This is so tests can be run without interfering with a normal server.
2013-03-02 13:19:44 -05:00
8917bcd4bf Fix test case failures due to increased client chunk size 2013-03-01 16:04:00 -05:00
3df96fdfdd Reorder code 2013-02-26 19:41:55 -05:00
740ab76eaf Re-add persistent connection test for Requests based httpclient 2013-02-26 19:41:27 -05:00
ce13a47fea Save full response object for tests 2013-02-26 17:45:41 -05:00
50a4a60786 Replace pyCurl with Requests
Only tested with v1.1.0.  It's not clear how well older versions will
work.
2013-02-26 17:45:40 -05:00
14afa02db6 Temporarily remove curl-specific keepalive tests 2013-02-26 17:45:40 -05:00
cc990d6ce4 Test persistent connections 2013-02-26 13:41:40 -05:00
a4a4bc61ba Switch to using pycurl.Multi instead of Iteratorizer 2013-02-25 21:05:01 -05:00
3d82888580 Enforce method types, and require POST for actions that change things.
This is a pretty big change that will render existing clients unable
to modify the database, but it's important that we use POST or PUT
instead of GET for anything that may change state, in case this
is ever put behind a cache.
2013-02-25 21:05:01 -05:00
749b878904 Add an explicit lock to httpclient's public methods
This is to prevent possible reentrancy problems.
2013-02-25 18:06:00 -05:00
317c53ab6f Improve serializer_proxy and verify_thread_proxy
These functions can now take an object or a type (class).

If given an object, they will wrap subsequent calls to that object.
If given a type, they will return an object that can be instantiated
to create a new object, and all calls including __init__ will be
covered by the serialization or thread verification.
2013-02-23 14:28:37 -05:00
dfbbe23512 Switch to explicitly wrapping nilmdb objects in a serializer_proxy
This is quite a bit simpler than the class decorator method, so it
may be more reliable.
2013-02-23 11:23:54 -05:00
1431e41d16 Allow inserting empty intervals in the database, and add tests for it.
Previously, we could get empty intervals anyway by having a non-empty
interval and removing a smaller interval around each piece of data.
Turns out that empty intervals are OK and needed in some situations,
so explicitly allow and test for it.
2013-02-21 14:07:35 -05:00
a49c655816 Strictly enforce (start < end) for all intervals.
Previously, we allowed start == end, but this doesn't make sense with
half-open intervals.
2013-02-21 14:06:40 -05:00
ca5253ddee Fix and test stream_count 2013-02-19 18:26:44 -05:00
e19da84b2e server: always return None instead of sometimes returning "ok"
Previously some functions returned the string "ok".
2013-02-19 18:26:44 -05:00
3e8e3542fd Test for detecting nested HTTP requests 2013-02-19 18:26:44 -05:00
bba9ad131e Add test for client.stream_insert_context 2013-02-19 17:19:45 -05:00
ee24380d1f Replace duplicated URL in tests with a variable 2013-02-19 15:27:51 -05:00
bfcd91acf8 client tests: renumber 2013-02-19 15:25:34 -05:00
b53ff31212 client: Add must_close() decorator to nilmdb.Client, and fix tests
Test suite wasn't closing connections correctly.
2013-02-16 18:55:23 -05:00
2045e89f24 client: Add context manager functionality, test closing 2013-02-16 18:55:20 -05:00
1593e181a3 Switch to versioneer-provided versions everywhere 2013-02-05 19:07:38 -05:00
f316026592 Move datetime_tz package under nilmdb.utils
datetime_tz isn't readily available, so it's a lot easier to just
package it within the nilmdb tree.
2013-01-30 19:03:42 -05:00
225003f412 Huge cleanup of namespaces, modules, packages, imports.
Now nilmdb.client, nilmdb.server, nilmdb.cmdline, and nilmdb.utils
are each their own modules, and there is a little bit more of a
logical separation between them.  Various changes scattered throughout
to fix naming (for example, nilmdb.nilmdb.NilmDBError is now
nilmdb.server.errors.NilmDBError).

Reduced usage of "from __future__ import absolute_import" as much
as possible.  It's still needed for the functions in the nilmdb/server
directory to be able to import the nilmdb module rather than the
nilmdb.py script.

This should hopefully ease future packaging a bit.
2013-01-28 19:04:52 -05:00
5cd7899e98 Send a Access-Control-Allow-Origin (CORS) header with all responses 2013-01-22 14:42:03 -05:00
ceec5fb9b3 Force /stream/interval and /stream/extract responses to be text/plain 2013-01-22 12:47:06 -05:00
62354b4dce Add test for bad-parameters-give-500-error 2013-01-17 19:58:48 -05:00
4f6a742e6c Fix test failure 2013-01-16 17:31:31 -05:00
fe3b087435 Remove implemented in nilmdb; still needs bulkdata changes. 2013-01-08 21:07:52 -05:00
3d212e7592 Move test helpers into subdirectory 2013-01-05 15:00:34 -05:00
493bbed82c More coverage and tests 2013-01-03 19:21:12 -05:00
c083d63c96 Tests for Unicode compliance 2013-01-03 17:03:52 -05:00
f355c73209 Refactor utility classes into nilmdb.utils subdir/namespace
There's some bug with the testing harness where placing e.g.
  from du import du
in nilmdb/utils/__init__.py doesn't quite work -- sometimes the
module "du" replaces the function "du".  Not exactly sure why;
we work around that by just renaming files so they don't match
the imported names directly.
2012-12-31 15:55:36 -05:00
9082cc9f44 Merging adjacent intervals is working now!
Adjust test expectations accordingly, since the number of intervals
they print out will now be smaller.
2012-12-12 19:25:27 -05:00
32dbeebc09 More insertion checks. Need to get interval concatenation working. 2012-12-11 18:08:00 -05:00
66ddc79b15 Inserting works again, with proper end/start for paired blocks.
timeit.sh script works too!
2012-12-07 20:30:39 -05:00
ea3e92be3f Some prep work for merging adjacent insertions.
Doesn't actually merge them yet; need to change Interval
implementation to allow deletes.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@11354 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-11-08 21:27:52 +00:00
97bec3b1ee Optimization that uses slices on the table rather than checking each
row individually, when extracting data.

Switch to using bisect module when doing the bisection, to lessen the
chance of errors.

Added syslog ability for timer module, for timing stuff deep inside
the server.

Make the chunked/non-chunked test just give a warning, rather than
failing the tests, for debugging purposes.  Alternate approach would
be to disable "die on error" for the tests.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10896 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-06-04 23:46:33 +00:00
27f8dcf06d Extract works now and we're back to 100% test coverage! It's slow,
though; need to figure out where the slowdown lies.

Add stream existence check to server's /intervals and /extract paths,
add tests for it.

Make start and end arguments optional for /extract, like /intervals

Move --quiet command line option to just the insert subcommand.
It's the only one that uses it right now, and otherwise it doesn't
show up in after a "nilmtool.py intervals --help".  Might revisit this
later if more commands start supporting --quiet.

Change cmdline/extract's write into a print, to keep the trailing
newline.

Fix lingering uses of Interval in nilmdb and change to DBInterval
instead.

Fix nilmdb interval bisection:
  - handle common case optimization correctly
  - db_endpos is always one after the last row, so use hi=db_endpos-1

Finish nlimdb stream_extract

Add a bunch of cmdline tests for extract, particularly testing border
cases around start/end.  Compares output to a set of files stored in
the tests/data dir.

Some more tests in test_client to get better coverage.



git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10893 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-30 18:24:36 +00:00
e2daeb5e54 - Some updates to max_results handling on server side
- Flesh out tests for the new nilmdb.layout.Formatter
  Coverage doesn't handle the cython module, so this is just
  functional stuff, not necessarily complete.
  Still need to finish each Layout.format()

- Split out test_client_5_chunked from test_client_4_misc
  so it's easier to skip while debugging.  Turning off streaming
  lets us see tracebacks from within the server's content()
  functions.

- More work on stream/extract in cmdline, client, server, nilmdb.
  Still needs work on server side, but should be complete in nilmdb.

- Start nilmdb.layout.Formatter class


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10888 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-25 16:44:24 +00:00
cbc7c5125d WIP on getting stream extract to work.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10886 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-24 21:05:38 +00:00
27fd9d54f9 Streaming responses from the server weren't actually streaming. Fix
that.  This requires a bit of restructuring of server.py:intervals()
to allow us to properly report errors before beginning the stream.

Make the nilmdb.httpclient save a copy of HTTP header responses, and
add a test that the saved responses to verify that the
transfer-encoding is Chunked for the /stream/interval request.  This
should check that the above bug is fixed and doesn't show up again
if we switch to a different WSGI server, etc.

Tweak size estimates in nilmdb for /stream/interval


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10884 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 23:18:24 +00:00
315bc57ac3 More complete test coverage of nilmdb.httpclient, yay
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10882 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-23 20:00:01 +00:00
c327378373 Split MyCurl into a HTTPClient class instead
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10879 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-21 21:50:01 +00:00
aaffd61e4e rework MyCurl stuff
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10878 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-21 21:43:30 +00:00
3f436e6dfd Use simplejson instead of json for performance
Misc cleanups

More work on tests to get more coverage


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10869 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-05-16 22:19:00 +00:00
ac32647fac Remove PyTables indexing support; it's not necessary since we're just
using pytables as a row-indexed data store.


git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10754 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-13 20:51:18 +00:00
f3b0dfabea More work on commandline and commandline tests.
git-svn-id: https://bucket.mit.edu/svn/nilm/nilmdb@10682 ddd99763-3ecb-0310-9145-efcb8ce7c51f
2012-04-06 18:25:09 +00:00