Structure
---------
nilmdb.nilmdb is the NILM database interface.  It tracks a PyTables
database holds actual rows of data, and a SQL database tracks metadata
and ranges.

Access to the nilmdb must be single-threaded.  This is handled with
the nilmdb.serializer class.

nilmdb.server is a HTTP server that provides an interface to talk,
thorugh the serialization layer, to the nilmdb object.

nilmdb.client is a HTTP client that connects to this.

Sqlite performance
------------------

Committing a transaction in the default sync mode (PRAGMA synchronous=FULL)
takes about 125msec.  sqlite3 will commit transactions at 3 times:

  1: explicit con.commit()

  2: between a series of DML commands and non-DML commands, e.g.
   after a series of INSERT, SELECT, but before a CREATE TABLE or
   PRAGMA.

3: at the end of an explicit transaction, e.g. "with self.con as con:"

To speed up testing, or if this transaction speed becomes an issue,
the sync=False option to NilmDB will set PRAGMA synchronous=OFF.


Inserting streams
-----------------

We need to send the contents of "data" as POST.  Do we need chunked
transfer?

- Don't know the size in advance, so we would need to use chunked if
  we send the entire thing in one request.
- But we shouldn't send one chunk per line, so we need to buffer some
  anyway; why not just make new requests?
- Consider the infinite-streaming case, we might want to send it
  immediately?  Not really -- server still should do explicit inserts
  of fixed-size chunks.
- Even chunked encoding needs the size of each chunk beforehand, so
  everything still gets buffered.  Just a tradeoff of buffer size.

Before timestamps are added:
- Raw data is about 440 kB/s    (9 channels)
- Prep data is about 12.5 kB/s  (1 phase)
- How do we know how much data to send?

  - Remember that we can only do maybe 8-50 transactions per second on
    the sqlite database.  So if one block of inserted data is one
    transaction, we'd need the raw case to be around 64kB per request,
	ideally more.
  - Maybe use a range, based on how long it's taking to read the data
    - If no more data, send it
    - If data > 1 MB, send it
	- If more than 10 seconds have elapsed, send it
  - Should those numbers come from the server?

Converting from ASCII to PyTables:
- For each row getting added, we need to set attributes on a PyTables
  Row object and call table.append().  This means that there isn't a
  particularly efficient way of converting from ascii.
- Could create a function like nilmdb.layout.Layout("foo".fillRow(asciiline)
  - But this means we're doing parsing on the serialized side
  - Let's keep parsing on the threaded server side so we can detect
    errors better, and not block the serialized nilmdb for a slow
    parsing process.
- Client sends ASCII data
- Server converts this ACSII data to a list of values
  - Maybe:

		# threaded side creates this object
		parser = nilmdb.layout.Parser("layout_name")
		# threaded side parses and fills it with data
		parser.parse(textdata)
		# serialized side pulls out rows
		for n in xrange(parser.nrows):
		    parser.fill_row(rowinstance, n)
			table.append()


Inserting streams, inside nilmdb
--------------------------------

- First check that the new stream doesn't overlap.
  - Get minimum timestamp, maximum timestamp from data parser.
    - (extend parser to verify monotonicity and track extents)
  - Get all intervals for this stream in the database
  - See if new interval overlaps any existing ones
    - If so, bail
  - Question: should we cache intervals inside NilmDB?
    - Assume database is fast for now, and always rebuild fom DB.
	- Can add a caching layer later if we need to.
  - `stream_get_ranges(path)` -> return IntervalSet?

Speed
-----

- Currently abysmal

    $ time zcat /home/jim/bpnilm-data/snapshot-1-20110513-110002.raw.gz | ./nilmtool.py insert -s 20110513-1100 /bpnilm/1/raw 
	Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 11:00:00 EDT", rate=8000)

	real    24m31.093s
	user    2m32.846s
	sys     0m7.696s

  24 minutes for 1 hour of data.

  Interesting that "user" was so low.  What took the time?

  - If client was really only 2m32s, was it waiting on the server?


- Second run

	pilot:~/devl/lees/bucket/nilm/nilmdb$ time zcat /home/jim/bpnilm-data/snapshot-1-20110513-110002.raw.gz | ./nilmtool.py insert -s 20110513-120001 /bpnilm/1/raw 
	Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 12:00:01 EDT", rate=8000)

	real    43m44.528s
	user    2m29.265s
	sys     0m7.736s

- Third run

	pilot:~/devl/lees/bucket/nilm/nilmdb$ time zcat /home/jim/bpnilm-data/snapshot-1-20110513-110002.raw.gz | ./nilmtool.py insert -s 20110513-130002 /bpnilm/1/raw 
	Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 13:00:02 EDT", rate=8000)

	real    93m29.713s
	user    2m23.841s
	sys     0m6.928s

- Fourth run

    $ time zcat /home/jim/bpnilm-data/snapshot-1-20110513-110002.raw.gz | ./nilmtool.py insert -s 20110513-140003 /bpnilm/1/raw 
	Input file: -

    Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 14:00:03 EDT", rate=8000)
	real    166m53.007s
	user    2m22.037s
	sys     0m7.688s

- This is bad, must be slowing down due to pytables.
- Database also seems to be ballooning -- something like 1.1G after
  the first 3 inserts, 2.3 G afterwards
- Maybe due to indexing, need to see if disabling that is good.  Otherwise,
  ditch pytables.

- Disabled indexing, still bad:

    Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 11:00:00 EDT", rate=8000)
	
	real    31m21.492s
	user    2m35.206s
	sys     0m7.728s
	Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 12:00:01 EDT", rate=8000)
	
	real    52m51.963s
	user    2m40.902s
	sys     0m8.273s
	Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 13:00:02 EDT", rate=8000)
	
	real    102m8.151s
	user    2m38.970s
	sys     0m7.696s
	Input file: -
	Timestamper: TimestamperRate(..., start="Fri, 13 May 2011 14:00:03 EDT", rate=8000)
	
	real    176m12.469s
	user    2m37.538s
	sys     0m7.704s
	
- Server RAM usage isn't growing, but maybe there's some other bug there.  Restarting it:


    
