Browse Source

Add some notes about rocket interface to design.md

tags/nilmdb-1.3.0
Jim Paris 11 years ago
parent
commit
31e2c7c8b4
1 changed files with 37 additions and 0 deletions
  1. +37
    -0
      docs/design.md

+ 37
- 0
docs/design.md View File

@@ -266,3 +266,40 @@ Each table contains:
from the end of the file will not shorten it; it will only be
deleted when it has been fully filled and all of the data has been
subsequently removed.


Rocket
------

Original design had the nilmdb.nilmdb thread (through bulkdata)
convert from on-disk layout to a Python list, and then the
nilmdb.server thread (from cherrypy) converts to ASCII. For at least
the extraction side of things, it's easy to pass the bulkdata a layout
name instead, and have it convert directly from on-disk to ASCII
format, because this conversion can then be shoved into a C module.
This module, which provides a means for converting directly from
on-disk format to ASCII or Python lists, is the "rocket" interface.
Python is still used to manage the files and figure out where the
data should go; rocket just puts binary data directly in or out of
those files at specified locations.

Before rocket, testing speed with uint16_6 data, with an end-to-end
test (extracting data with nilmtool):

- insert: 65 klines/sec
- extract: 120 klines/sec

After switching to the rocket design, but using the Python version
(pyrocket):

- insert: (hasn't yet been switched completely; more complicated due
to error handling when parsing ASCII)
- extract: 120 klines/sec

After switching to a C extension module (rocket.c)

- insert:
- extract:




Loading…
Cancel
Save