Add some notes about rocket interface to design.md

This commit is contained in:
Jim Paris 2013-03-03 14:43:16 -05:00
parent 2a725ee13f
commit 31e2c7c8b4

View File

@ -266,3 +266,40 @@ Each table contains:
from the end of the file will not shorten it; it will only be
deleted when it has been fully filled and all of the data has been
subsequently removed.
Rocket
------
Original design had the nilmdb.nilmdb thread (through bulkdata)
convert from on-disk layout to a Python list, and then the
nilmdb.server thread (from cherrypy) converts to ASCII. For at least
the extraction side of things, it's easy to pass the bulkdata a layout
name instead, and have it convert directly from on-disk to ASCII
format, because this conversion can then be shoved into a C module.
This module, which provides a means for converting directly from
on-disk format to ASCII or Python lists, is the "rocket" interface.
Python is still used to manage the files and figure out where the
data should go; rocket just puts binary data directly in or out of
those files at specified locations.
Before rocket, testing speed with uint16_6 data, with an end-to-end
test (extracting data with nilmtool):
- insert: 65 klines/sec
- extract: 120 klines/sec
After switching to the rocket design, but using the Python version
(pyrocket):
- insert: (hasn't yet been switched completely; more complicated due
to error handling when parsing ASCII)
- extract: 120 klines/sec
After switching to a C extension module (rocket.c)
- insert:
- extract: