Add some info about binary interface to design docs

This commit is contained in:
Jim Paris 2013-04-08 18:46:20 -04:00
parent b4d6aad6de
commit 231963538e

View File

@ -389,3 +389,35 @@ Possible solutions:
are always printed as int64 values, and a new format
"@1234567890123456" is added to the parser for specifying them
exactly.
Binary interface
----------------
The ASCII interface is too slow for high-bandwidth processing, like
sinefits, prep, etc. A binary interface was added so that you can
extract the raw binary out of the bulkdata storage. This binary is
a little-endian format, e.g. in C a uint16_6 stream would be:
#include <endian.h>
#include <stdint.h>
struct {
int64_t timestamp_le;
uint16_t data_le[6];
} __attribute__((packed));
Remember to byteswap (with e.g. `letoh` in C)!
This interface is used by the new `nilmdb.client.numpyclient.NumpyClient`
class, which is a subclass of the normal `nilmcb.client.client.Client`
and has all of the same functions. It adds three new functions:
- `stream_extract_numpy` to extract data as a Numpy array
- `stream_insert_numpy` to insert data as a Numpy array
- `stream_insert_numpy_context` is the context manager for
incrementally inserting data
It is significantly faster! It is about 20 times faster to decimate a
stream with `nilm-decimate` when the filter code is using the new
binary/numpy interface.