Browse Source

Add some info about binary interface to design docs

Jim Paris 11 years ago
1 changed files with 32 additions and 0 deletions
  1. +32

+ 32
- 0
docs/ View File

@@ -389,3 +389,35 @@ Possible solutions:
are always printed as int64 values, and a new format
"@1234567890123456" is added to the parser for specifying them

Binary interface

The ASCII interface is too slow for high-bandwidth processing, like
sinefits, prep, etc. A binary interface was added so that you can
extract the raw binary out of the bulkdata storage. This binary is
a little-endian format, e.g. in C a uint16_6 stream would be:

#include <endian.h>
#include <stdint.h>
struct {
int64_t timestamp_le;
uint16_t data_le[6];
} __attribute__((packed));

Remember to byteswap (with e.g. `letoh` in C)!

This interface is used by the new `nilmdb.client.numpyclient.NumpyClient`
class, which is a subclass of the normal `nilmcb.client.client.Client`
and has all of the same functions. It adds three new functions:

- `stream_extract_numpy` to extract data as a Numpy array

- `stream_insert_numpy` to insert data as a Numpy array

- `stream_insert_numpy_context` is the context manager for
incrementally inserting data

It is significantly faster! It is about 20 times faster to decimate a
stream with `nilm-decimate` when the filter code is using the new
binary/numpy interface.