Jim Paris
517b237636
tests: fix test databases
2 years ago
Jim Paris
07f138e0f4
flake8: fix style
2 years ago
Jim Paris
7538c6201b
tests: new fsck tests for interval overlap
2 years ago
Jim Paris
4d9a106ca1
fsck: add fix for file position (row) overlap in database
The following sequence could lead to this corruption:
(1) Append new data to bulkdata
(2) Update interval file positions in SQL
(3) Flush (2)
(4) Crash before flushing (1)
(5) Reload database without running fsck
(6) Start writing new data to end of bulkdata and introduce new interval
2 years ago
Jim Paris
e90a79ddad
fsck: increase max restarts from 100 to 1000
This is effectively the number of problems with the database that can
be fixed, since we restart fsck after each one.
2 years ago
Jim Paris
7056c5b4ec
tests: new fsck tests
2 years ago
Jim Paris
df4e7f0967
fsck: If data timestamps are unexpectedly zero, truncate data
This probably means that the file was written, and metadata was journaled,
but the system crashed before data was written. If that happens, the
end of the file will be zeroed out. We don't bother checking the entire
file here; if we see just one timestamp that is unexpectedly zero, let's
truncate the data there.
2 years ago
Jim Paris
b6bba16505
fsck: fix error in reporting row number for timestamp errors
Since we process in chunks, we need to add "start" to the row number;
however, we don't need to use this when accessing data['timestamp']
(aka ts)
2 years ago
Jim Paris
d4003d0d34
tests: fill out coverage for new fsck features
2 years ago
Jim Paris
759492298a
bulkdata: write _format file atomically, to reduce chance of corruption
Some databases were seeing _format truncated to 0 bytes, after crashing
soon after a new stream was created (e.g. during decimation).
2 years ago
Jim Paris
b5f6fcc253
fsck: detect problems with _format file, and remove stream if empty
A bulkdata dir may get created for a new stream with an empty or
corrupted _format, before any data gets actually written. In that
case, we can just delete the new stream; worst case, we lose some
metadata.
Note: The info in _format should really get moved into the database.
This was born when bulkdata switched from PyTables to a custom storage
system, and was probably stored this way to avoid tying the main DB
to specific implementation details while they were still in flux.
2 years ago
Jim Paris
905e325ded
fsck: add fixer that fully removes a stream
2 years ago
Jim Paris
648b6f4b70
fsck: improve instructions about removing leftover data
2 years ago
Jim Paris
7f8a2c7027
fsck: remove unnecessary raises
The called functions already raise RetryFsck.
2 years ago
Jim Paris
276fbc652a
Bump Python version requirement
2 years ago
Jim Paris
10b34f5937
Fix issue with test suite and empty dirs on git
Git doesn't save empty dirs, so put a placeholder there that is
ignored when the test data is copied to its final location during
testing.
2 years ago
Jim Paris
83daeb148a
Add fsck scan for any data timestamps outside interval range
2 years ago
Jim Paris
d65f00e8b2
Add fsck to default tests
2 years ago
Jim Paris
71dc01c9a7
Replace deprecated numpy.fromstring usage
2 years ago
Jim Paris
bcd21b3498
Improve fsck test coverage to 100%
2 years ago
Jim Paris
a1dee0e6f2
Improve fsck test coverage to 85%
2 years ago
Jim Paris
99ac47cf0d
Start implementing fsck test and porting fsck to Python 3
2 years ago
Jim Paris
4cdaef51c1
Fix flake8-reported issues with fsck
2 years ago
Jim Paris
88466dcafe
Add yappi dependency for "nilmdb-server -y", but don't require ipython
2 years ago
Jim Paris
8dfb8da15c
Freeze requirements to specific versions
2 years ago
Jim Paris
6cc1f6b7b2
Fix #! at top of shell scripts for py3 and venvs
2 years ago
Jim Paris
8dc36c2d37
Fix stream_insert_context docstring
This was never updated when timestamps were changed from floats to
ints.
2 years ago
Jim Paris
3738430103
Fix flake8 warnings
2 years ago
Jim Paris
a41111b045
Fix some Python 3.8 related issues
2 years ago
Jim Paris
85f822e1c4
Decode non-JSON HTTP responses when possible
This doesn't affect anything in nilmdb, but is needed by nilmrun.
2 years ago
Jim Paris
0222dfebf0
Update git URL
2 years ago
Jim Paris
70914690c1
Update README for Python 3.8 and newer
2 years ago
Jim Paris
10400f2b07
rocket: suppress build warnings
2 years ago
Jim Paris
56153ff7ad
Update installation instructions
3 years ago
Jim Paris
671f87b047
Clean up README
3 years ago
Jim Paris
2f2faeeab7
Add pylint config and selectively fix some pylint errors
Some pylint errors are worth fixing; many are not.
3 years ago
Jim Paris
2ed544bd30
Fix flake8-reported code style issues
3 years ago
Jim Paris
6821b2a97b
Merge branch 'py3'
3 years ago
Jim Paris
b20bb92988
Improve test coverage and remove the last "#pragma: no cover"
We now have full coverage of the main code.
3 years ago
Jim Paris
699de7b11f
Change fast shutdown timeout to 0
This used to have problems with older CherryPy versions, but the
current one seems to handle it just fine.
3 years ago
Jim Paris
ea67e45be9
Clean up how we handle cherrypy's calls of os._exit(70)
With this solution, we can catch it cleanly in the standalone
nilmdb-server, and test the error paths in our normal test suite.
3 years ago
Jim Paris
ca440a42bd
Fix nilmdb-server argument parsing, logging, and profiling
It works again now.
3 years ago
Jim Paris
4ff4b263b4
Fill out code coverage for nilmdb/server/bulkdata.py
3 years ago
Jim Paris
79e544c733
Fix python2->3 conversion of pickle
3 years ago
Jim Paris
9acf99ff25
Fill out coverage for server/nilmdb.py and remove dead code
3 years ago
Jim Paris
4958a5ab2e
Improve test coverage
3 years ago
Jim Paris
f2d89e2da5
Remove very outdated pytables tests
3 years ago
Jim Paris
1952f245c0
Try to clean up some issues with cherrypy startup and os._exit
This is hard and finicky to test, so there's unfortunately not a clear
way to get 100% test coverage in cherrypy_start. However, that
function is only used in the test suite and the standalone
nilmdb-server script, not in production (which goes through wsgi and
skips all this cherrypy server stuff entirely).
3 years ago
Jim Paris
7cbc0c11c3
Run cherrypy always in "embedded" mode
Non-embedded mode is not used in the test suite or wsgi server;
it was an option in the standalong nilmdb-server script, but it's
really not necessary, and removing it gets rid of some untested
code.
3 years ago
Jim Paris
9f2651c35e
Stop ignoring errors when disabling SIGPIPE
3 years ago