517b237636
tests: fix test databases
2020-08-18 10:23:56 -04:00
07f138e0f4
flake8: fix style
2020-08-18 10:22:50 -04:00
7538c6201b
tests: new fsck tests for interval overlap
2020-08-18 10:18:54 -04:00
4d9a106ca1
fsck: add fix for file position (row) overlap in database
...
The following sequence could lead to this corruption:
(1) Append new data to bulkdata
(2) Update interval file positions in SQL
(3) Flush (2)
(4) Crash before flushing (1)
(5) Reload database without running fsck
(6) Start writing new data to end of bulkdata and introduce new interval
2020-08-18 10:04:33 -04:00
e90a79ddad
fsck: increase max restarts from 100 to 1000
...
This is effectively the number of problems with the database that can
be fixed, since we restart fsck after each one.
2020-08-18 10:03:57 -04:00
7056c5b4ec
tests: new fsck tests
2020-08-18 10:03:34 -04:00
df4e7f0967
fsck: If data timestamps are unexpectedly zero, truncate data
...
This probably means that the file was written, and metadata was journaled,
but the system crashed before data was written. If that happens, the
end of the file will be zeroed out. We don't bother checking the entire
file here; if we see just one timestamp that is unexpectedly zero, let's
truncate the data there.
2020-08-18 00:32:51 -04:00
b6bba16505
fsck: fix error in reporting row number for timestamp errors
...
Since we process in chunks, we need to add "start" to the row number;
however, we don't need to use this when accessing data['timestamp']
(aka ts)
2020-08-18 00:31:40 -04:00
d4003d0d34
tests: fill out coverage for new fsck features
2020-08-17 23:27:32 -04:00
759492298a
bulkdata: write _format file atomically, to reduce chance of corruption
...
Some databases were seeing _format truncated to 0 bytes, after crashing
soon after a new stream was created (e.g. during decimation).
2020-08-17 23:04:36 -04:00
b5f6fcc253
fsck: detect problems with _format file, and remove stream if empty
...
A bulkdata dir may get created for a new stream with an empty or
corrupted _format, before any data gets actually written. In that
case, we can just delete the new stream; worst case, we lose some
metadata.
Note: The info in _format should really get moved into the database.
This was born when bulkdata switched from PyTables to a custom storage
system, and was probably stored this way to avoid tying the main DB
to specific implementation details while they were still in flux.
2020-08-17 22:55:44 -04:00
905e325ded
fsck: add fixer that fully removes a stream
2020-08-17 22:55:32 -04:00
648b6f4b70
fsck: improve instructions about removing leftover data
2020-08-17 22:55:09 -04:00
7f8a2c7027
fsck: remove unnecessary raises
...
The called functions already raise RetryFsck.
2020-08-17 22:54:36 -04:00
276fbc652a
Bump Python version requirement
2020-08-07 15:56:53 -04:00
10b34f5937
Fix issue with test suite and empty dirs on git
...
Git doesn't save empty dirs, so put a placeholder there that is
ignored when the test data is copied to its final location during
testing.
2020-08-07 10:49:29 -04:00
83daeb148a
Add fsck scan for any data timestamps outside interval range
2020-08-07 10:25:22 -04:00
d65f00e8b2
Add fsck to default tests
2020-08-07 02:56:49 -04:00
71dc01c9a7
Replace deprecated numpy.fromstring usage
2020-08-07 02:55:20 -04:00
bcd21b3498
Improve fsck test coverage to 100%
2020-08-07 02:54:45 -04:00
a1dee0e6f2
Improve fsck test coverage to 85%
2020-08-07 01:26:30 -04:00
99ac47cf0d
Start implementing fsck test and porting fsck to Python 3
2020-08-07 00:11:45 -04:00
4cdaef51c1
Fix flake8-reported issues with fsck
2020-08-06 23:10:51 -04:00
88466dcafe
Add yappi dependency for "nilmdb-server -y", but don't require ipython
2020-08-06 22:55:27 -04:00
8dfb8da15c
Freeze requirements to specific versions
2020-08-06 18:11:33 -04:00
6cc1f6b7b2
Fix #! at top of shell scripts for py3 and venvs
2020-08-05 17:02:30 -04:00
8dc36c2d37
Fix stream_insert_context docstring
...
This was never updated when timestamps were changed from floats to
ints.
2020-08-05 14:22:35 -04:00
3738430103
Fix flake8 warnings
2020-08-03 23:40:53 -04:00
a41111b045
Fix some Python 3.8 related issues
2020-08-03 17:48:51 -04:00
85f822e1c4
Decode non-JSON HTTP responses when possible
...
This doesn't affect anything in nilmdb, but is needed by nilmrun.
2020-08-03 17:31:11 -04:00
0222dfebf0
Update git URL
2020-08-03 16:48:54 -04:00
70914690c1
Update README for Python 3.8 and newer
2020-08-03 16:36:10 -04:00
10400f2b07
rocket: suppress build warnings
2020-08-03 16:27:55 -04:00
56153ff7ad
Update installation instructions
2019-08-30 17:14:50 -04:00
671f87b047
Clean up README
2019-08-30 16:29:39 -04:00
2f2faeeab7
Add pylint config and selectively fix some pylint errors
...
Some pylint errors are worth fixing; many are not.
2019-08-30 16:29:39 -04:00
2ed544bd30
Fix flake8-reported code style issues
2019-08-30 16:29:39 -04:00
6821b2a97b
Merge branch 'py3'
2019-08-30 16:27:11 -04:00
b20bb92988
Improve test coverage and remove the last "#pragma: no cover"
...
We now have full coverage of the main code.
2019-08-30 13:52:13 -04:00
699de7b11f
Change fast shutdown timeout to 0
...
This used to have problems with older CherryPy versions, but the
current one seems to handle it just fine.
2019-08-30 12:01:09 -04:00
ea67e45be9
Clean up how we handle cherrypy's calls of os._exit(70)
...
With this solution, we can catch it cleanly in the standalone
nilmdb-server, and test the error paths in our normal test suite.
2019-08-30 11:59:52 -04:00
ca440a42bd
Fix nilmdb-server argument parsing, logging, and profiling
...
It works again now.
2019-08-30 11:24:17 -04:00
4ff4b263b4
Fill out code coverage for nilmdb/server/bulkdata.py
2019-08-30 11:14:06 -04:00
79e544c733
Fix python2->3 conversion of pickle
2019-08-30 11:13:55 -04:00
9acf99ff25
Fill out coverage for server/nilmdb.py and remove dead code
2019-08-30 10:32:52 -04:00
4958a5ab2e
Improve test coverage
2019-08-30 01:28:16 -04:00
f2d89e2da5
Remove very outdated pytables tests
2019-08-30 01:28:03 -04:00
1952f245c0
Try to clean up some issues with cherrypy startup and os._exit
...
This is hard and finicky to test, so there's unfortunately not a clear
way to get 100% test coverage in cherrypy_start. However, that
function is only used in the test suite and the standalone
nilmdb-server script, not in production (which goes through wsgi and
skips all this cherrypy server stuff entirely).
2019-08-30 01:21:37 -04:00
7cbc0c11c3
Run cherrypy always in "embedded" mode
...
Non-embedded mode is not used in the test suite or wsgi server;
it was an option in the standalong nilmdb-server script, but it's
really not necessary, and removing it gets rid of some untested
code.
2019-08-30 00:24:16 -04:00
9f2651c35e
Stop ignoring errors when disabling SIGPIPE
2019-08-30 00:16:26 -04:00
