3.6 MongoRocks PeriodicKillSecondaries:ValidateCollections test hook fails
Description
Environment
Activity

Igor Solodovnikov March 1, 2018 at 11:31 AM
Here is the description of what could happen before the fix for this issue:
In some specific conditions data records could disappear at arbitrary moment of time from the collections or indexes created after server restart. This could happen as the result of the following sequence of events:
1. User deletes one or more indexes or collections. These should be the ones using maximum existing prefixes values.
2. User shuts down the server before mongo-rocks compaction thread executes compactions of deleted ranges.
3. User restarts the server and creates some new collections. Due to the bug those new collections and their indexes may get the same prefix values which were deleted and not yet compacted. User inserts some data into the new collections.
4. After the server restart mongo-rocks compaction thread continues executing compactions of the deleted ranges and this process may eventually delete data from the collections sharing prefixes with deleted ranges.

Igor Solodovnikov February 15, 2018 at 9:05 PM
There is a bug in mongo-rocks engine initialization code which causes wrong initialization of _maxPrefix value. This may lead to reuse of dropped prefix and to accidental remove of data from the collection using reused prefix. The same issue exists in v3.2 and v3.4.

Tomislav Plavcic February 12, 2018 at 8:06 AM
I made a comparison for this suite between 3.4 and 3.6 and I don't see the issue with 3.4. Here is the summary for 50 test runs for both versions. If needed I can give access to test logs since they're to big to upload to Jira.
3.4:
3.6:

Tomislav Plavcic February 6, 2018 at 9:30 AM
Can you please work on this?
Details
Details
Assignee

Reporter

This fails in current v3.6 branch and also in Igor's fix-reindex branch, it doesn't fail for WiredTiger.
Test suite is: replica_sets_kill_secondaries_jscore_passthrough
Can be observed here:
fix-reindex branch: https://jenkins.percona.com/view/PSMDB-3.6/job/percona-server-for-mongodb-3.6-param/71/testReport/
v3.6 branch: https://jenkins.percona.com/view/PSMDB-3.6/job/percona-server-for-mongodb-3.6-param/72/testReport/
Looks like this: