Missing HCS messages in previewnet mirror node

[mike-burrage-hedera]

There are messages missing both in the mirror-node database and forwarded to the latest SDK in previewnet.
I'm investigating to find out what component is dropping the messages.

From SDK, created topic 1039 and sent 8 messages.
SDK output

topic ID: 0.0.1039
2019-12-16T17:39:58.968346Z received seq#(2): message: hello, HCS! 2
2019-12-16T17:40:02.231541002Z received seq#(3): message: hello, HCS! 3
2019-12-16T17:40:05.340475Z received seq#(4): message: hello, HCS! 4
2019-12-16T17:40:12.834566Z received seq#(6): message: hello, HCS! 6
2019-12-16T17:40:15.958943001Z received seq#(7): message: hello, HCS! 7

Possibly terminated PubSub client before 8th message was received, but #5 is missing.

From consensus-service-2's mirror-node DB:

mirror_node=# select to_timestamp(consensus_timestamp / 1000000000.0), consensus_timestamp, sequence_number, running_hash from topic_message where topic_num=1039 order by consensus_timestamp asc;
         to_timestamp          | consensus_timestamp | sequence_number |                                            running_hash                                            
-------------------------------+---------------------+-----------------+----------------------------------------------------------------------------------------------------
 2019-12-16 17:39:55.820149+00 | 1576517995820149000 |               1 | \x93234971f12210976732e7c1565b180034a00515728638a1e1626a4995893777f022677a3bb41b6cd0aff68d8e42ec5c
 2019-12-16 17:39:58.968346+00 | 1576517998968346000 |               2 | \x8c2dbcb2a1acf4bc60bb73c6914c9032530ae0a3b2a0e544d38dead6d4818a131844a0468fc417729608999b0676738e
 2019-12-16 17:40:02.231541+00 | 1576518002231541002 |               3 | \xb3d1759bca8071eda2599ead835c477bffaa5ded4bfee20a73124573ec67668042048d6216d862879079d058647175e4
 2019-12-16 17:40:05.340475+00 | 1576518005340475000 |               4 | \x0669285b7b668186a59d8ecefe789cd537e880e3a4b7f230682c747b79902ae9013e0ece775cdc8cc1d7968b7f98ef11
 2019-12-16 17:40:12.834566+00 | 1576518012834566000 |               6 | \xa867bee52ab106481bf873c5afb4364ce3d4906069979eba316024309fcb8331ab9fd8ab4db9eb39b6ca6ea9a9d20d71
 2019-12-16 17:40:15.958943+00 | 1576518015958943001 |               7 | \x3191fb8b2c62d27e9ec01c24df1517813ab2724be17ceed95ac03676d61c567ab6d6ccd7cbb0ab9317d78f33a77f77eb
 2019-12-16 17:40:20.194305+00 | 1576518020194305000 |               8 | \xf6f7577837f5b1e5919fb1b79c062492b73ab6d602e46945911f31188e7dba198a00805b1f860a5d7638709bb3e1b3eb

#5 is missing

From the legacy mirror-node's DB it is there at timestamp 1576518009575609000 which means it was in a record stream. Possibly an issue with the stream mover on consensus-service-2. Not sure about why #1 was missed by the SDK's subscriber.

mirror=# select consensus_timestamp, topic_id from record_stream where topic_id = '0.0.1039' order by consensus_timestamp asc;
 consensus_timestamp | topic_id 
---------------------+----------
 1576517995055462000 | 0.0.1039
 1576517995820149000 | 0.0.1039
 1576517998968346000 | 0.0.1039
 1576518002231541002 | 0.0.1039
 1576518005340475000 | 0.0.1039
 1576518009575609000 | 0.0.1039
 1576518012834566000 | 0.0.1039
 1576518015958943001 | 0.0.1039
 1576518020194305000 | 0.0.1039

[mike-burrage-hedera]

Problem was stream-mover.sh was moving record stream files prematurely. Fixed.