Disappearing logs, suspected race condition
We have a problem where some logs seem to be disappearing. In our system, with the way our logs are created, logs with the same identifying header data are created and saved in the designated WatsStandardJsonFormat folder multiple times as subsequent tests are performed on the same UUT. What we are seeing is that for one particular process, the logs are sometimes missing the last tests.
What I am wondering then, is if there is a possible race condition, that if a log file is over-written after the converter as read the old file to memory, but before it has finished processing and deleting it, then we might end up deleting the new log all-together before it is even processed.
-
Hi, a UUT report with same key header data (SN, PN, Process/Operation and datetime) will overwritten at the server, see https://support.virinco.com/hc/en-us/articles/360018085340-What-happens-to-a-report-when-it-is-submitted-to-WATS.
Subsequent tests on the same UUT must be included in the same report, or differentiated with different processes (test operation)
-
That is indeed the mechanism I am relying on. To elaborate, for each test at a given station, we will start by loading any file already present locally for the same procedure/UUT, after which we append the results of the current test and re-post the report with the new results added. Thus each new report posted will be more complete than the last up until the last one which will contain all the steps.The reason we cannot post the report only once is that the reporting system does not know if the current test is the last one or not.
Now, what I am wondering is if the following sequence of events is possible (for a procedure of N tests):
1. We save the report containing all results for tests up until N-1 in the WatsStandardJsonFormat folder
2. The Wats json converter starts processing file N-1
3. We finish test N and save the new (more complete) report into WatsStandardJsonFormat, overwriting the previous one
4. The Wats json converter finished processing file N-1, uploads the result and proceeds to delete the local file. Since the local file present is actually report N, we have accidentally lost the results from the final test
This would be consistent with the fact that the losses we experience are sporadic and do not follow any other pattern we can find thus far, as well as explain why it is only the last step that is sometimes lost.If this race condition is possible, do we then have to wait until all reports with the same key data has finished processing before we save a new one? Or would it be enough to ensure that we don't use the same filename, trusting that the Wats json converter processes files in order of oldest to newest?
-
Q: If this race condition is possible, do we then have to wait until all reports with the same key data has finished processing before we save a new one?
A: Yes.Q: Or would it be enough to ensure that we don't use the same filename, trusting that the Wats json converter processes files in order of oldest to newest?
A: It will try to start with the oldest but there is no guaranty, so may still be a race condition issue. This is complex, we also have to deal with offline scenarios, retransmit etc.Is it not an option to build the json file in a different folder and move it to the converter file once completed? There "has" to be something that indicates test completed, like "no updates" for a certain amount of time or something?
-
It is possible of course. We just were not aware of a possible race condition at all until recently, but since it is possible we will mitigate it from our end. Thanks for the help! ^^
While deleting files under processing (which we are doing, effectively) is not safe practice, I would still suggest the converter have a built-in mechanism to ensure it only deletes the file it actually processed (checking file metadata), as this is somewhat unexpected behaviour from my perspective.
Please sign in to leave a comment.
Comments
4 comments