Source caches loses blobs along the way?

[Cynical-Optimist]

See original issue on GitLab
In GitLab by [Gitlab user @BenjaminSchubert] on Aug 30, 2020, 10:13

Summary

I was trying to ingest my sources before running my builds.
I have plenty enough of disk space (200GB free, while trying FSDK), and my cache has no quota.

For this I did:

bst source fetch my_target.bst
bst source push --deps all my_target.bst

I expected this to succeed and be able to build without any external connection.

I however got a few sources that failed to be pushed with errors as seen below. (Size does not seem to affect the success rate).

I tried a second bst source push --deps all my_target.bst, and it started downloading sources again? And failing again to upload them.

What is the current bug behavior?

I get:

[--:--:--] START   cross-compilers/freedesktop-sdk-arm.bst:bootstrap/build/dwz-build.bst: Src-push
[--:--:--] STATUS  cross-compilers/freedesktop-sdk-arm.bst:bootstrap/build/dwz-build.bst: Pushing data for source 46b59054 -> http://bst-source-cache:8000
[--:--:--] BUG     cross-compilers/freedesktop-sdk-arm.bst:bootstrap/build/dwz-build.bst: Src-push    An unhandled exception occured:    Traceback (most recent call last):
      File "/home/buildstream/workspace/buildstream/src/buildstream/_scheduler/jobs/job.py", line 650, in child_action
        result = self.child_process()  # pylint: disable=assignment-from-no-return
      File "/home/buildstream/workspace/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 93, in child_process
        return self._action_cb(self._element)
      File "/home/buildstream/workspace/buildstream/src/buildstream/_scheduler/queues/sourcepushqueue.py", line 44, in _push_or_skip
        if not element._source_push():
      File "/home/buildstream/workspace/buildstream/src/buildstream/element.py", line 1753, in _source_push
        return self.__sources.push()
      File "/home/buildstream/workspace/buildstream/src/buildstream/_elementsources.py", line 140, in push
        if self._sourcecache.push(source):
      File "/home/buildstream/workspace/buildstream/src/buildstream/_sourcecache.py", line 204, in push
        self.cas._send_directory(remote, source_proto.files)
      File "/home/buildstream/workspace/buildstream/src/buildstream/_cas/cascache.py", line 672, in _send_directory
        self.send_blobs(remote, required_blobs)
      File "/home/buildstream/workspace/buildstream/src/buildstream/_cas/cascache.py", line 661, in send_blobs
        for digest in digests:
      File "/home/buildstream/workspace/buildstream/src/buildstream/_cas/cascache.py", line 456, in required_blobs_for_directory
        with open(self.objpath(directory_digest), "rb") as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/home/buildstream/.cache/buildstream/cas/objects/53/307cae796d1262c6834ef9814d41563b10d30731423e17059cd68bf5d37cc9'

What is the expected correct behavior?

I should have all my sources synced and pushed

---

[Cynical-Optimist]

In GitLab by [Gitlab user @BenjaminSchubert] on Aug 30, 2020, 10:14

[Gitlab user @juergbi] any guess on that?

[Cynical-Optimist]

In GitLab by [Gitlab user @juergbi] on Aug 31, 2020, 10:30

I haven't been able to reproduce the issue so far in master and also haven't found a relevant logic error reviewing code. The stack trace doesn't make sense to me, though. send_blobs() shouldn't call required_blobs_for_directory(). Is this with master? Any idea about the odd stack trace? Are you running recent buildbox-casd?

I've attempted to reproduce the issue with freedesktop-sdk and the targets from its Makefile. What exact version and targets of freedesktop-sdk did you use?

I do see 46b59054 for bootstrap/build/dwz-build.bst as well, however, push succeeded even though the blob from the stack trace doesn't exist in my cache either.