Skip to content

beesh: Abort if download fails #58

Merged
merged 1 commit into from
Oct 19, 2023
Merged

beesh: Abort if download fails #58

merged 1 commit into from
Oct 19, 2023

Conversation

donald
Copy link
Contributor

@donald donald commented Sep 11, 2023

Currently, bee doesn't detect the situation when a partial file is downloaded. An incomplete archive usually makes the following extract to fail, but the broken file is kept in the cache an needs to be removed manually. The situation, when a failed download produces an empty file, is already handled. Empty cache files are removed when bee is run the next time.

Currently, bee contains explicit code to ignore errors from wget. This code existed in the initial commit 067ccf2 ("initial revision") from
2010. The reasoning is unclear today and has to be rediscovered if it's still valid. So this patch is kind of experimental and might need to be undone.

Remove code to explicitly ignore wget failures. As the script is running with -e, a wget failure should invoke the EXIT trap, which removes the incomplete file. This is untested.

Fixes #57

@donald
Copy link
Contributor Author

donald commented Sep 11, 2023

May be wrong.

buczek@theinternet:~/git/bee-files (add-linux-5.15.131)$ unshare -r ./nvidia_linux-5.15.131-453-510.60.02-0.bee
BEE v1.2.26 2009-2016
  by  Marius Tolzmann <marius@mariux.de>
      Matthias Ruester <ruester@molgen.mpg.de>
      Tobias Dreyer <dreyer@molgen.mpg.de>

[BEE]   BEE_SKIPLIST           /etc/bee/skiplist
[BEE]   BEE_REPOSITORY_PREFIX  /src/mariux/beeroot
[BEE]   BEE_METADIR            /usr/share/bee
[BEE]   BEE_TMP_TMPDIR         /scratch/local
[BEE]   BEE_TMP_BUILDROOT      /scratch/local/bee-buczek
[BEE] current working directory: /home/buczek/git/bee-files
[BEE] ==> initializing build environment ..
[BEE]  -> creating source dir /scratch/local/bee-buczek/nvidia/nvidia_linux-5.15.131-453-510.60.02-0/source
[BEE]  -> creating image dir /scratch/local/bee-buczek/nvidia/nvidia_linux-5.15.131-453-510.60.02-0/image
[BEE] ==> checking repositories ..
[BEE] ==> building nvidia_linux-5.15.131-453-510.60.02-0.x86_64 ..
[BEE] => entering bee_getsources() ..
[BEE] fetching https://beehive.molgen.mpg.de/2288e18f3b767e66d465fb4add1df080/NVIDIA-Linux-x86_64-510.60.02.run
File ‘/scratch/local/bee-buczek/nvidia/files/NVIDIA-Linux-x86_64-510.60.02.run’ already there; not retrieving.
buczek@theinternet:~/git/bee-files (add-linux-5.15.131)$ r^C

Currently, bee doesn't detect the situation when a partial file is
downloaded. An incomplete archive usually makes the following extract to
fail, but the broken file is kept in the cache an needs to be removed
manually. Only the the situation, when a failed download produced an
empty file is explicitly handled.

Currently, bee contains explicit code to ignore errors from wget.  This
code existed in the initial commit 067ccf2 ("initial revision") from
2010. The probably reason for it is, that the option `--no-globber` is
used to avoid downloading a file, which already exists, a second time.
However, wget terminates with a non-zero exit status in that case.

Handle the case of an existing file in the calling code, so that we can
treat any wget error as a real error and abort (by the `-e` setting of
the shell).

While we are at it, change code to download into a tempfile and rename
it in the success case to avoid problems with partial downloads. This is
more robust than the EXIT trap used by current code.
Sign in to join this conversation on GitHub.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bee download should remove relicts on download failure
1 participant