Skip to content

Cleaning up a full Bosh director

Jump to bottom
Chris Selzo edited this page Jul 3, 2019 · 1 revision

Sometime, you may notice that the pipeline jobs are failing like:

Uploading release file:
  Director responded with non-successful status code '500' response ''

Exit code 1

or

Error: Failed to upload blob, code 1, output: 'Error running app - Putting dav blob 71998771-9645-4fac-94d7-1826f8b7d94b: Wrong response code: 500; body: <html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>
', error: ''

This is most likely because your Bosh director's disk is full. To verify, target the environment and run:

~/workspace/capi-ci-private/mulan
± cs |master ✓| → bbl ssh --director --cmd "df -h"
...
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   52M  1.8G   3% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1       2.8G  1.5G  1.2G  58% /
/dev/sda3        33G  2.9G   29G  10% /var/vcap/data
tmpfs           1.0M   56K  968K   6% /var/vcap/data/sys/run
/dev/sdb1        63G   60G     0 100% /var/vcap/store        <-- yo dawg your disk is full
Connection to 10.0.0.6 closed.

The easiest way to get this under 100% (most likely to 99%) is to ssh to the director and delete the task logs:

± cs |master ✓| → bbl ssh --director
bosh/0:~$ sudo su -
bosh/0:~# cd /var/vcap/store/director/tasks
bosh/0:/var/vcap/store/director/tasks# rm -rf ./*

Then, log off of the director and run bosh clean-up --all to clear up more space. After doing this on Mulan, the result was:

bbl ssh --director --cmd "df -h"
...
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  152M  1.7G   9% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1       2.8G  1.5G  1.2G  58% /
/dev/sda3        33G  3.0G   29G  10% /var/vcap/data
tmpfs           1.0M   56K  968K   6% /var/vcap/data/sys/run
/dev/sdb1        63G   27G   34G  44% /var/vcap/store        <-- yo dawg your disk is no longer full
Connection to 10.0.0.6 closed.
Clone this wiki locally