Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jenkins builds failing due to lack of free space #428

Closed
jleveque opened this issue Mar 23, 2017 · 11 comments
Closed

Jenkins builds failing due to lack of free space #428

jleveque opened this issue Mar 23, 2017 · 11 comments
Assignees
@lguohan
Labels
Bug 🐛 Critical ❗

Comments

@jleveque
Copy link
Contributor

jleveque commented Mar 23, 2017 •
edited
Loading

https://sonic-jenkins.westus.cloudapp.azure.com/job/common/job/buildimage-baseimage-pr/413/:

make[4]: Entering directory '/sonic/src/sonic-linux-kernel/linux-3.16.36'
dh_installdocs 
dh_installchangelogs
dh_strip
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol -- -Vkernel:Recommends=irqbalance,
dh_md5sums
dh_builddeb -- -Zxz 
dpkg-deb: building package `linux-image-3.16.0-4-amd64' in `../linux-image-3.16.0-4-amd64_3.16.36-1+deb8u2_amd64.deb'.
make[4]: Leaving directory '/sonic/src/sonic-linux-kernel/linux-3.16.36'
dpkg-deb (subprocess): compressing data member: lzma write error: No space left on device
dpkg-deb: error: subprocess <compress> from tar -cf returned error exit status 2
dh_builddeb: dpkg-deb -Zxz --build debian/linux-image-3.16.0-4-amd64-dbg .. returned exit code 2
debian/rules.real:173: recipe for target 'install-base' failed

https://sonic-jenkins.westus.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all-pr/83/:

running build
running build_ext
error: [Errno 28] No space left on device: 'build/temp.linux-x86_64-3.6-pydebug'
Makefile:627: recipe for target 'sharedmods' failed
make[3]: *** [sharedmods] Error 1

https://sonic-jenkins.westus.cloudapp.azure.com/job/cavium/job/buildimage-cavm-all-pr/410/:

/tmp/ccJe73Wu.s: Assembler messages:
/tmp/ccJe73Wu.s: Fatal error: can't write tests-converter_ut.o: No space left on device
@jleveque jleveque mentioned this issue Mar 23, 2017
Merged
@stcheng
Copy link
Contributor

stcheng commented Mar 28, 2017

https://sonic-jenkins.westus.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all/130/console

dpkg-deb: building package `swss' in `../swss_1.0.0_amd64.deb'.
dpkg-deb (subprocess): compressing control member: internal gzip write error: No space left on device
dpkg-deb: error: subprocess gzip -9c returned error exit status 2
dh_builddeb: dpkg-deb --build debian/swss .. returned exit code 2
dpkg-deb: building package `swss-dbg' in `../swss-dbg_1.0.0_amd64.deb'.
dpkg-deb (subprocess): compressing control member: internal gzip write error: No space left on device
dpkg-deb: error: subprocess gzip -9c returned error exit status 2
dh_builddeb: dpkg-deb --build debian/swss-dbg .. returned exit code 2
debian/rules:22: recipe for target 'binary' failed

@lguohan
Copy link
Collaborator

lguohan commented Apr 6, 2017

allocated new 1T disk for jenkins build.

@lguohan lguohan closed this as completed Apr 6, 2017
@stcheng stcheng reopened this Apr 14, 2017
@stcheng
Copy link
Contributor

stcheng commented Apr 14, 2017

fatal error: error writing to /tmp/ccCK5Whn.s: No space left on device

@lguohan
Copy link
Collaborator

lguohan commented Apr 14, 2017

sonic-slave consume the disk, manually cleaned them up. Should be good for a while. Will figure out way to automatically clean up the obseleted sonic-slave

@lguohan lguohan self-assigned this Apr 14, 2017
@stcheng
Copy link
Contributor

stcheng commented May 1, 2017 •
edited
Loading

#560
https://sonic-jenkins.westus.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all-pr/765/console

@jleveque
Copy link
Contributor Author

jleveque commented May 1, 2017

Also: https://sonic-jenkins.westus.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all-pr/764/

@stcheng
Copy link
Contributor

stcheng commented May 2, 2017

Also: https://sonic-jenkins.westus.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all-pr/771/consoleFull

@stcheng stcheng closed this as completed Jun 7, 2017
@jleveque
Copy link
Contributor Author

Looks like it's happening once again: https://sonic-jenkins.westus.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all-pr/752/console

@jleveque jleveque reopened this Jul 24, 2017
@stcheng
Copy link
Contributor

stcheng commented Jul 26, 2017

🚑

@lguohan
Copy link
Collaborator

lguohan commented Jul 26, 2017

on worker-1 we have 221G flash for the docker

/dev/sdb1 221G 126G 84G 61% /mnt

it looks like when there are 10 concurrent jobs running at the same time, the /mnt partition will become full.

I clean up the /mnt partition about 40G garbage data. If this disk partition happens again, I will increase the /mnt partition to 512G.

@jleveque
Copy link
Contributor Author

Closing as we haven't experienced this for a while now.

yxieca added a commit to yxieca/sonic-buildimage that referenced this issue Mar 7, 2019
@yxieca
[201811 sub module] advance sairedis sub module
f166f49 
Submodule src/sonic-sairedis 54b02a1..8182916:
  > Add pre match to comparison logic and unittests (sonic-net#423)
  > Drop FDB notifications if they contain invalid OIDs (sonic-net#428)

Signed-off-by: Ying Xie <[email protected]>
@yxieca yxieca mentioned this issue Mar 7, 2019
Merged
yxieca added a commit that referenced this issue Mar 7, 2019
@yxieca
[201811 sub module] advance sairedis sub module (#2657)
23aa759 
Submodule src/sonic-sairedis 54b02a1..8182916:
  > Add pre match to comparison logic and unittests (#423)
  > Drop FDB notifications if they contain invalid OIDs (#428)

Signed-off-by: Ying Xie <[email protected]>
@wendani wendani mentioned this issue Apr 9, 2019
Merged
tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this issue Apr 10, 2019
@yxieca @tiantianlv
[201811 sub module] advance sairedis sub module (sonic-net#2657)
3a220d8 
Submodule src/sonic-sairedis 54b02a1..8182916:
  > Add pre match to comparison logic and unittests (sonic-net#423)
  > Drop FDB notifications if they contain invalid OIDs (sonic-net#428)

Signed-off-by: Ying Xie <[email protected]>
tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this issue Apr 10, 2019
@yxieca @tiantianlv
[201811 sub module] advance sairedis sub module (sonic-net#2657)
7402bee 
Submodule src/sonic-sairedis 54b02a1..8182916:
  > Add pre match to comparison logic and unittests (sonic-net#423)
  > Drop FDB notifications if they contain invalid OIDs (sonic-net#428)

Signed-off-by: Ying Xie <[email protected]>
tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this issue Apr 10, 2019
@yxieca @tiantianlv
[201811 sub module] advance sairedis sub module (sonic-net#2657)
3c57b3c 
Submodule src/sonic-sairedis 54b02a1..8182916:
  > Add pre match to comparison logic and unittests (sonic-net#423)
  > Drop FDB notifications if they contain invalid OIDs (sonic-net#428)

Signed-off-by: Ying Xie <[email protected]>
lguohan pushed a commit that referenced this issue Apr 12, 2019
@wendani @lguohan
[submodule]: submodule update for PFC watchdog warm reboot commits (#…
9b6a2fb 
…2755)

sonic-sairedis:
e8cb879 Make object list deterministic when iterating (#438)
5486f97 Ignore ACL_COUNTER bytes and packets during comparison logic (#443)
b138ff9 Notify OA about exception and process only restart query events (#437)
0974a43 Set MTU value on created tap device for virtual switch (#436)
e2f50e8 Increase eth buffer size to 16k for virtual switch (#435)
79fb388 [bfn] Ged rid of ld_preload. Link against libsai only (#429)
2e47b78 Remove MAC alignment WA for Mellanox platforms. (#430)
e1354fe Add pre match to comparison logic and unittests (#423)
18a5ebb Drop FDB notifications if they contain invalid OIDs (#428)

sonic-swss-common:
8af58ad sonic-swss-common: Add vxlan macros to schema (#269)
76837bf Make class Select support batch read from selectables (#270)
4cf643e Add multiple fields hdel support (#267)
a710529 Update PFC_WD table name in CONFIG_DB (#266)
3c452c1 Update README.md (#268)

sonic-swss:
e329dbd Survive pfc watchdog storm action across warm-reboot (#794)

sonic-utilities:
6ee0aea (HEAD, origin/master, origin/HEAD) [config]: Change the order of interface commands (#504)
5ae30d2 [show vlan brief] Support 'alias' interface naming mode (#497)
bafebf9 Update neighbor advertiser (#498)
fa90083 [clear/main.py]: clear ndp command. (#450)
65f69e4 [show interface neighbor expected] Support 'alias' interface naming mode (#495)
aae39e7 updated show ipv6 interface for alias mode (#493)
170fed9 [warm-reboot] initialize warm reboot state table before warm rebooting (#492)
06cd99f Allow config shutdown and startup operations on valid PortChannel interface names (#474)
98cdebb [show ip interface] Add support for 'alias' interface naming mode (#486)
5f1de81 [show] Add serial numbers/uptime/hwinfo to 'show version' output (#488)
e78a866 [route_check] Move scripts under scripts/ directory, add to setup.py (#489)
d347527 Change PFC watchdog CONFIG_DB table name from PFC_WD_TABLE to PFC_WD (#475)

Signed-off-by: Wenda Ni <[email protected]>
madhanmellanox pushed a commit to madhanmellanox/sonic-buildimage that referenced this issue Mar 23, 2020
@sihuihan88
[pfcwd]: rename FLEX_COUNTER_POLL_MSECS to PFC_WD_POLL_MSECS and move…
88636f9 
… define in swss common (sonic-net#428)

Signed-off-by: Sihui Han <[email protected]>
dmytroxshevchuk pushed a commit to dmytroxshevchuk/sonic-buildimage that referenced this issue Aug 31, 2020
@kcudnik
Drop FDB notifications if they contain invalid OIDs (sonic-net#428)
18a5ebb
@tonytitus tonytitus mentioned this issue Dec 29, 2020
Merged
lguohan pushed a commit that referenced this issue Jan 1, 2021
[sonic-swss-common] Update submodule (#6317)
70ba591 
Including commits in sonic-swss-common repo:

b423b9c Add support for hexists call (#432) [Tony Titus]
0982996 Remove extension of tableNameSeparatorMap (#430) [Qi Luo]
d16cc76 [build]: add azure pipeline build badge (#429) [lguohan]
f2aaf55 Set up CI with Azure Pipelines (#428) [lguohan]
@guillaumelambert guillaumelambert mentioned this issue Dec 15, 2022
Open
mssonicbld added a commit that referenced this issue Aug 29, 2024
@mssonicbld
[submodule] Update submodule sonic-linux-kernel to the latest HEAD au…
4061373 
…tomatically (#20068)

#### Why I did it
src/sonic-linux-kernel
```
* 2cfa620 - (HEAD -> 202311, origin/202311) tg3: fix broadcom NIC 57766 staying down issue (#421) (#428) (7 hours ago) [Vasundhara Volam]
```
#### How I did it
#### How to verify it
#### Description for the changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lguohan lguohan

Labels
Bug 🐛 Critical ❗
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lguohan @stcheng @jleveque