[Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

Tim.Bird at sony.com Tim.Bird at sony.com
Wed Nov 14 07:33:03 UTC 2018



> -----Original Message-----
> From: Dhinakar Kalyanasundaram 
> 
> Hi Tim,
> 
> Thanks for your prompt response.
> 
> Please find below the information that you asked for.
> 
> >> Can you do: "git describe" in your fuego-core directory, and tell me what it
> says?
> 
> 
> dhinakar at userl2:~/fuego-core$ <mailto:dhinakar at userl2:~/fuego-core$>
> git describe
> Draft-126-gb011f54
> 
> 
> Now the size of the container is 78GB
> 
> 
> 
> root at userl2:/var/log <mailto:root at userl2:/var/log> # du -shx * | sort -h
> 0       btmp
> 0       netserver.debug_79
> 0       netserver.debug_84
> 0       netserver.debug_85
> 0       wtmp
> 4.0K    dmesg
> 4.0K    exim4
> 4.0K    faillog
> 4.0K    fontconfig.log
> 4.0K    unattended-upgrades
> 12K     fsck
> 32K     lastlog
> 36K     alternatives.log
> 184K    apt
> 352K    dpkg.log
> 74G     Jenkins
> 
> 
> Looks like jenkins.log under /var/log/Jenkins folder is the culprit.
> 
> 
> 
> root at userl2:/var/log/jenkins <mailto:root at userl2:/var/log/jenkins> # ls -l
> total 77020892
> -rw-r--r-- 1 jenkins jenkins 78869378191 Nov 14 05:20 jenkins.log
> 
> 
> The truncated contents of it are listed below:
> 
> Since the file is too big I could not send you the whole contents.
> 
> Looks like some network logging is going on. I hope you get some clue based
> on the below data.
> 
> ..........
> 
>         [AllRecords at 453309601 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1649968098 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 557489375 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords at 252773764 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords at 645997782 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords at 1754886489 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 836812725 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 441064872 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 2011237907 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1482899169 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1538609948 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1566802509 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 805289249 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1054464549 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1350619999 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 968174403 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 623597902 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1089741554 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1545756462 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1419179075 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 41846296 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1083544303 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 467773961 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 2086814507 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 388496345 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1107783535 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 1982136902 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 2008579650 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1902730357 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1010122622 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 1272678419 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 969064013 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 542887643 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1802083445 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 1654586474 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 869123515 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords at 175149711 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 1688596387 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 1243081204 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 1744877612 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
>         [AllRecords at 121341452 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords at 1423802761 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 442725606 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 2005906834 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name:
> 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 2077868432 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 355669110 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: MacShare.local.]
>         [AllRecords at 1283742169 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: MacShare.local.]
>         [AllRecords at 86432974 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 980048646 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 24.56.108.107.in-addr.arpa.]
>         [AllRecords at 478020832 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 3.f.d.4.0.0.e.f.f.f.0.0.1.e.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa.]
>         [AllRecords at 359799997 type: TYPE_ANY index 255, class: CLASS_IN index
> 1, name: 24.56.108.107.in-addr.arpa.]
>         [AllRecords at 1992083274 type: TYPE_ANY index 255, class: CLASS_IN
> index 1, name: 184.56.108.107.in-addr.arpa.]
> ............

Looks like it could be related to this bug in Jenkins, having to do with multicast:
https://issues.jenkins-ci.org/browse/JENKINS-25369

I would truncate the log file, stop the container, and see if there's some way to
disable DNSMultiCast.  Something appears to have gone wrong with your network
configuration.  It could be the docker virtual network, or maybe something has
changed about your physical networks.

 -- Tim



More information about the Fuego mailing list