[Fuego] Fuego docker container size keeps growing and hits maximum storage capacity

Tim.Bird at sony.com Tim.Bird at sony.com
Mon Nov 19 22:24:02 UTC 2018


Thanks - this looks good.
 -- Tim


> -----Original Message-----
> From: Dhinakar Kalyanasundaram [mailto:dhinakar.k at samsung.com]
> Sent: Sunday, November 18, 2018 9:07 PM
> To: Bird, Timothy <Tim.Bird at sony.com>; fuego at lists.linuxfoundation.org
> Cc: Dhinakar Kalyanasundaram <dhinakar.k at samsung.com>
> Subject: RE: RE: RE: RE: [Fuego] Fuego docker container size keeps growing
> and hits maximum storage capacity
> 
> Hi Tim,
> 
> 
> 
> I have added it to 'docker tips' (contents pasted below).
> 
> Please feel free to edit it if you find anything inappropriate.
> 
> 
> 
> = Copy/Replace a file into a non-running container =
> Background:
> Consider a case where you make some changes to '/etc/default/jenkins' file
> when your container is running and restarted the container. Unfortunately
> your container did not start because of an issue in the '/etc/default/jenkins'
> file. How do you fix it as the container itself is not running?
> 
> Solution:
> Get the container id (of the non-running container) via
> {{{
> $ 'docker ps -a' command
> }}}
> 
> Replace the faulty file with original/corrected one via 'docker cp' command as
> shown in the example below.
> {{{
> $ sudo docker cp jenkins 6b4e6e63rfg7:/etc/default/
> }}}
> where '6b4e6e63rfg7' is the container id of the non-running container
> 
> Now you will able to start the docker container successfully.
> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : Tim.Bird at sony.com <Tim.Bird at sony.com>
> 
> Date : 2018-11-17 05:15 (GMT+5:30)
> 
> Title : RE: RE: RE: [Fuego] Fuego docker container size keeps growing and hits
> maximum storage capacity
> 
> To : Dhinakar Kalyanasundaram<dhinakar.k at samsung.com>,
> fuego at lists.linuxfoundation.org
> 
> 
> 
> Wow. I didn't know you could copy a file to a non-running container! That's
> handy to know. I'm glad you were able to find a workaround. We should add
> that to our docket tips page: http://fuegotest.org/wiki/Docker_Tips
> 
> - - Tim
> 
> 
> -------- Original Message --------
> Subject: RE: RE: RE: [Fuego] Fuego docker container size keeps growing and
> hits maximum storage capacity
> From: Dhinakar Kalyanasundaram <dhinakar.k at samsung.com>
> Date: Nov 16, 2018, 3:43 AM
> To: "Bird, Timothy" <Tim.Bird at sony.com>,fuego at lists.linuxfoundation.org
> 
> Hi Tim,
> 
> 
> 
> Just an update to the issue that I mentioned at the end of my email.
> 
> I was able to find a way to fix the issue that I made because of which the
> container was not starting.
> 
> I got the container id via 'docker ps -a' command and replaced the
> 'etc/default/jenkins' file with the working one using 'docker cp' command as
> shown below.
> 
> Then I was able to start the docker container successfully.
> 
> 
> 
> $ sudo docker cp jenkins 6b4e6e63rfg7:/etc/default/
> 
> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : Dhinakar Kalyanasundaram <dhinakar.k at samsung.com> Staff
> Engineer/FDS SW /SSIR/Samsung Electronics
> 
> Date : 2018-11-14 18:57 (GMT+5:30)
> 
> Title : RE: RE: [Fuego] Fuego docker container size keeps growing and hits
> maximum storage capacity
> 
> To : null<Tim.Bird at sony.com>, null<fuego at lists.linuxfoundation.org>
> 
> CC : Dhinakar Kalyanasundaram<dhinakar.k at samsung.com>
> 
> 
> 
> Hi Tim,
> 
> 
> 
> I have disabled 'DNSMultiCast' temporarily via Manage Jenkins -> System Log
> -> Log Levels (on the left)
> and also deleted the /var/log/jenkins/jenkins.log file.
> 
> I tried a few things to permanently disable 'DNSMultiCast' inside the
> container (/etc/default/jenkins file)
> 
> Add the below JAVA_ARGS and restarted fuego but did not see it getting
> added under http://107.108.34.124:8080/fuego/systemInfo
> 
> JAVA_ARGS="-Djava.awt.headless=true -
> Dhudson.DNSMultiCast.disabled=true"
> 
> 
> 
> I also tried -> JAVA_ARGS="-Dhudson.DNSMultiCast.disabled=true" but did
> not work. Only Manage Jenkins -> System Log -> Log Levels (on the left)
> works.
> 
> Is there any way to permanently disable it within the container, so that each
> time I restart the container I don't have to do this -> Manage Jenkins ->
> System Log -> Log Levels (on the left)
> 
> 
> Also, I tried setting the timezone to 'Asia/Kolkata' in /etc/default/jenkins file
> (add below arguments) but it also never gets updated
> 
> JAVA_ARGS="-
> Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Kolkata"
> 
> Please let me know how to set timezone permanent as well.
> 
> 
> 
> Basically I want to keep everything updated in the container so that I can
> easily port it across machines.
> 
> 
> 
> During the whole process there was an interesting observation detailed
> below.
> 
> To edit timezone to 'Asia/Kolkata' I tried many things in /etc/default/jenkins
> file.
> 
> In one such case, I appended the below phrase to 'JENKINS_ARGS' in
> /etc/default/jenkins file, which seems erroneous,
> 
> JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=8080 --
> prefix=/storm --
> Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Kolkata"
> 
> saved the file, exited fuego, and tried to restart the container but it failed to
> start as shown below,
> 
> 
> 
> dhinakar at userl2:~/fuego$ <mailto:dhinakar at userl2:~/fuego$>  ./fuego-
> host-scripts/docker-start-container.sh fuego-docker-container-12thNov2018
> 
> Starting Fuego container (fuego-docker-container-12thNov2018)
> [FAIL] Starting Jenkins Continuous Integration Server: jenkins failed!
> Please create Fuego docker container via docker-create-container.sh script
> 
> 
> 
> So the question is, if we make such mistake to /etc/default/jenkins file and
> not able to start the container, is there a way to fix it from outside the
> container?
> 
> Because in my case I had to create a new container, so lost all changes done
> inside my previous container.
> 
> 
> Regards,
> 
> Dhinakar
> 
> 
> 
> 
> 
> --------- Original Message ---------
> 
> Sender : Tim.Bird at sony.com <Tim.Bird at sony.com>
> 
> Date : 2018-11-14 13:13 (GMT+5:30)
> 
> Title : RE: [Fuego] Fuego docker container size keeps growing and hits
> maximum storage capacity
> 
> To : Tim.Bird at sony.com, Dhinakar
> Kalyanasundaram<dhinakar.k at samsung.com>,
> fuego at lists.linuxfoundation.org
> 
> 
> 
> This seems like a band-aid rather than a fix, but here's some information
> about turning down the logging for Jenkins.  A proper fix would be to figure
> out what error in your networking setup is causing the DNS errors, and fix
> that.  But disabling logging might help.
> 
> Here's the logging level fix:
> https://stackoverflow.com/questions/31719756/how-to-stop-jenkins-log-
> from-becoming-huge
> 
> See the last answer in that thread.  It has a workaround to turn off
> DNSMultiCast,
> which might be the cause of the problem.
>  --  Tim
> 
> 
> 
> > > -----Original Message-----
> ...
> 
> >
> > Looks like it could be related to this bug in Jenkins, having to do with
> > multicast:
> > https://issues.jenkins-ci.org/browse/JENKINS-25369
> >
> > I would truncate the log file, stop the container, and see if there's some
> way
> > to
> > disable DNSMultiCast.  Something appears to have gone wrong with your
> > network
> > configuration.  It could be the docker virtual network, or maybe something
> > has
> > changed about your physical networks.
> >
> >  -- Tim
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego at lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 
> 
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhina
> kar.k&do=bWFpbElEPTIwMTgxMTE5MDUwNzAyZXBjbXM1cDgxNGUwZGUx
> ODRkZmUzZTY3ZDlmOTZhNWU5MjM1NTRiNiZyZWNpcGllbnRBZGRyZXNzPV
> RpbS5CaXJkQHNvbnkuY29t>


More information about the Fuego mailing list