[agl-discussions] To mirror sources, or not....

Jan-Simon Moeller jsmoeller at linuxfoundation.org
Fri Dec 4 18:48:26 UTC 2015


Hi Gunnar, all !

Am Donnerstag, 3. Dezember 2015, 11:26:43 schrieb Andersson, Gunnar:
> Hello Jan-Simon
> 
> I still owe you a USB cable ;-)
> 
Lol, we can convert that into a beer then ;) .


Wrt our discussion, we have to differ 2, actually 3 sides here:
1) build process (done through the MIRROR_TARBALLS)
2) sources for the released images as per license requirements 
(Archiver)
3) sources during code development (mostly git)


> > echo 'BB_GENERATE_MIRROR_TARBALLS = "1"' >> conf/local.conf
> 
> I can confirm, and to be clear that packs up any cloned git repos into
> tarballs inside of the downloads directory and deletes the git repos. 
> The tarballs are already tarballs...  so you get a consistent
> directory that includes only tarballs.  Well, sort of consistent 
> - the naming of these git balls is still ugly, but functional.
> And last I checked it did _not_ tar up the subversion checkout(?).

OK, so in all meta-folders I've here right now, we have just:
grep -rn 'SRC_URI.*=.*"svn:'  meta-* | wc -l
14
svn SRC_URI. 
git has 128, 
http has 316, 
https 28, 
ftp 27 .

Btw, svn checkouts are always (even w/o MIRROR_TARBALLS) 
tarred-up in downloads/ with the name:
<PN>_<SRCHOST>_.<BRANCH>_<REVISION>.tar.gz
e.g.
Tremor_svn.xiph.org_.trunk_18221_.tar.gz


> There are still lots of weird inconsistencies like that in 
> Yocto/bitbake.
Yes, there are some left and should be fixed upstream.


> > bitbake -k -c fetchall  <target|world>
> > 
> > (possibly repeat for different MACHINE=)
> > 
> > Now you have a download directory that you can rsync to a webserver.
> 
> Nitpick but arguably you have that also without tarring the git repos.
> There is no difference in content, just convenience agreed? 
Without the mirror set to one, you have "--bare" clones which are 
currently cannot be mirrored and fetched "as is". Thats why the 
workaround with the tarballs of git trees.


> I'm just
> pointing it out to state that this is not really a new mirroring
> function, but just something that adjusts the content of downloads/ .
Basically it makes ./downloads mirrorable - nothing more.

> The development communities have a choice to make, IMHO - either all
> tar-balls (or packages / SRPMs and the like) or all-git.  This
> undecided mix is a bit unsatisfying.
Well, git is the winner on the development side as it looks like, but it 
did not tickle down into the packaging - releases are still tarballs in 
the end. See my recipe stats above.



> Also my understanding is that the recommendation in the Yocto manual
> [1], is a little different from simply BB_GENERATE_MIRROR_TARBALLS.
>  The archiver class suggested there, is it a bit more refined?  It
> appears to be focused on packaging the precise source code that was
> really used in the build?  I could be wrong, and obviously for all the
> sources that were built using versioned tarballs anyway, then the
> content must be pretty much identical.
Lets talk shop by example:
- the mirror_tarballs produces this:
https://download.automotivelinux.org/AGL/mirror/
  which can be used by the bitbake build process to fetch all its 
sources. It is a simple and flat directory.

- the archiver class produces this:
https://download.automotivelinux.org/AGL/snapshots/master/2015-12-03-b61/qemux86-64/deploy/sources/
  which is "per package" subfolder.

While the content is pretty identical (and we use fdupes on the download 
host to reduces disk usage), the directory structure is not. 

I'm even not sure if bitbake can reuse the output of the archive for 
build purposes.


> For example, what about applied patches? Are they included in the
> 'archiver' approach (no time to dive into the source now).   If not 
you
>  should likely mirror at least the meta layers since they could 
include
> patches.  Patches are at least not applied on the downloads folder, I
> guess we can say that much.
So for:
- the mirror does not have the patches as they are part of the layers 
alongside the recipes. So there are no patch files in download/ .
So no *.patch in downloads and on the mirror. 
https://download.automotivelinux.org/AGL/mirror/

- the archiver they're in the respective package folder alongside the 
unpatched tarball. bitbake is so nice to also add a quilt series file to 
that folder.
e.g.:
https://download.automotivelinux.org/AGL/snapshots/master/2015-12-03-b61/qemux86-64/deploy/sources/x86_64-poky-linux/acl-2.2.52-r0/


> > To use it, you can set your PREMIRROR or
> >  SOURCE_MIRROR_URL ?= "http://source_mirror/sources/"
> >  INHERIT += "own-mirrors"
> 
> I have used the BB_GENERATE_MIRROR_TARBALLS and 'own-mirrors' approach
> to version control all the sources required for a build.  As it is
> intended, I wanted to distribute this source archive and liked having
> exactly one file per package.
> 
> However, bitbake does not always behave properly (IMHO) when reusing
> the source from a BB_GENERATE_MIRROR_TARBALLS archive on subsequent
> builds.  In some cases bitbake went out to check an online git remote
> anyhow.  This can be seen by adding BB_NO_NETWORK so that bitbake 
stops
> if it is about to attempt internet access.
There are 2 sides here:
- recipes using a git branch likely check for the "latest" branch state
- BB_NO_NETWORK will prohibit that and continue if things are in 
  downloads/ already. If not this is a bug in bitbake.

> I suppose this git check was a code block intended to see if there is
> anything new on the branch, but this happens despite that the hash is
> locked down and that commit already exists in the tarred up git!  
- some recipes have a bug and iirc use the wrong magic SRCwhatever in 
the package version statement - same result. I don't recall the full 
details right now from the top of my head. 
 
> Triggering this behavior depends on how your SRC URI and version is
> written.  It's a bug IMHO, but after some initial discussions on IRC I
> didn't get that far, and I've not been able to condense it into a
> useful bug report.  Some might say user error but I don't believe so. 
Exactly. IIRC the "standard" way to specify the version changed over 
time and not all recipes got rewritten (most are meanwhile).



Best,
JS


More information about the automotive-discussions mailing list