[yocto] Perforce fetcher ignores module and label

Thu Aug 31 10:54:51 PDT 2017

Hi Katu,

On 08/28 17:43, Katu Txakur wrote:
> Thanks for looking at this. See my recipe below. I've left the bits related
> to systemd service but I don't think they matter for this. I'm using an old
> implementation of Perforce (2010) in case this matters. I've tried going
> back to the old perforce.py fetcher but I get a license error... do you
> know if it would be easy to revert to the old version in my bitbake folder
> until we make this work?

Sorry, I've been swamped this week and haven't been able to look into
this yet.  This coming weekend is a holiday weekend in the USA, too.
But I plan to look at this early next week, hope that's OK.

We should be able to create a patch series to revert my changes so you
can go back to the old perforce fetcher.  It might also be worth
investigating how to take the current perforce fetcher and enable some
of the use cases that you have (but we can do this after we successfully
revert).

I'll try to send a patch to you next week for the reverting.
Thanks,
Andrew

> DESCRIPTION = "Dummy recipe to fetch from Perforce"
> SECTION = "PerforceRecipes"
> LICENSE = "CLOSED"
> PR = "r0"
> 
> inherit cmake
> DEPENDS = "boost alsa-tools"
> 
> P4USER = "myuser"
> P4PASSWD = ""
> P4PORT = "192.168.1.55:1666"
> FETCHCMD_p4 = "/usr/local/bin/p4"
> 
> SRC_URI = " \
> p4://${P4USER}:${P4PASSWD}@myrepo/myfolder/...; \
> file://perforce-recipe.service \
> "
> SRCREV = "${AUTOREV}"
> S = "${WORKDIR}/p4"
> 
> do_install_append () {
>     install -d ${D}${systemd_unitdir}/system
>     install -m 0644 ../perforce-recipe.service ${D}${systemd_unitdir}/system
> }
> 
> NATIVE_SYSTEMD_SUPPORT = "1"
> SYSTEMD_PACKAGES = "perforce-recipe"
> SYSTEMD_SERVICE_${PN} = "perforce-recipe.service"
> 
> SYSTEMD_AUTO_ENABLE = "disable"
> 
> FILES_${PN} = "${systemd_unitdir} ${bindir}"
> 
> 
> 2017-08-28 15:27 GMT+01:00 Andrew Bradford <andrew at bradfordembedded.com>:
> 
> > Hi Katu,
> >
> > On 08/28 08:56, Katu Txakur wrote:
> > > 2017-08-11 13:52 GMT+01:00 Andrew Bradford <andrew at bradfordembedded.com
> > >:
> > > On 08/11 11:34, Katu Txakur wrote:
> > > > > 2017-08-07 12:17 GMT+01:00 Andrew Bradford <
> > andrew at bradfordembedded.com
> > > > >:
> > > > > > On 08/02 10:28, Paul Eggleton wrote:
> > > > > > > On Wednesday, 2 August 2017 9:40:10 AM CEST Katu Txakur wrote:
> > > > > > > > I'm still having problems fetching from Perforce. Is there any
> > > > > > > > documentation based on the latest implementation of the
> > > > > > > > poky/bitbake/lib/bb/fetch2/perforce.py file?
> > > > > >
> > > > > > There's some documentation now in the official Yocto Project
> > > > > > documentation location for bitbake fetchers [1].  Prior to my
> > changes
> > > > > > there was very little if any documentation, afaik.
> > > > > >
> > > > > > [1]: http://www.yoctoproject.org/docs/2.3.1/bitbake-user-
> > > > > > manual/bitbake-user-manual.html#perforce-fetcher
> > > > > >
> > > > > > > > 2017-07-25 10:05 GMT+01:00 Katu Txakur <katutxakurra at gmail.com
> > >:
> > > > > > > > > I'm upgrading a recipe that fetches the source code from
> > > > Perforce.
> > > > > > > > >
> > > > > > > > > The old recipe was:
> > > > > > > > >
> > > > > > > > > SRC_URI = " \
> > > > > > > > >   p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot/path/
> > pe
> > > > > > > > > rforce/...;module=local/path/relativeto/p4;label=${
> > P4CHANGELIST}
> > > > \
> > > > > > > > >   "
> > > > > > > > >
> > > > > > > > > With the new version of /lib/bb/fetch2/perforce.py, I had to
> > set
> > > > > > P4PORT
> > > > > > > > > outside SRC_URI, leaving the recipe with:
> > > > > > > > >
> > > > > > > > > SRC_URI = " \
> > > > > > > > >   p4://${P4USER}:${P4PASSWD}@Depot/path/perforce/...;module=
> > > > > > > > > local/path/relativeto/p4;label=${P4CHANGELIST} \
> > > > > > > > >   "
> > > > > > > > >
> > > > > > > > > The Perforce fetcher kind of works, but it seems to be
> > ignoring
> > > > the
> > > > > > > > > "module" and the "label" attributes. After reading the Python
> > > > script
> > > > > > I can
> > > > > > > > > see that after the "@", only the substring before the first
> > ";"
> > > > is
> > > > > > used.
> > > > > >
> > > > > > Sorry for not making the label change more clear, but you should be
> > > > able
> > > > > > to simply set SRCREV="labelname" and have the functionality you
> > desire.
> > > > > > However, we don't use labels very often internally, so the use of
> > > > labels
> > > > > > in SRCREV isn't tested that often by me.
> > > > > >
> > > > >
> > > > > I don't use labels that much either, however, I used to be able to
> > write
> > > > > the changeslist number as a label and it would
> > > > > fetch the code for that changelist. This doesn't work using SRCREV =
> > > > > "changelistnumber"
> > > >
> > > > This does work for me using oe-core morty branch and bitbake
> > > > 1.32 which is the versions where these changes were first applied and
> > > > where I did my testing.
> > > >
> > > > Like I have a recipe which I put:
> > > >
> > > > SRCREV = "184127"
> > > >
> > > > The p4 fetcher will go fetch changelist 184127 of the SRC_URI path in
> > > > perforce.
> > > >
> > > > Or if I put:
> > > >
> > > > SRCREV = "${AUTOREV}"
> > > >
> > > > Then it'll fetch the HEAD/tip of the SRC_URI path in perforce.
> > > >
> > > > Is this not working the same way with poky now?
> > > >
> > >
> > > I'm using yocto pyro with bitbake 1.34.0. When I first fetch the code for
> > > my recipe, it gets the latest changelist.
> > > I submit a change and the next time, the recipe thinks there nothing new
> > to
> > > fetch. Also, using SRCREV = "184127"
> > > doesn't work for me, it always gets the latest changelist.
> >
> > Can you post the recipe where this is failing for you?
> >
> > I could then try to recreate this issue in a sandbox depot we have in
> > our perforce server and see what's going on.
> >
> > > >
> > > > > > This change to using labels, changelist numbers, and allowing the
> > use
> > > > of
> > > > > > AUTOREV was made so that the perforce fetcher would act more like
> > the
> > > > > > other source code control system fetchers.  Prior to my changes to
> > the
> > > > > > p4 fetcher, the way that recipes using p4 had to be written seemed
> > > > > > rather awkward to me.  The way it is now isn't perfect, but it's
> > closer
> > > > > > to how the other fetchers act, I think (and hope).
> > > > > >
> > > > >
> > > > > > > > Is there a way to set module and label/changelist? I have
> > several
> > > > > > folders
> > > > > > > > > per recipe that I need to map to different subfolders and
> > with
> > > > the
> > > > > > current
> > > > > > > > > script some of the folders don't get fetched.
> > > > > >
> > > > > > I'm not sure I understand enough about how you use perforce and
> > what
> > > > you
> > > > > > mean by mapping different subfolders.  Can you explain this in a
> > more
> > > > > > expanded way to me?  Maybe with some examples?
> > > > > >
> > > > >
> > > > > What I mean by this is taking folders in different locations in
> > Perforce
> > > > > and fetch them to a custom folder structure inside the {WORKDIR}/p4
> > > > folder.
> > > > > This is an example of how I was doing this with the old version of
> > the
> > > > > fetcher:
> > > > >
> > > > > #leave blank, "", for latest revision
> > > > > P4CHANGELIST = "${PR}"
> > > > > S = "${WORKDIR}"
> > > > > SRC_URI = " \
> > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot/project
> > > > 1/folder1/...;module=subfolderunderp4/src/;label=${P4CHANGELIST}
> > > > > \
> > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot/project
> > > > 2/folder3/...;module=subfolderunderp4/pkg/;label=${P4CHANGELIST}
> > > > > \
> > > > > "
> > > > >
> > > > > Where folder1 contains the src files and folder3 contains the pkg
> > > > > information. These need to be in an specific folder structure for the
> > > > > recipe to work.
> > > > > I had some issue when adding more than one line because it was
> > trying to
> > > > > copy everything to the same p4 folder.
> > > >
> > > > Ah, yeah... Sorry, that is a bit of a shortcoming of my changes.  The
> > > > reason I made this change is because putting the given SRC_URI's
> > > > repository directly into a directory like ${WORKDIR}/p4/ is how some of
> > > > the other fetchers operate and it makes the automatic tasks just work
> > > > correctly.  This is how the git fetcher does things so I tried to
> > mimmic
> > > > that but maybe I missed some nuance in how the git fetcher deals with
> > > > more than one SRC_URI line being a git repo.  I hadn't realized that
> > > > there was users of the p4 fetcher who did things like you are doing,
> > > > sorry.
> > > >
> > > > I think it should be reasonably easy to make a patch to the p4 fetcher
> > > > which will put each SRC_URI line which uses the p4 fetcher into its own
> > > > directory within the ${WORKDIR}/p4/ directory such that it supports
> > your
> > > > style of fetching.  This then will likely require your recipe to know
> > > > that this is the directory structure and properly change into
> > > > directories as needed, but it sounds like you already comprehend that.
> > > > And for users of the p4 fetcher who only use one SRC_URI line, I
> > believe
> > > > they would just need to set the S variable correctly to the directory
> > > > name within ${WORKDIR}/p4/ to get the current operation.
> > >
> > > I have been looking at this and it wasn't straight forward for me... I
> > > don't have much experience with this. I will continue trying but I might
> > > need some help.
> > > It's not only not being able to fetch more than one SRC_URI. We are also
> > > loosing the flexibility of fetching different changelists for each line.
> > > I'm not sure if this can be solved in this new fetcher whilst keeping git
> > > and p4 "similar" as git doesn't work this way afaik.
> >
> > It sounds like you really just want to revert to how the p4 fetcher used
> > to work.  My changes were because I wanted the p4 fetcher to work more
> > like the git and svn fetchers.  But that doesn't sound like what you
> > want.
> >
> > My p4 fetcher changes which are impacting you on poky's pyro are these:
> >
> > 35081f55185a8a9804c21b16cd4600ba70646a10 bitbake: bitbake-user-manual:
> > Addeds support for the Perforce Fetcher
> > 40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 base.bbclass: p4 fetcher
> > supports srcrev
> > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: fetch2/perforce: Rework
> > to support SRCREV and P4CONFIG
> >
> > But that last one will run into conflicts if you just try to revert it
> > directly due to having other commits since then which git can't easily
> > figure out, but if you revert all of these commits too, then you may
> > have to redo some of the changes they made which are related to general
> > things and are not p4 specific.  Commits you'll want to look at are:
> >
> > b16192c93834d0a6530169557aa34122e1417bcf bitbake: fetch2: don't use
> > deprecated bb.data APIs
> > 6c611d697f9c03867c938cba1b481f38eebed8bf bitbake: fetch2: Rename
> > "setup_revisons" to "setup_revisions"
> > 38438b6cf42fb7ad45b9a901f57913af7e7591a3 bitbake: fetch2: obey
> > BB_ALLOWED_NETWORKS when checking network access
> > 1fce7ecbbb004a5ad82da3eef79cfd52b276708d bitbake: bitbake: remove True
> > option to getVar calls
> > ab09541d5517da9b1a23923ea8f5c26ddf745084 bitbake: fetch2: preserve
> > current working directory
> > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: fetch2/perforce: Rework
> > to support SRCREV and P4CONFIG
> >
> > I haven't actually reviewed any of these commits after the "preserve
> > current working directory" one as I'm still doing my day-to-day work on
> > the morty branch of oe-core and bitbake 1.32.  But possibly one of these
> > commits are unintentionally impacting your ability to specify SRCREV and
> > that's worth checking into.
> >
> > Thanks,
> > Andrew
> >