[yocto] Perforce fetcher ignores module and label

Mon Sep 18 03:32:57 PDT 2017

Hi Andrew,

I'm getting an error when I apply your patch. Do you now how can I work
around this?

user at pc:~/yocto/pyro/poky$ git fetch
user at pc:~/yocto/pyro/poky$ git reset
user at pc:~/yocto/pyro/poky$ git clean -fd
user at pc:~/yocto/pyro/poky$ git reset --hard
HEAD is now at b51b4f5 gawk: Enable native building
user at pc:~/yocto/pyro/poky$ git branch
* pyro
user at pc:~/yocto/pyro/poky$ git apply
../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch
../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:28:
trailing whitespace.
                  'DBUS_SESSION_BUS_ADDRESS']
../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:41:
trailing whitespace.
BitBake 'Fetch' implementations
../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:42:
trailing whitespace.

../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:43:
trailing whitespace.
Classes for obtaining upstream sources for the
../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:44:
trailing whitespace.
BitBake build tools.
error: patch failed: bitbake/lib/bb/fetch2/__init__.py:817
error: bitbake/lib/bb/fetch2/__init__.py: patch does not apply
error: patch failed: bitbake/lib/bb/fetch2/perforce.py:1
error: bitbake/lib/bb/fetch2/perforce.py: patch does not apply

Cheers,
Katu

2017-09-14 17:17 GMT+01:00 Andrew Bradford <andrew at bradfordembedded.com>:

> Hi Katu,
>
> Sorry for my delay in responding.
>
> On 09/14 11:13, Katu Txakur wrote:
> > Have you had time to look at this? I tried to go back to the old perforce
> > fetcher but I got a license error and I couldn't work around it.
>
> On the pyro branch of poky, if you revert these two commits, in this
> order:
>
> 35081f55185a8a9804c21b16cd4600ba70646a10 "bitbake: bitbake-user-manual:
> Addeds support for the Perforce Fetcher"
> 40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 "base.bbclass: p4 fetcher
> supports srcrev"
>
> And then apply the patch attached, this should bring the pyro branch of
> poky back to how the old perforce fetcher worked.  I have tested having
> two different depot paths in a recipe SRC_URI and they both are fetched
> how I believe you want them to be, next to each other in WORKDIR.
>
> Just be aware, I believe this loses the ability to use
> SRC_REV="${AUTOREV}" to automatically find the newest changelist which
> my previous changes introduced.
>
> The format for SRC_URI that I tested was like:
>
> SRC_URI = "p4://username:password:server.hostname.example.com:1666 at depot
> /path/to/directory/...;cset=178374"
>
> You should also be able to specify P4PORT to set the host and port
> number but you may then lose the ability to have the username and
> password (although I haven't tested this and don't remember how it used
> to work).  You should also be able to specify the P4DATE variable in
> your recipe to apply to all p4 fetches instead of using the "cset="
> parameter, and instead of using the "cset=" param you should also be
> able to specify a "revision=" for a single file or a "label=" for a
> label.  I've only tested using the "cset=" way as the others don't
> easily apply to how my team internally uses perforce, sorry.
>
> I think I understand how I could make the current perforce fetcher
> (without the above reverts or attached patch) do the multi-directory
> fetching that you want, but I don't personally want to do that in my
> workflow so I'm not going to spend a bunch of time implementing it now.
> But if you do implement it, I'd be happy to test patches for you.
>
> Thanks,
> Andrew
>
> > 2017-08-31 18:54 GMT+01:00 Andrew Bradford <andrew at bradfordembedded.com
> >:
> >
> > > Hi Katu,
> > >
> > > On 08/28 17:43, Katu Txakur wrote:
> > > > Thanks for looking at this. See my recipe below. I've left the bits
> > > related
> > > > to systemd service but I don't think they matter for this. I'm using
> an
> > > old
> > > > implementation of Perforce (2010) in case this matters. I've tried
> going
> > > > back to the old perforce.py fetcher but I get a license error... do
> you
> > > > know if it would be easy to revert to the old version in my bitbake
> > > folder
> > > > until we make this work?
> > >
> > > Sorry, I've been swamped this week and haven't been able to look into
> > > this yet.  This coming weekend is a holiday weekend in the USA, too.
> > > But I plan to look at this early next week, hope that's OK.
> > >
> > > We should be able to create a patch series to revert my changes so you
> > > can go back to the old perforce fetcher.  It might also be worth
> > > investigating how to take the current perforce fetcher and enable some
> > > of the use cases that you have (but we can do this after we
> successfully
> > > revert).
> > >
> > > I'll try to send a patch to you next week for the reverting.
> > > Thanks,
> > > Andrew
> > >
> > > > DESCRIPTION = "Dummy recipe to fetch from Perforce"
> > > > SECTION = "PerforceRecipes"
> > > > LICENSE = "CLOSED"
> > > > PR = "r0"
> > > >
> > > > inherit cmake
> > > > DEPENDS = "boost alsa-tools"
> > > >
> > > > P4USER = "myuser"
> > > > P4PASSWD = ""
> > > > P4PORT = "192.168.1.55:1666"
> > > > FETCHCMD_p4 = "/usr/local/bin/p4"
> > > >
> > > > SRC_URI = " \
> > > > p4://${P4USER}:${P4PASSWD}@myrepo/myfolder/...; \
> > > > file://perforce-recipe.service \
> > > > "
> > > > SRCREV = "${AUTOREV}"
> > > > S = "${WORKDIR}/p4"
> > > >
> > > > do_install_append () {
> > > >     install -d ${D}${systemd_unitdir}/system
> > > >     install -m 0644 ../perforce-recipe.service
> > > ${D}${systemd_unitdir}/system
> > > > }
> > > >
> > > > NATIVE_SYSTEMD_SUPPORT = "1"
> > > > SYSTEMD_PACKAGES = "perforce-recipe"
> > > > SYSTEMD_SERVICE_${PN} = "perforce-recipe.service"
> > > >
> > > > SYSTEMD_AUTO_ENABLE = "disable"
> > > >
> > > > FILES_${PN} = "${systemd_unitdir} ${bindir}"
> > > >
> > > >
> > > > 2017-08-28 15:27 GMT+01:00 Andrew Bradford <
> andrew at bradfordembedded.com
> > > >:
> > > >
> > > > > Hi Katu,
> > > > >
> > > > > On 08/28 08:56, Katu Txakur wrote:
> > > > > > 2017-08-11 13:52 GMT+01:00 Andrew Bradford <
> > > andrew at bradfordembedded.com
> > > > > >:
> > > > > > On 08/11 11:34, Katu Txakur wrote:
> > > > > > > > 2017-08-07 12:17 GMT+01:00 Andrew Bradford <
> > > > > andrew at bradfordembedded.com
> > > > > > > >:
> > > > > > > > > On 08/02 10:28, Paul Eggleton wrote:
> > > > > > > > > > On Wednesday, 2 August 2017 9:40:10 AM CEST Katu Txakur
> > > wrote:
> > > > > > > > > > > I'm still having problems fetching from Perforce. Is
> there
> > > any
> > > > > > > > > > > documentation based on the latest implementation of the
> > > > > > > > > > > poky/bitbake/lib/bb/fetch2/perforce.py file?
> > > > > > > > >
> > > > > > > > > There's some documentation now in the official Yocto
> Project
> > > > > > > > > documentation location for bitbake fetchers [1].  Prior to
> my
> > > > > changes
> > > > > > > > > there was very little if any documentation, afaik.
> > > > > > > > >
> > > > > > > > > [1]: http://www.yoctoproject.org/docs/2.3.1/bitbake-user-
> > > > > > > > > manual/bitbake-user-manual.html#perforce-fetcher
> > > > > > > > >
> > > > > > > > > > > 2017-07-25 10:05 GMT+01:00 Katu Txakur <
> > > katutxakurra at gmail.com
> > > > > >:
> > > > > > > > > > > > I'm upgrading a recipe that fetches the source code
> from
> > > > > > > Perforce.
> > > > > > > > > > > >
> > > > > > > > > > > > The old recipe was:
> > > > > > > > > > > >
> > > > > > > > > > > > SRC_URI = " \
> > > > > > > > > > > >   p4://${P4USER}:${P4PASSWD}:${
> P4HOST}:${P4PORT}@Depot
> > > /path/
> > > > > pe
> > > > > > > > > > > > rforce/...;module=local/path/relativeto/p4;label=${
> > > > > P4CHANGELIST}
> > > > > > > \
> > > > > > > > > > > >   "
> > > > > > > > > > > >
> > > > > > > > > > > > With the new version of /lib/bb/fetch2/perforce.py, I
> > > had to
> > > > > set
> > > > > > > > > P4PORT
> > > > > > > > > > > > outside SRC_URI, leaving the recipe with:
> > > > > > > > > > > >
> > > > > > > > > > > > SRC_URI = " \
> > > > > > > > > > > >   p4://${P4USER}:${P4PASSWD}@
> Depot/path/perforce/...;
> > > module=
> > > > > > > > > > > > local/path/relativeto/p4;label=${P4CHANGELIST} \
> > > > > > > > > > > >   "
> > > > > > > > > > > >
> > > > > > > > > > > > The Perforce fetcher kind of works, but it seems to
> be
> > > > > ignoring
> > > > > > > the
> > > > > > > > > > > > "module" and the "label" attributes. After reading
> the
> > > Python
> > > > > > > script
> > > > > > > > > I can
> > > > > > > > > > > > see that after the "@", only the substring before the
> > > first
> > > > > ";"
> > > > > > > is
> > > > > > > > > used.
> > > > > > > > >
> > > > > > > > > Sorry for not making the label change more clear, but you
> > > should be
> > > > > > > able
> > > > > > > > > to simply set SRCREV="labelname" and have the
> functionality you
> > > > > desire.
> > > > > > > > > However, we don't use labels very often internally, so the
> use
> > > of
> > > > > > > labels
> > > > > > > > > in SRCREV isn't tested that often by me.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I don't use labels that much either, however, I used to be
> able
> > > to
> > > > > write
> > > > > > > > the changeslist number as a label and it would
> > > > > > > > fetch the code for that changelist. This doesn't work using
> > > SRCREV =
> > > > > > > > "changelistnumber"
> > > > > > >
> > > > > > > This does work for me using oe-core morty branch and bitbake
> > > > > > > 1.32 which is the versions where these changes were first
> applied
> > > and
> > > > > > > where I did my testing.
> > > > > > >
> > > > > > > Like I have a recipe which I put:
> > > > > > >
> > > > > > > SRCREV = "184127"
> > > > > > >
> > > > > > > The p4 fetcher will go fetch changelist 184127 of the SRC_URI
> path
> > > in
> > > > > > > perforce.
> > > > > > >
> > > > > > > Or if I put:
> > > > > > >
> > > > > > > SRCREV = "${AUTOREV}"
> > > > > > >
> > > > > > > Then it'll fetch the HEAD/tip of the SRC_URI path in perforce.
> > > > > > >
> > > > > > > Is this not working the same way with poky now?
> > > > > > >
> > > > > >
> > > > > > I'm using yocto pyro with bitbake 1.34.0. When I first fetch the
> > > code for
> > > > > > my recipe, it gets the latest changelist.
> > > > > > I submit a change and the next time, the recipe thinks there
> nothing
> > > new
> > > > > to
> > > > > > fetch. Also, using SRCREV = "184127"
> > > > > > doesn't work for me, it always gets the latest changelist.
> > > > >
> > > > > Can you post the recipe where this is failing for you?
> > > > >
> > > > > I could then try to recreate this issue in a sandbox depot we have
> in
> > > > > our perforce server and see what's going on.
> > > > >
> > > > > > >
> > > > > > > > > This change to using labels, changelist numbers, and
> allowing
> > > the
> > > > > use
> > > > > > > of
> > > > > > > > > AUTOREV was made so that the perforce fetcher would act
> more
> > > like
> > > > > the
> > > > > > > > > other source code control system fetchers.  Prior to my
> > > changes to
> > > > > the
> > > > > > > > > p4 fetcher, the way that recipes using p4 had to be written
> > > seemed
> > > > > > > > > rather awkward to me.  The way it is now isn't perfect, but
> > > it's
> > > > > closer
> > > > > > > > > to how the other fetchers act, I think (and hope).
> > > > > > > > >
> > > > > > > >
> > > > > > > > > > > Is there a way to set module and label/changelist? I
> have
> > > > > several
> > > > > > > > > folders
> > > > > > > > > > > > per recipe that I need to map to different
> subfolders and
> > > > > with
> > > > > > > the
> > > > > > > > > current
> > > > > > > > > > > > script some of the folders don't get fetched.
> > > > > > > > >
> > > > > > > > > I'm not sure I understand enough about how you use
> perforce and
> > > > > what
> > > > > > > you
> > > > > > > > > mean by mapping different subfolders.  Can you explain
> this in
> > > a
> > > > > more
> > > > > > > > > expanded way to me?  Maybe with some examples?
> > > > > > > > >
> > > > > > > >
> > > > > > > > What I mean by this is taking folders in different locations
> in
> > > > > Perforce
> > > > > > > > and fetch them to a custom folder structure inside the
> > > {WORKDIR}/p4
> > > > > > > folder.
> > > > > > > > This is an example of how I was doing this with the old
> version
> > > of
> > > > > the
> > > > > > > > fetcher:
> > > > > > > >
> > > > > > > > #leave blank, "", for latest revision
> > > > > > > > P4CHANGELIST = "${PR}"
> > > > > > > > S = "${WORKDIR}"
> > > > > > > > SRC_URI = " \
> > > > > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot/project
> > > > > > > 1/folder1/...;module=subfolderunderp4/src/;label=${
> P4CHANGELIST}
> > > > > > > > \
> > > > > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot/project
> > > > > > > 2/folder3/...;module=subfolderunderp4/pkg/;label=${
> P4CHANGELIST}
> > > > > > > > \
> > > > > > > > "
> > > > > > > >
> > > > > > > > Where folder1 contains the src files and folder3 contains
> the pkg
> > > > > > > > information. These need to be in an specific folder structure
> > > for the
> > > > > > > > recipe to work.
> > > > > > > > I had some issue when adding more than one line because it
> was
> > > > > trying to
> > > > > > > > copy everything to the same p4 folder.
> > > > > > >
> > > > > > > Ah, yeah... Sorry, that is a bit of a shortcoming of my
> changes.
> > > The
> > > > > > > reason I made this change is because putting the given
> SRC_URI's
> > > > > > > repository directly into a directory like ${WORKDIR}/p4/ is how
> > > some of
> > > > > > > the other fetchers operate and it makes the automatic tasks
> just
> > > work
> > > > > > > correctly.  This is how the git fetcher does things so I tried
> to
> > > > > mimmic
> > > > > > > that but maybe I missed some nuance in how the git fetcher
> deals
> > > with
> > > > > > > more than one SRC_URI line being a git repo.  I hadn't realized
> > > that
> > > > > > > there was users of the p4 fetcher who did things like you are
> > > doing,
> > > > > > > sorry.
> > > > > > >
> > > > > > > I think it should be reasonably easy to make a patch to the p4
> > > fetcher
> > > > > > > which will put each SRC_URI line which uses the p4 fetcher into
> > > its own
> > > > > > > directory within the ${WORKDIR}/p4/ directory such that it
> supports
> > > > > your
> > > > > > > style of fetching.  This then will likely require your recipe
> to
> > > know
> > > > > > > that this is the directory structure and properly change into
> > > > > > > directories as needed, but it sounds like you already
> comprehend
> > > that.
> > > > > > > And for users of the p4 fetcher who only use one SRC_URI line,
> I
> > > > > believe
> > > > > > > they would just need to set the S variable correctly to the
> > > directory
> > > > > > > name within ${WORKDIR}/p4/ to get the current operation.
> > > > > >
> > > > > > I have been looking at this and it wasn't straight forward for
> me...
> > > I
> > > > > > don't have much experience with this. I will continue trying but
> I
> > > might
> > > > > > need some help.
> > > > > > It's not only not being able to fetch more than one SRC_URI. We
> are
> > > also
> > > > > > loosing the flexibility of fetching different changelists for
> each
> > > line.
> > > > > > I'm not sure if this can be solved in this new fetcher whilst
> > > keeping git
> > > > > > and p4 "similar" as git doesn't work this way afaik.
> > > > >
> > > > > It sounds like you really just want to revert to how the p4 fetcher
> > > used
> > > > > to work.  My changes were because I wanted the p4 fetcher to work
> more
> > > > > like the git and svn fetchers.  But that doesn't sound like what
> you
> > > > > want.
> > > > >
> > > > > My p4 fetcher changes which are impacting you on poky's pyro are
> these:
> > > > >
> > > > > 35081f55185a8a9804c21b16cd4600ba70646a10 bitbake:
> bitbake-user-manual:
> > > > > Addeds support for the Perforce Fetcher
> > > > > 40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 base.bbclass: p4 fetcher
> > > > > supports srcrev
> > > > > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: fetch2/perforce:
> > > Rework
> > > > > to support SRCREV and P4CONFIG
> > > > >
> > > > > But that last one will run into conflicts if you just try to
> revert it
> > > > > directly due to having other commits since then which git can't
> easily
> > > > > figure out, but if you revert all of these commits too, then you
> may
> > > > > have to redo some of the changes they made which are related to
> general
> > > > > things and are not p4 specific.  Commits you'll want to look at
> are:
> > > > >
> > > > > b16192c93834d0a6530169557aa34122e1417bcf bitbake: fetch2: don't
> use
> > > > > deprecated bb.data APIs
> > > > > 6c611d697f9c03867c938cba1b481f38eebed8bf bitbake: fetch2: Rename
> > > > > "setup_revisons" to "setup_revisions"
> > > > > 38438b6cf42fb7ad45b9a901f57913af7e7591a3 bitbake: fetch2: obey
> > > > > BB_ALLOWED_NETWORKS when checking network access
> > > > > 1fce7ecbbb004a5ad82da3eef79cfd52b276708d bitbake: bitbake: remove
> True
> > > > > option to getVar calls
> > > > > ab09541d5517da9b1a23923ea8f5c26ddf745084 bitbake: fetch2: preserve
> > > > > current working directory
> > > > > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: fetch2/perforce:
> > > Rework
> > > > > to support SRCREV and P4CONFIG
> > > > >
> > > > > I haven't actually reviewed any of these commits after the
> "preserve
> > > > > current working directory" one as I'm still doing my day-to-day
> work on
> > > > > the morty branch of oe-core and bitbake 1.32.  But possibly one of
> > > these
> > > > > commits are unintentionally impacting your ability to specify
> SRCREV
> > > and
> > > > > that's worth checking into.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20170918/c77ce730/attachment.html>