[yocto] Problems using MIRRORS variables

Ryan Harkin ryan.harkin at linaro.org
Wed Feb 20 10:03:35 PST 2019 

Hello,

I'm having a few problems when using the MIRRORS variable for GIT repos. I
see the same problems with the PREMIRRORS variable.

My use-case is:

1. company A has a private layer with SRC_URI that points to private git
2. company A needs to share layers with company B
3. company B cannot access company A's private git tree
4. company B has a mirror of the git trees with the same contents,
including SRC_URIs pointing to the company A private git trees

I have tried to resolve this using the MIRRORS variable in local.conf.

If SRCREV is a SHA, it seems to work for me.

However, it fails when the SRCREV is not a fixed SHA, eg. if SRCREV is a
branch, tag, or ${AUTOREV}. It looks like do_fetch is trying to pull the
repo to work out the SHA, but before decoding MIRRORS or PREMIRRORS. It
appears to do an ls-remote on the original repo, not the mirror, and fails.

The next issue is when using MIRRORS to replace a GIT repo with one using
SSH. This may simply be a syntax problem on my side.

Example:

MIRRORS += \
 git://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git \
 ssh://
git at git.linaro.org/landing-teams/working/qualcomm/wcnss-config.git;protocol=ssh
\n \
"

The MIRROR doesn't seem to be used at all in the above example. I've tried
adding ";protocol=https" to the first line; it doesn't help, and fails in a
different way, giving a "Exception: KeyError: 'protocol'" error.

I've shown detailed examples below. People may not need them, but if my
email isn't clear enough, hopefully they will help explain.

Setup
--------------------------------------------------
For an example, I'm using Linaro's OE-RPB master. More info here:
https://github.com/96boards/oe-rpb-manifest

First, set up the environment:

  $ mkdir -p ${WORKSPACE}
  $ repo init -u https://github.com/96boards/oe-rpb-manifest.git -b master
  $ repo sync
  $ export MACHINE=dragonboard-410c ; export DISTRO=rpb ; .
setup-environment

Then, make sure it's working:

  $ bitbake -c do_fetch wcnss-config

Mod a recipe to use a fake SRC_URI, and test that it no longer works.

  $ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#git.linaro.org#git.fake.org#g'
  $ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
  $ bitbake -c do_fetch wcnss-config

I see this error output, as expected:
----------------------------------------------------
WARNING: wcnss-config-1.13-r0 do_fetch: Failed to fetch URL git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https,
attempting MIRRORS if available
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; LANG=C git -c core.fsyncobjectfiles=0 clone
--bare --mirror
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git
--progress failed with exit code 128, output:
Cloning into bare repository
'/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git'...
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org

ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure for URL: 'git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https'.
Unable to fetch URL from any source.
ERROR: wcnss-config-1.13-r0 do_fetch:
ERROR: wcnss-config-1.13-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in:
/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/temp/log.do_fetch.2525
ERROR: Task
(/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/wcnss-config_1.13.bb:do_fetch)
failed with exit code '1'
----------------------------------------------------


SRCREV PROBLEM
--------------------------------------------------
Set MIRRORS in local.conf and re-test. It should work again.

$ cat << EOF >> conf/local.conf
MIRRORS += " \\
 git://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git \\
 git://git.linaro.org/landing-teams/working/qualcomm/wcnss-config.git \n \\
"
EOF
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config

All I see in the build is a warning, and the downloads/git2 directory is
populated as expected:

$ ll
total 12K
lrwxrwxrwx 1 ryan ryan  110 Feb 20 17:15
git.fake.org.landing-teams.working.qualcomm.wcnss-config.git ->
/linaro/oe-rpb-workspace-testing/downloads/git2/git.linaro.org.landing-teams.working.qualcomm.wcnss-config.git/
-rw-rw-r-- 1 ryan ryan    6 Feb 20 17:15
git.fake.org.landing-teams.working.qualcomm.wcnss-config.git.done
drwxrwxr-x 7 ryan ryan 4.0K Feb 20 17:15
git.linaro.org.landing-teams.working.qualcomm.wcnss-config.git/

Change SRCREV to ${AUTOREV} and retest.

$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#^SRCREV.*#SRCREV = "${AUTOREV}"#'
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config

I see this failure:

ERROR: ExpansionError during parsing
/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb######################################
 | ETA:  0:00:05
Traceback (most recent call last):
  File
"/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/__init__.py", line
1170, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
name='default'):
         if srcrev == "AUTOINC":
    >        srcrev = ud.method.latest_revision(ud, d, name)

  File
"/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/__init__.py", line
1565, in Git.latest_revision(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
name='default'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/git.py",
line 594, in Git._latest_revision(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
name='default'):
             """
    >        output = self._lsremote(ud, d, "")
             # Tags of the form ^{} may not work, need to fallback to other
form
  File "/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/git.py",
line 583, in Git._lsremote(ud=<bb.fetch2.FetchData object at
0x7fd7884c8668>, d=<bb.data_smart.DataSmart object at 0x7fd78988e748>,
search=''):
                     bb.fetch2.check_network_access(d, cmd, repourl)
    >            output = runfetchcmd(cmd, d, True)
                 if not output:
  File
"/linaro/oe-rpb-workspace-testing/bitbake/lib/bb/fetch2/__init__.py", line
888, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; git -c core.fsyncobjectfiles=0 ls-remote
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git ',
d=<bb.data_smart.DataSmart object at 0x7fd78988e748>, quiet=True,
cleanup=[], log=None, workdir=None):

    >        raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression
was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; git -c core.fsyncobjectfiles=0 ls-remote
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
failed with exit code 128, output:
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org

A similar error happens if you use a tag rather than ${AUTOREV}:

$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#^SRCREV.*#SRCREV = "debian/1.13"#'
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config

These are the messages I see:

ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; git -c core.fsyncobjectfiles=0 ls-remote
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
failed with exit code 128, output:
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org

ERROR: wcnss-config-1.13-r0 do_fetch:
ERROR: wcnss-config-1.13-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in:
/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/temp/log.do_fetch.5673
ERROR: Task
(/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/wcnss-config_1.13.bb:do_fetch)
failed with exit code '1'



SSH MIRRORS Problem
--------------------------------------------
>From a clean setup as above with the SRC_URI point to to git.fake.org...

$ sed -i ${WORKSPACE}/layers/meta-qcom/recipes-bsp/wcnss-config/
wcnss-config_1.13.bb -e 's#git.linaro.org#git.fake.org#g'
$ cat << EOF >> conf/local.conf
MIRRORS += " \\
 git://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git \\
 ssh://
git at git.linaro.org/landing-teams/working/qualcomm/wcnss-config.git;protocol=ssh
\n \\
"
EOF
$ rm -rf tmp-rpb-glibc/ ../downloads/git2/*wcnss-config*
$ bitbake -c do_fetch wcnss-config

These are the errors I see:

WARNING: wcnss-config-1.13-r0 do_fetch: Failed to fetch URL git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https,
attempting MIRRORS if available
ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
SSH_AGENT_PID="3371"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh";
export
PATH="/linaro/oe-rpb-workspace-testing/layers/openembedded-core/scripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin/allarch-linaro-linux:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot/usr/bin/crossscripts:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/usr/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/sbin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/recipe-sysroot-native/bin:/linaro/oe-rpb-workspace-testing/bitbake/bin:/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/hosttools";
export HOME="/home/ryan"; LANG=C git -c core.fsyncobjectfiles=0 clone
--bare --mirror
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git
/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git
--progress failed with exit code 128, output:
Cloning into bare repository
'/linaro/oe-rpb-workspace-testing/downloads/git2/git.fake.org.landing-teams.working.qualcomm.wcnss-config.git'...
fatal: unable to access '
https://git.fake.org/landing-teams/working/qualcomm/wcnss-config.git/':
Could not resolve host: git.fake.org

ERROR: wcnss-config-1.13-r0 do_fetch: Fetcher failure for URL: 'git://
git.fake.org/landing-teams/working/qualcomm/wcnss-config.git;branch=master;protocol=https'.
Unable to fetch URL from any source.
ERROR: wcnss-config-1.13-r0 do_fetch:
ERROR: wcnss-config-1.13-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in:
/linaro/oe-rpb-workspace-testing/build-rpb/tmp-rpb-glibc/work/all-linaro-linux/wcnss-config/1.13-r0/temp/log.do_fetch.7701
ERROR: Task
(/linaro/oe-rpb-workspace-testing/build-rpb/conf/../../layers/meta-qcom/recipes-bsp/wcnss-config/wcnss-config_1.13.bb:do_fetch)
failed with exit code '1'

It looks like the MIRRORS variable hasn't replaced the repo at all.

Regards,
Ryan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190220/64b8ee7d/attachment-0001.html>

More information about the yocto mailing list