[yocto] bitbake SRC_URI fetch Azure DevOps repository Azure DevOps Services Basic

Sun Nov 3 18:52:39 PST 2019

Dear yocto developer,
The below disscussed about azure devops clone issue including test result with Microsoft support team member.
We wonder know how Bitbake access repository through SSH. Does it different between git and Azue DevOps?

Thanks,

Samuel Jiang
---------- Forwarded message ----------
From: Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com>
Date: 2019年10月31日 PM5:39 [+0800]
To: Samuel Jiang (江騏先) <Samuel.Jiang at quantatw.com>
Cc: support <support at mail.support.microsoft.com>, Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com>
Subject: RE: 119102923000220 use yocto bitbake SRC_URI fetch Azure DevOps repository Azure DevOps Services Basic

Hi Samuel,

Thank you for reply.

I test and read the BitBake doc, let me share with you my finding and analysis.

  1.  I have verified that our Azure DevOps Organization Repo could be clone through SSH authentication successfully.

  1.  I create a new SSH key and add it to Azure DevOps SSH public keys.
[cid:image008.png at 01D5900C.6DB2CF60] [cid:image009.png at 01D5900C.6DB2CF60]

  1.  Then I use git clone command and clone this repo successfully. The url is “v-chucho-microsoft at vs-ssh.visualstudio.com:v3/v-chucho-microsoft/testCodeCoverage/testCodeCoverage”
[cid:image011.png at 01D5900C.C1971910]
My testing verify that SSH Authentication is available, and the url given by Azure DevOps is correct.

  1.  Then I also test it in GitHub, and succeed. The only difference is the url of GitHub is “git at github.com:xxxxxxxxxxxxxxxxxx.git”

  1.  So let’s come back to BitBake. I read more of the BitBake doc.( https://www.yoctoproject.org/docs/1.8/bitbake-user-manual/bitbake-user-manual.html#bb-fetchers) . Your command SRC_URI = "git://quanta01@vs-ssh.visualstudio.com:v3/quanta01/OpenBMC/crashdump;protocol=ssh;nobranch=1". The BitBake example is below.
[cid:image012.png at 01D59011.A1A7F340]
We can see that the BitBake syntax is like git SSH but not Azure DevOps SSH. Do you try to use this command clone a repo from GitHub?

  1.  Then I read the doc you shared. The customer also met your issue when trying to clone Azure DevOps repo with SSH url. In my understanding, seems BitBake Git fetcher is good for Git SSH but not ready to access the Azure DevOps SSH.
[cid:image010.png at 01D5900C.6DB2CF60]

Action Plan
Could you involve BitBake support team? We can deliver this testing result and consult them how BitBake access Azure DevOps through SSH.

If you have any conern or query, please feel free to let me know.

Best Regards,

Alex Chong

[image001]
Support Engineer
Microsoft APAC Developer Support Team
Customer Service & Support (CSS)
Email: v-chucho at microsoft.com<mailto:v-chucho at microsoft.com>
Office: +86 (21) 52638610
Time zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
Working time: 9:00am-6:00pm, Mon-Tue-Wed-Thu-Fri

From: Samuel Jiang (江騏先) <Samuel.Jiang at quantatw.com>
Sent: Thursday, October 31, 2019 12:16 PM
To: Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com>
Cc: support <support at mail.support.microsoft.com>
Subject: RE: 119102923000220 use yocto bitbake SRC_URI fetch Azure DevOps repository Azure DevOps Services Basic

Hi Alex,

I catch same problem on the yocto mail list
link: https://lists.yoctoproject.org/pipermail/yocto/2018-October/042736.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.yoctoproject.org%2Fpipermail%2Fyocto%2F2018-October%2F042736.html&data=02%7C01%7Cv-chucho%40microsoft.com%7C5b51185a798a4fdf6cd608d75db90d70%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637080921717407210&sdata=Uf0%2Fq1FwlmsXRaoFfMbDfpiWKNzN3MgBlgZXphM3jwo%3D&reserved=0>

when I use SRC_URI = "git://quanta01@vs-ssh.visualstudio.com:v3/quanta01/OpenBMC/crashdump;protocol=ssh;nobranch=1"

the bitbake response below error message:
git -c core.fsyncobjectfiles=0 ls-remote ssh://quanta01@vs-ssh.visualstudio.com:v3/quanta01/OpenBMC/crashdump  failed with exit code 128, output:
ssh: Could not resolve hostname vs-ssh.visualstudio.com:v3: Name or service not known
fatal: Could not read from remote repository.

I think the bitbake call “git ls-remote” command however it could not parser below host vs-ssh.visualstudio.com:v3
Could you help confirm the  “git ls-remote”command could work on ssh method clone with ssh:// prefix?

BTW, the protocol will define the uri prefix, I didn’t find any way removing this prefix.

Thanks,
Samuel Jiang

From: Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com>
Sent: Thursday, October 31, 2019 10:32 AM
To: Samuel Jiang (江騏先) <Samuel.Jiang at quantatw.com>
Cc: Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com>; support <support at mail.support.microsoft.com>
Subject: RE: 119102923000220 use yocto bitbake SRC_URI fetch Azure DevOps repository Azure DevOps Services Basic

Hi Samuel,

How are you?

Do you try our Azure DevOps SSH authentication? Do you contact BitBake support for help? Is there any suggestion or new issue? Please feel free to contact me if there is something I can do for you.

Best Regards,

Alex Chong

[image001]
Support Engineer
Microsoft APAC Developer Support Team
Customer Service & Support (CSS)
Email: v-chucho at microsoft.com<mailto:v-chucho at microsoft.com>
Office: +86 (21) 52638610
Time zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
Working time: 9:00am-6:00pm, Mon-Tue-Wed-Thu-Fri

From: Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com<mailto:v-chucho at microsoft.com>>
Sent: Tuesday, October 29, 2019 3:50 PM
To: Samuel.Jiang at quantatw.com<mailto:Samuel.Jiang at quantatw.com>
Cc: Alex Chong (Shanghai Wicresoft Co,.Ltd.) <v-chucho at microsoft.com<mailto:v-chucho at microsoft.com>>
Subject: 119102923000220 use yocto bitbake SRC_URI fetch Azure DevOps repository Azure DevOps Services Basic

[Image removed by sender. 郵件模板標頭]

Hi Samuel,

Thank you for contacting Microsoft.

My name is Alex and I am the Microsoft Support Engineer from APGC who will be working with you on this support request . You can reach me using the contact information in my signature.

I just called you at +886 3 327 2345 and had a quick discussion.
Issue Description
1.Customer is using a 3rd-party application BitBake to clone Azure DevOps repo.
2.Customer could fetch file from https url successfully.
3.Customer wants to know whether Azure DevOps support SSH authentication method.

Analysis
1.Yes, Azure DevOps has SSH authentication method. (Reference: https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdevops%2Frepos%2Fgit%2Fuse-ssh-keys-to-authenticate%3Fview%3Dazure-devops&data=02%7C01%7Cv-chucho%40microsoft.com%7C5b51185a798a4fdf6cd608d75db90d70%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637080921717417205&sdata=RT5w4vibwi5VapG3Sr8cDthwGeu%2BudWf98s9FdsbxP4%3D&reserved=0>)

  *   Create your SSH keys
    Create your SSH keys with the ssh-keygen command from the bash prompt.

  *   Add the public key to your Azure DevOps Organization
Open your security settings by browsing to the web portal and selecting your avatar in the upper right of the user interface. Select Security in the menu that appears.
[cid:image002.png at 01D58FF1.31F5A490]
Select SSH public keys , then select +New Key.
[cid:image003.png at 01D58FF1.31F5A490]
Copy the contents of the public key (for example, id_rsa.pub) that you generated into the Public Key Data field.

[cid:image004.png at 01D58FF1.31F5A490]

  *   Clone the Git repository with SSH
Copy the SSH clone URL from the web portal. In this example the SSL clone URL is for a repo in an organization named fabrikam-fiber, as indicated by the first part of the URL after dev.azure.com.
[cid:image005.png at 01D58FF1.31F5A490]
Run git clone from the command prompt.
My example is git clone v-chucho-microsoft at vs-ssh.visualstudio.com:v3/v-chucho-microsoft/testCodeCoverage/testCodeCoverage<mailto:v-chucho-microsoft at vs-ssh.visualstudio.com:v3/v-chucho-microsoft/testCodeCoverage/testCodeCoverage>

2.Regarding to BitBake. (Reference: https://www.yoctoproject.org/docs/1.8/bitbake-user-manual/bitbake-user-manual.html#bb-fetchers<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.yoctoproject.org%2Fdocs%2F1.8%2Fbitbake-user-manual%2Fbitbake-user-manual.html%23bb-fetchers&data=02%7C01%7Cv-chucho%40microsoft.com%7C5b51185a798a4fdf6cd608d75db90d70%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637080921717417205&sdata=pAVBdFRKoXlV2dcCBZi4aHVvIWN%2BfLYhnXg2Yf1fbYY%3D&reserved=0>)

This is a 3rd-party application which is out of our Microsoft support. Our Azure DevOps has SSH authentication method. But I have no experience whether there is such option in BitBake.

I did some research on this application and find Git fetcher and ssh:// fetcher functionalities. In my understanding, it should support git with SSH or SSH fecher directly. It's suggested to consult BitBake support for more details of these 2 fetchers. If there is something our Microsoft could help, please feel free to involve me directly.
[cid:image006.png at 01D58FF1.31F5A490]
[cid:image007.png at 01D58FF1.31F5A490]

If you have any concern or query, please feel free to contact me.

Best Regards,

Alex Chong

Support Engineer
Microsoft APAC Developer Support Team
Customer Service & Support (CSS)
Email: v-chucho at microsoft.com<mailto:v-chucho at microsoft.com>
Office: +86 (21) 52638610
Time zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
Working time: 9:00am-6:00pm, Mon-Tue-Wed-Thu-Fri

這則來自 Microsoft 的訊息是您或貴公司所購買或參與之計畫、服務或產品的重要部分。請檢閱我們的線上隱私權聲明<https://go.microsoft.com/fwlink/?LinkId=521839>

Microsoft Corporation
One Microsoft Way
Redmond, WA, USA 98052

[Image removed by sender. Microsoft]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 29915 bytes
Desc: image005.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1999 bytes
Desc: image001.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 141341 bytes
Desc: image003.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.png
Type: image/png
Size: 5383 bytes
Desc: image010.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image012.png
Type: image/png
Size: 4298 bytes
Desc: image012.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 44254 bytes
Desc: image006.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.png
Type: image/png
Size: 15632 bytes
Desc: image008.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 37907 bytes
Desc: image004.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0019.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image009.png
Type: image/png
Size: 10853 bytes
Desc: image009.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0020.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image011.png
Type: image/png
Size: 10066 bytes
Desc: image011.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0021.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD331.jpg
Type: image/jpeg
Size: 823 bytes
Desc: ~WRD331.jpg
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 19994 bytes
Desc: image007.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0022.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 28693 bytes
Desc: image002.png
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191104/34000569/attachment-0023.png>