[meta-virtualization] [Resolved] Re: cgroups and iptables problems running docker - maybe my config wrong?

Bruce Ashfield bruce.ashfield at windriver.com
Wed Jun 6 07:35:17 PDT 2018 

On 2018-06-06 10:33 AM, Jakob Hasse wrote:
> Hello Bruce, hello Yoctoers,
> 
> my problem is solved. It was the kernel options. I added all additional 
> kernel options which this check-script here suggests: 
> https://github.com/moby/moby/blob/master/contrib/check-config.sh. This 
> script was a big help
> 
> Finally, docker runs, with one ERROR message and one WARNING though, but 
> it does :) If I have more problems, I'll come back to the list.
> 

Great news.

I was about to do another run of this, and still will, but this is
good to know.

Definitely let us know what breaks next :D

Bruce

> Thank you for the help so far.
> 
> All the Best,
> Jakob
> 
> On 05.06.2018 18:23, Jakob Hasse wrote:
>> Hello,
>>
>> right now, there is no insmod warning anymore, only iptables does 
>> still not work with the nat table:
>>
>> root at ccimx6uluvalue:~# dockerd
>> INFO[0000] libcontainerd: new containerd process, pid: 1003
>> WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 
>> max=4096
>> INFO[0001] [graphdriver] using prior storage driver: overlay2
>> INFO[0001] Graph migration to content-addressability took 0.00 seconds
>> WARN[0001] Your kernel does not support cgroup memory limit
>> WARN[0001] Unable to find cpu cgroup in mounts
>> WARN[0001] Unable to find blkio cgroup in mounts
>> WARN[0001] Unable to find cpuset cgroup in mounts
>> WARN[0001] mountpoint for pids not found
>> INFO[0001] Loading containers: start.
>> Error starting daemon: Error initializing network controller: error 
>> obtaining controller instance: failed to create NAT chain: iptables 
>> failed: iptables --wait -t nat -N DOCKER: iptables v1.6.1: can't 
>> initialize iptables table `nat': Table does not exist (do you need to 
>> insmod?)
>> Perhaps iptables or your kernel needs to be upgraded.
>>  (exit status 3)
>>
>> Best Regards,
>> Jakob
>>
>> On 05.06.2018 18:11, Jakob Hasse wrote:
>>> On 04.06.2018 17:53, Bruce Ashfield wrote:
>>>> On 2018-06-01 12:15 PM, Jakob Hasse wrote:
>>>>> Hello Bruce,
>>>>>
>>>>> Thank you very much for the quick response. I tried to built in the 
>>>>> kernel changes. But the iptables error persists.
>>>>
>>>> I double checked over the weekend, and I have no problems with
>>>> linux-yocto + the meta-virtualization fragment and docker.
>>> Thanks a lot for the work!
>>>>
>>>> Did you say that you confirmed on target via /proc/config.gz that
>>>> all the options you tried to enable are still present in the running
>>>> kernel ?
>>> I double checked with /proc/config.gz and all the modules are 
>>> builting except for CONFIG_NF_TABLES, CONFIG_NF_TABLES_IPV4. The last 
>>> two might not even exists, I was so desperate that I put everything 
>>> into the configuration which I could find related to NAT.
>>> Some modules are builtin as modules instead (...=m instead of ...=y), 
>>> however.
>>>>
>>>>> Eventually, I tried to enable systemd again and it still breaks my 
>>>>> build -.-:
>>>>>
>>>>> test$ bitbake core-image-base
>>>>> NOTE: Started PRServer with DBfile: 
>>>>> /home/build/test/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 40169, 
>>>>> PID: 2335
>>>>> Loading cache: 100% 
>>>>> |########################################################################################################| 
>>>>> Time: 0:00:00
>>>>> Loaded 3023 entries from dependency cache.
>>>>> Parsing recipes: 100% 
>>>>> |######################################################################################################| 
>>>>> Time: 0:00:01
>>>>> Parsing of 2194 .bb files complete (2193 cached, 1 parsed). 3024 
>>>>> targets, 146 skipped, 0 masked, 0 errors.
>>>>> NOTE: Resolving any missing task queue dependencies
>>>>>
>>>>> Build Configuration:
>>>>> BB_VERSION           = "1.36.0"
>>>>> BUILD_SYS            = "x86_64-linux"
>>>>> NATIVELSBSTRING      = "universal"
>>>>> TARGET_SYS           = "arm-dey-linux-gnueabi"
>>>>> MACHINE              = "ccimx6ulstarter"
>>>>> DISTRO               = "dey"
>>>>> DISTRO_VERSION       = "2.4-r1"
>>>>> TUNE_FEATURES        = "arm armv7ve vfp thumb neon 
>>>>> callconvention-hard cortexa7"
>>>>> TARGET_FPU           = "hard"
>>>>> meta
>>>>> meta-poky
>>>>> meta-yocto-bsp       = "HEAD:3befe6d7b7fa8c8481519aa8dd0cae52207ad339"
>>>>> meta-oe
>>>>> meta-python
>>>>> meta-networking
>>>>> meta-webserver       = "HEAD:dacfa2b1920e285531bec55cd2f08743390aaf57"
>>>>> meta-qt5             = "HEAD:cfe02f26de53e5c20e6f9555059cbaaf5ab9b22f"
>>>>> meta-swupdate        = "HEAD:6e4eab4f475b0129d6510815a3bbc4748c97dbbe"
>>>>> meta-freescale       = "HEAD:d6141ea291a1ac9ab8fb1dd1110d408f840fda57"
>>>>> meta-fsl-demos       = "HEAD:0ec6d7e206705702b5b534611754de0787f92b72"
>>>>> meta-digi-arm
>>>>> meta-digi-dey        = "HEAD:1246ecff2cecea9247d94f36385608ac844d7abb"
>>>>>
>>>>> Initialising tasks: 100% 
>>>>> |###################################################################################################| 
>>>>> Time: 0:00:04
>>>>> NOTE: Executing SetScene Tasks
>>>>> NOTE: Executing RunQueue Tasks
>>>>> ERROR: core-image-base-1.0-r0 do_rootfs: Could not invoke dnf. 
>>>>> Command 
>>>>> '/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/recipe-sysroot-native/usr/bin/dnf 
>>>>> -y -c 
>>>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs/etc/dnf/dnf.conf 
>>>>> --setopt=reposdir=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs/etc/yum.repos.d 
>>>>> --repofrompath=oe-repo,/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo 
>>>>> --installroot=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs 
>>>>> --setopt=logdir=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/temp 
>>>>> -x udev-cache --nogpgcheck install locale-base-en-us 
>>>>> locale-base-en-gb packagegroup-core-boot 
>>>>> packagegroup-core-eclipse-debug run-postinsts 
>>>>> packagegroup-dey-bluetooth psplash packagegroup-core-ssh-dropbear 
>>>>> packagegroup-dey-network packagegroup-base-extended 
>>>>> packagegroup-dey-audio packagegroup-dey-wireless rpm dnf' returned 1:
>>>>> Added oe-repo repo from 
>>>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo 
>>>>>
>>>>> Last metadata expiration check: 0:00:00 ago on Fri 01 Jun 2018 
>>>>> 03:49:44 PM UTC.
>>>>> Error:
>>>>>   Problem: package packagegroup-base-1.0-r83.0.ccimx6ulstarter 
>>>>> requires packagegroup-distro-base, but none of the providers can be 
>>>>> installed
>>>>>    - package packagegroup-base-extended-1.0-r83.0.ccimx6ulstarter 
>>>>> requires packagegroup-base, but none of the providers can be installed
>>>>>    - package packagegroup-distro-base-1.0-r83.0.ccimx6ulstarter 
>>>>> requires packagegroup-dey-core, but none of the providers can be 
>>>>> installed
>>>>>    - conflicting requests
>>>>>    - nothing provides busybox-hwclock needed by 
>>>>> packagegroup-dey-core-1.0-r0.0.ccimx6ulstarter
>>>>>
>>>>> ERROR: core-image-base-1.0-r0 do_rootfs: Function failed: do_rootfs
>>>>> ERROR: Logfile of failure stored in: 
>>>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.2380 
>>>>>
>>>>> ERROR: Task 
>>>>> (/usr/local/dey-2.4/sources/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs) 
>>>>> failed with exit code '1'
>>>>> NOTE: Tasks Summary: Attempted 4106 tasks of which 4105 didn't need 
>>>>> to be rerun and 1 failed.
>>>>>
>>>>> Summary: 1 task failed:
>>>>> /usr/local/dey-2.4/sources/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs 
>>>>>
>>>>> Summary: There were 2 ERROR messages shown, returning a non-zero 
>>>>> exit code.
>>>>>
>>>>> Seems that the introduction of systemd "confuses" dnf. I found a 
>>>>> quite similar description in this bugreport here:
>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12025
>>>>> only that I'm not using different machines. This bug doesn't seem 
>>>>> to be resolved yet.
>>>>>
>>>>> Our Yocto-distribution is DIGI Embedded Yocto. So non-standard.
>>>>> Shall I still log a bug against oe-core?
>>>>
>>>> You could, but without a reproducer on oe-core, chances are that it
>>>> won't be easily resolved.
>>>>
>>>> At a glance, it isn't clear to me how systemd is causing your 
>>>> busybox-hwclock
>>>> issue. But if you check the list, that particular package has recently
>>>> been moved around/broken out. So I'd suggest making sure that all of
>>>> your layers are on the correct branches and that releases aren't being
>>>> mixed.
>>> Our hardware supplier told me that systemd is broken in their 
>>> distribution and recommended to try it without systemd. That's what 
>>> I'm doing right now.
>>>
>>> Thank you again for all the information already. All the Best,
>>> Jakob
>>>>
>>>> Bruce
>>>>
>>>>> If someone here has some more helpful ideas, I would be very thankful.
>>>>>
>>>>> All the Best,
>>>>> Jakob
>>>>>
>>>>> On 31.05.2018 13:44, Bruce Ashfield wrote:
>>>>>> On 2018-05-31 7:00 AM, Jakob Hasse wrote:
>>>>>>> Hello,
>>>>>>
>>>>>> Make sure to cc meta-virtualization on questions like this, since
>>>>>> that is where you'll get more eyes that are running docker
>>>>>> all the time.
>>>>>>
>>>>>>> I ran into trouble running docker on our target.
>>>>>>> 1. When I want to start docker, I first have to re-mount cgroups:
>>>>>>> root at target:~# cgroups-umount
>>>>>>> root at target:~# cgroups-mount
>>>>>>> Otherwise docker would produce an error:
>>>>>>> ERRO[0002] Failed to built-in GetDriver graph btrfs /var/lib/docker
>>>>>>>
>>>>>>> 2. When I then start dockerd, it complains about a missing nat 
>>>>>>> table:
>>>>>>> root at target:~# dockerd
>>>>>>> INFO[0000] libcontainerd: new containerd process, pid: 929
>>>>>>> WARN[0000] containerd: low RLIMIT_NOFILE changing to max 
>>>>>>> current=1024 max=4096
>>>>>>> INFO[0001] [graphdriver] using prior storage driver: overlay2
>>>>>>> INFO[0001] Graph migration to content-addressability took 0.00 
>>>>>>> seconds
>>>>>>> WARN[0001] Your kernel does not support cgroup memory limit
>>>>>>> WARN[0001] Unable to find cpu cgroup in mounts
>>>>>>> WARN[0001] Unable to find blkio cgroup in mounts
>>>>>>> WARN[0001] Unable to find cpuset cgroup in mounts
>>>>>>> WARN[0001] mountpoint for pids not found
>>>>>>> INFO[0001] Loading containers: start.
>>>>>>> WARN[0001] Running modprobe nf_nat failed with message: 
>>>>>>> `modprobe: WARNING: Module nf_nat not found in directory 
>>>>>>> /lib/modules/4.9.81-dey+g2c6ae4c`, error: exit status 1
>>>>>>> WARN[0001] Running modprobe xt_conntrack failed with message: 
>>>>>>> `modprobe: WARNING: Module xt_conntrack not found in directory 
>>>>>>> /lib/modules/4.9.81-dey+g2c6ae4c`, error: exit status 1
>>>>>>> Error starting daemon: Error initializing network controller: 
>>>>>>> error obtaining controller instance: failed to create NAT chain: 
>>>>>>> iptables failed: iptables --wait -t nat -N DOCKER: iptables 
>>>>>>> v1.6.1: can't initialize iptables table `nat': Table does not 
>>>>>>> exist (do you need to insmod?)
>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>   (exit status 3)
>>>>>>>
>>>>>>> Our configuration is as suggested here: 
>>>>>>> https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage, 
>>>>>>> except 
>>>>>>
>>>>>> I've never seen that wiki page before (or at least I don't remember
>>>>>> seeing it), so I can't confirm or deny the validity of the content :)
>>>>>>
>>>>>>> that I don't include the system systemd stuff  (it lets my build 
>>>>>>> fail) 
>>>>>>
>>>>>> If systemd is breaking your build, make sure to log a bugzilla 
>>>>>> against
>>>>>> oe-core
>>>>>>
>>>>>>> and connman (using NetworkManager).
>>>>>>> Furthermore, I added the following lines to the kernel bbappend 
>>>>>>> file:
>>>>>>>
>>>>>>> # remove old defconfig
>>>>>>> SRC_URI_remove = " defconfig"
>>>>>>> # replace with new defconfig
>>>>>>> SRC_URI_append = " file://defconfig"
>>>>>>>
>>>>>>> KERNEL_FEATURES_append = " features/cgroups/cgroups.scc "
>>>>>>>
>>>>>>> I also added a lot of configurations manually to the defconfig 
>>>>>>> (mostly via menuconfig) to enable NAT:
>>>>>>>
>>>>>>> CONFIG_CGROUP_DEVICE=y
>>>>>>> CONFIG_IP_MULTICAST=y
>>>>>>> CONFIG_IP_ADVANCED_ROUTER=y
>>>>>>> CONFIG_NETFILTER=y
>>>>>>> CONFIG_NF_CONNTRACK=y
>>>>>>> CONFIG_NF_TABLES=y
>>>>>>> CONFIG_NF_NAT=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_COMMENT=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_HL=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_LIMIT=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
>>>>>>> CONFIG_NETFILTER_XT_MATCH_RECENT=y
>>>>>>> CONFIG_IP_VS=y
>>>>>>> CONFIG_NF_TABLES_IPV4=y
>>>>>>> CONFIG_IP_NF_IPTABLES=y
>>>>>>> CONFIG_IP_NF_NAT=y
>>>>>>> CONFIG_IP_NF_FILTER=y
>>>>>>> CONFIG_IP_NF_MANGLE=y
>>>>>>> CONFIG_IP6_NF_IPTABLES=y
>>>>>>> CONFIG_IP6_NF_FILTER=y
>>>>>>> CONFIG_IP6_NF_MANGLE=y
>>>>>>> CONFIG_BTRFS_FS=y
>>>>>>> CONFIG_OVERLAY_FS=y
>>>>>>>
>>>>>>> Apart from that, I added virtualization and aufs as 
>>>>>>> DISTRO_FEATURE in local.conf and also enabled it in menuconfig.
>>>>>>>
>>>>>>> But I still keep getting the above mentioned iptables error when 
>>>>>>> trying to start docker. All this hassle makes me suspicious, 
>>>>>>> especially as I'm quite sure that I once had docker running 
>>>>>>> already with an image on our target and it wasn't that hard. So 
>>>>>>> maybe it's just a misconfiguration and I need to add something in 
>>>>>>> local.conf or the kernel recipe? Is systemd necessary? Or am I 
>>>>>>> missing some life-or-death kernel configuration? It would also be 
>>>>>>> nice if I could avoid the cgroup re-mounting before starting docker.
>>>>>>
>>>>>> What release branch are you using ?
>>>>>>
>>>>>> I'm running docker from meta-virt every day, as are many others,
>>>>>> but you have several differences in your configuration.
>>>>>>
>>>>>>  - most use systemd as the init manager, I know that I do. That
>>>>>>    is going to impact how cgroups is set up on your 'host' image.
>>>>>>    You shouldn't need to touch cgroups at all if systemd is used,
>>>>>>    since it is correct out of the box.
>>>>>>
>>>>>>  - You are using a different kernel and kernel configuration.
>>>>>>    linux-yocto + the configuration fragments in the layer are what
>>>>>>    is routinely tested. Are you using linux-yocto, or something
>>>>>>    different ? If it is different, all you can do is run the various
>>>>>>    checks to make sure that the docker prereqs are in place.
>>>>>>
>>>>>>    The errors you see in dockerd tells me that the options you are
>>>>>>    turning on, are not making it into the final kernel that is
>>>>>>    running on target.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Bruce
>>>>>>
>>>>>>>
>>>>>>> Thanks for every answer!
>>>>>>> All the Best,
>>>>>>> Jakob
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

More information about the meta-virtualization mailing list