[meta-virtualization] [Resolved] Re: cgroups and iptables problems running docker - maybe my config wrong?
Jakob Hasse
jakob.hasse at qiio.com
Wed Jun 6 07:33:10 PDT 2018
Hello Bruce, hello Yoctoers,
my problem is solved. It was the kernel options. I added all additional
kernel options which this check-script here suggests:
https://github.com/moby/moby/blob/master/contrib/check-config.sh. This
script was a big help
Finally, docker runs, with one ERROR message and one WARNING though, but
it does :) If I have more problems, I'll come back to the list.
Thank you for the help so far.
All the Best,
Jakob
On 05.06.2018 18:23, Jakob Hasse wrote:
> Hello,
>
> right now, there is no insmod warning anymore, only iptables does
> still not work with the nat table:
>
> root at ccimx6uluvalue:~# dockerd
> INFO[0000] libcontainerd: new containerd process, pid: 1003
> WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024
> max=4096
> INFO[0001] [graphdriver] using prior storage driver: overlay2
> INFO[0001] Graph migration to content-addressability took 0.00 seconds
> WARN[0001] Your kernel does not support cgroup memory limit
> WARN[0001] Unable to find cpu cgroup in mounts
> WARN[0001] Unable to find blkio cgroup in mounts
> WARN[0001] Unable to find cpuset cgroup in mounts
> WARN[0001] mountpoint for pids not found
> INFO[0001] Loading containers: start.
> Error starting daemon: Error initializing network controller: error
> obtaining controller instance: failed to create NAT chain: iptables
> failed: iptables --wait -t nat -N DOCKER: iptables v1.6.1: can't
> initialize iptables table `nat': Table does not exist (do you need to
> insmod?)
> Perhaps iptables or your kernel needs to be upgraded.
> (exit status 3)
>
> Best Regards,
> Jakob
>
> On 05.06.2018 18:11, Jakob Hasse wrote:
>> On 04.06.2018 17:53, Bruce Ashfield wrote:
>>> On 2018-06-01 12:15 PM, Jakob Hasse wrote:
>>>> Hello Bruce,
>>>>
>>>> Thank you very much for the quick response. I tried to built in the
>>>> kernel changes. But the iptables error persists.
>>>
>>> I double checked over the weekend, and I have no problems with
>>> linux-yocto + the meta-virtualization fragment and docker.
>> Thanks a lot for the work!
>>>
>>> Did you say that you confirmed on target via /proc/config.gz that
>>> all the options you tried to enable are still present in the running
>>> kernel ?
>> I double checked with /proc/config.gz and all the modules are
>> builting except for CONFIG_NF_TABLES, CONFIG_NF_TABLES_IPV4. The last
>> two might not even exists, I was so desperate that I put everything
>> into the configuration which I could find related to NAT.
>> Some modules are builtin as modules instead (...=m instead of ...=y),
>> however.
>>>
>>>> Eventually, I tried to enable systemd again and it still breaks my
>>>> build -.-:
>>>>
>>>> test$ bitbake core-image-base
>>>> NOTE: Started PRServer with DBfile:
>>>> /home/build/test/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 40169,
>>>> PID: 2335
>>>> Loading cache: 100%
>>>> |########################################################################################################|
>>>> Time: 0:00:00
>>>> Loaded 3023 entries from dependency cache.
>>>> Parsing recipes: 100%
>>>> |######################################################################################################|
>>>> Time: 0:00:01
>>>> Parsing of 2194 .bb files complete (2193 cached, 1 parsed). 3024
>>>> targets, 146 skipped, 0 masked, 0 errors.
>>>> NOTE: Resolving any missing task queue dependencies
>>>>
>>>> Build Configuration:
>>>> BB_VERSION = "1.36.0"
>>>> BUILD_SYS = "x86_64-linux"
>>>> NATIVELSBSTRING = "universal"
>>>> TARGET_SYS = "arm-dey-linux-gnueabi"
>>>> MACHINE = "ccimx6ulstarter"
>>>> DISTRO = "dey"
>>>> DISTRO_VERSION = "2.4-r1"
>>>> TUNE_FEATURES = "arm armv7ve vfp thumb neon
>>>> callconvention-hard cortexa7"
>>>> TARGET_FPU = "hard"
>>>> meta
>>>> meta-poky
>>>> meta-yocto-bsp = "HEAD:3befe6d7b7fa8c8481519aa8dd0cae52207ad339"
>>>> meta-oe
>>>> meta-python
>>>> meta-networking
>>>> meta-webserver = "HEAD:dacfa2b1920e285531bec55cd2f08743390aaf57"
>>>> meta-qt5 = "HEAD:cfe02f26de53e5c20e6f9555059cbaaf5ab9b22f"
>>>> meta-swupdate = "HEAD:6e4eab4f475b0129d6510815a3bbc4748c97dbbe"
>>>> meta-freescale = "HEAD:d6141ea291a1ac9ab8fb1dd1110d408f840fda57"
>>>> meta-fsl-demos = "HEAD:0ec6d7e206705702b5b534611754de0787f92b72"
>>>> meta-digi-arm
>>>> meta-digi-dey = "HEAD:1246ecff2cecea9247d94f36385608ac844d7abb"
>>>>
>>>> Initialising tasks: 100%
>>>> |###################################################################################################|
>>>> Time: 0:00:04
>>>> NOTE: Executing SetScene Tasks
>>>> NOTE: Executing RunQueue Tasks
>>>> ERROR: core-image-base-1.0-r0 do_rootfs: Could not invoke dnf.
>>>> Command
>>>> '/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/recipe-sysroot-native/usr/bin/dnf
>>>> -y -c
>>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs/etc/dnf/dnf.conf
>>>> --setopt=reposdir=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs/etc/yum.repos.d
>>>> --repofrompath=oe-repo,/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo
>>>> --installroot=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs
>>>> --setopt=logdir=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/temp
>>>> -x udev-cache --nogpgcheck install locale-base-en-us
>>>> locale-base-en-gb packagegroup-core-boot
>>>> packagegroup-core-eclipse-debug run-postinsts
>>>> packagegroup-dey-bluetooth psplash packagegroup-core-ssh-dropbear
>>>> packagegroup-dey-network packagegroup-base-extended
>>>> packagegroup-dey-audio packagegroup-dey-wireless rpm dnf' returned 1:
>>>> Added oe-repo repo from
>>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo
>>>>
>>>> Last metadata expiration check: 0:00:00 ago on Fri 01 Jun 2018
>>>> 03:49:44 PM UTC.
>>>> Error:
>>>> Problem: package packagegroup-base-1.0-r83.0.ccimx6ulstarter
>>>> requires packagegroup-distro-base, but none of the providers can be
>>>> installed
>>>> - package packagegroup-base-extended-1.0-r83.0.ccimx6ulstarter
>>>> requires packagegroup-base, but none of the providers can be installed
>>>> - package packagegroup-distro-base-1.0-r83.0.ccimx6ulstarter
>>>> requires packagegroup-dey-core, but none of the providers can be
>>>> installed
>>>> - conflicting requests
>>>> - nothing provides busybox-hwclock needed by
>>>> packagegroup-dey-core-1.0-r0.0.ccimx6ulstarter
>>>>
>>>> ERROR: core-image-base-1.0-r0 do_rootfs: Function failed: do_rootfs
>>>> ERROR: Logfile of failure stored in:
>>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.2380
>>>>
>>>> ERROR: Task
>>>> (/usr/local/dey-2.4/sources/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs)
>>>> failed with exit code '1'
>>>> NOTE: Tasks Summary: Attempted 4106 tasks of which 4105 didn't need
>>>> to be rerun and 1 failed.
>>>>
>>>> Summary: 1 task failed:
>>>> /usr/local/dey-2.4/sources/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs
>>>>
>>>> Summary: There were 2 ERROR messages shown, returning a non-zero
>>>> exit code.
>>>>
>>>> Seems that the introduction of systemd "confuses" dnf. I found a
>>>> quite similar description in this bugreport here:
>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12025
>>>> only that I'm not using different machines. This bug doesn't seem
>>>> to be resolved yet.
>>>>
>>>> Our Yocto-distribution is DIGI Embedded Yocto. So non-standard.
>>>> Shall I still log a bug against oe-core?
>>>
>>> You could, but without a reproducer on oe-core, chances are that it
>>> won't be easily resolved.
>>>
>>> At a glance, it isn't clear to me how systemd is causing your
>>> busybox-hwclock
>>> issue. But if you check the list, that particular package has recently
>>> been moved around/broken out. So I'd suggest making sure that all of
>>> your layers are on the correct branches and that releases aren't being
>>> mixed.
>> Our hardware supplier told me that systemd is broken in their
>> distribution and recommended to try it without systemd. That's what
>> I'm doing right now.
>>
>> Thank you again for all the information already. All the Best,
>> Jakob
>>>
>>> Bruce
>>>
>>>> If someone here has some more helpful ideas, I would be very thankful.
>>>>
>>>> All the Best,
>>>> Jakob
>>>>
>>>> On 31.05.2018 13:44, Bruce Ashfield wrote:
>>>>> On 2018-05-31 7:00 AM, Jakob Hasse wrote:
>>>>>> Hello,
>>>>>
>>>>> Make sure to cc meta-virtualization on questions like this, since
>>>>> that is where you'll get more eyes that are running docker
>>>>> all the time.
>>>>>
>>>>>> I ran into trouble running docker on our target.
>>>>>> 1. When I want to start docker, I first have to re-mount cgroups:
>>>>>> root at target:~# cgroups-umount
>>>>>> root at target:~# cgroups-mount
>>>>>> Otherwise docker would produce an error:
>>>>>> ERRO[0002] Failed to built-in GetDriver graph btrfs /var/lib/docker
>>>>>>
>>>>>> 2. When I then start dockerd, it complains about a missing nat
>>>>>> table:
>>>>>> root at target:~# dockerd
>>>>>> INFO[0000] libcontainerd: new containerd process, pid: 929
>>>>>> WARN[0000] containerd: low RLIMIT_NOFILE changing to max
>>>>>> current=1024 max=4096
>>>>>> INFO[0001] [graphdriver] using prior storage driver: overlay2
>>>>>> INFO[0001] Graph migration to content-addressability took 0.00
>>>>>> seconds
>>>>>> WARN[0001] Your kernel does not support cgroup memory limit
>>>>>> WARN[0001] Unable to find cpu cgroup in mounts
>>>>>> WARN[0001] Unable to find blkio cgroup in mounts
>>>>>> WARN[0001] Unable to find cpuset cgroup in mounts
>>>>>> WARN[0001] mountpoint for pids not found
>>>>>> INFO[0001] Loading containers: start.
>>>>>> WARN[0001] Running modprobe nf_nat failed with message:
>>>>>> `modprobe: WARNING: Module nf_nat not found in directory
>>>>>> /lib/modules/4.9.81-dey+g2c6ae4c`, error: exit status 1
>>>>>> WARN[0001] Running modprobe xt_conntrack failed with message:
>>>>>> `modprobe: WARNING: Module xt_conntrack not found in directory
>>>>>> /lib/modules/4.9.81-dey+g2c6ae4c`, error: exit status 1
>>>>>> Error starting daemon: Error initializing network controller:
>>>>>> error obtaining controller instance: failed to create NAT chain:
>>>>>> iptables failed: iptables --wait -t nat -N DOCKER: iptables
>>>>>> v1.6.1: can't initialize iptables table `nat': Table does not
>>>>>> exist (do you need to insmod?)
>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>> (exit status 3)
>>>>>>
>>>>>> Our configuration is as suggested here:
>>>>>> https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage,
>>>>>> except
>>>>>
>>>>> I've never seen that wiki page before (or at least I don't remember
>>>>> seeing it), so I can't confirm or deny the validity of the content :)
>>>>>
>>>>>> that I don't include the system systemd stuff (it lets my build
>>>>>> fail)
>>>>>
>>>>> If systemd is breaking your build, make sure to log a bugzilla
>>>>> against
>>>>> oe-core
>>>>>
>>>>>> and connman (using NetworkManager).
>>>>>> Furthermore, I added the following lines to the kernel bbappend
>>>>>> file:
>>>>>>
>>>>>> # remove old defconfig
>>>>>> SRC_URI_remove = " defconfig"
>>>>>> # replace with new defconfig
>>>>>> SRC_URI_append = " file://defconfig"
>>>>>>
>>>>>> KERNEL_FEATURES_append = " features/cgroups/cgroups.scc "
>>>>>>
>>>>>> I also added a lot of configurations manually to the defconfig
>>>>>> (mostly via menuconfig) to enable NAT:
>>>>>>
>>>>>> CONFIG_CGROUP_DEVICE=y
>>>>>> CONFIG_IP_MULTICAST=y
>>>>>> CONFIG_IP_ADVANCED_ROUTER=y
>>>>>> CONFIG_NETFILTER=y
>>>>>> CONFIG_NF_CONNTRACK=y
>>>>>> CONFIG_NF_TABLES=y
>>>>>> CONFIG_NF_NAT=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_COMMENT=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_HL=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_LIMIT=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
>>>>>> CONFIG_NETFILTER_XT_MATCH_RECENT=y
>>>>>> CONFIG_IP_VS=y
>>>>>> CONFIG_NF_TABLES_IPV4=y
>>>>>> CONFIG_IP_NF_IPTABLES=y
>>>>>> CONFIG_IP_NF_NAT=y
>>>>>> CONFIG_IP_NF_FILTER=y
>>>>>> CONFIG_IP_NF_MANGLE=y
>>>>>> CONFIG_IP6_NF_IPTABLES=y
>>>>>> CONFIG_IP6_NF_FILTER=y
>>>>>> CONFIG_IP6_NF_MANGLE=y
>>>>>> CONFIG_BTRFS_FS=y
>>>>>> CONFIG_OVERLAY_FS=y
>>>>>>
>>>>>> Apart from that, I added virtualization and aufs as
>>>>>> DISTRO_FEATURE in local.conf and also enabled it in menuconfig.
>>>>>>
>>>>>> But I still keep getting the above mentioned iptables error when
>>>>>> trying to start docker. All this hassle makes me suspicious,
>>>>>> especially as I'm quite sure that I once had docker running
>>>>>> already with an image on our target and it wasn't that hard. So
>>>>>> maybe it's just a misconfiguration and I need to add something in
>>>>>> local.conf or the kernel recipe? Is systemd necessary? Or am I
>>>>>> missing some life-or-death kernel configuration? It would also be
>>>>>> nice if I could avoid the cgroup re-mounting before starting docker.
>>>>>
>>>>> What release branch are you using ?
>>>>>
>>>>> I'm running docker from meta-virt every day, as are many others,
>>>>> but you have several differences in your configuration.
>>>>>
>>>>> - most use systemd as the init manager, I know that I do. That
>>>>> is going to impact how cgroups is set up on your 'host' image.
>>>>> You shouldn't need to touch cgroups at all if systemd is used,
>>>>> since it is correct out of the box.
>>>>>
>>>>> - You are using a different kernel and kernel configuration.
>>>>> linux-yocto + the configuration fragments in the layer are what
>>>>> is routinely tested. Are you using linux-yocto, or something
>>>>> different ? If it is different, all you can do is run the various
>>>>> checks to make sure that the docker prereqs are in place.
>>>>>
>>>>> The errors you see in dockerd tells me that the options you are
>>>>> turning on, are not making it into the final kernel that is
>>>>> running on target.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bruce
>>>>>
>>>>>>
>>>>>> Thanks for every answer!
>>>>>> All the Best,
>>>>>> Jakob
>>>>>>
>>>>>
>>>>
>>>
>>
More information about the meta-virtualization
mailing list