[meta-virtualization] cgroups and iptables problems running docker - maybe my config wrong?
Jakob Hasse
jakob.hasse at qiio.com
Tue Jun 5 09:23:13 PDT 2018
Hello,
right now, there is no insmod warning anymore, only iptables does still
not work with the nat table:
root at ccimx6uluvalue:~# dockerd
INFO[0000] libcontainerd: new containerd process, pid: 1003
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024
max=4096
INFO[0001] [graphdriver] using prior storage driver: overlay2
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support cgroup memory limit
WARN[0001] Unable to find cpu cgroup in mounts
WARN[0001] Unable to find blkio cgroup in mounts
WARN[0001] Unable to find cpuset cgroup in mounts
WARN[0001] mountpoint for pids not found
INFO[0001] Loading containers: start.
Error starting daemon: Error initializing network controller: error
obtaining controller instance: failed to create NAT chain: iptables
failed: iptables --wait -t nat -N DOCKER: iptables v1.6.1: can't
initialize iptables table `nat': Table does not exist (do you need to
insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
Best Regards,
Jakob
On 05.06.2018 18:11, Jakob Hasse wrote:
> On 04.06.2018 17:53, Bruce Ashfield wrote:
>> On 2018-06-01 12:15 PM, Jakob Hasse wrote:
>>> Hello Bruce,
>>>
>>> Thank you very much for the quick response. I tried to built in the
>>> kernel changes. But the iptables error persists.
>>
>> I double checked over the weekend, and I have no problems with
>> linux-yocto + the meta-virtualization fragment and docker.
> Thanks a lot for the work!
>>
>> Did you say that you confirmed on target via /proc/config.gz that
>> all the options you tried to enable are still present in the running
>> kernel ?
> I double checked with /proc/config.gz and all the modules are builting
> except for CONFIG_NF_TABLES, CONFIG_NF_TABLES_IPV4. The last two might
> not even exists, I was so desperate that I put everything into the
> configuration which I could find related to NAT.
> Some modules are builtin as modules instead (...=m instead of ...=y),
> however.
>>
>>> Eventually, I tried to enable systemd again and it still breaks my
>>> build -.-:
>>>
>>> test$ bitbake core-image-base
>>> NOTE: Started PRServer with DBfile:
>>> /home/build/test/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 40169,
>>> PID: 2335
>>> Loading cache: 100%
>>> |########################################################################################################|
>>> Time: 0:00:00
>>> Loaded 3023 entries from dependency cache.
>>> Parsing recipes: 100%
>>> |######################################################################################################|
>>> Time: 0:00:01
>>> Parsing of 2194 .bb files complete (2193 cached, 1 parsed). 3024
>>> targets, 146 skipped, 0 masked, 0 errors.
>>> NOTE: Resolving any missing task queue dependencies
>>>
>>> Build Configuration:
>>> BB_VERSION = "1.36.0"
>>> BUILD_SYS = "x86_64-linux"
>>> NATIVELSBSTRING = "universal"
>>> TARGET_SYS = "arm-dey-linux-gnueabi"
>>> MACHINE = "ccimx6ulstarter"
>>> DISTRO = "dey"
>>> DISTRO_VERSION = "2.4-r1"
>>> TUNE_FEATURES = "arm armv7ve vfp thumb neon
>>> callconvention-hard cortexa7"
>>> TARGET_FPU = "hard"
>>> meta
>>> meta-poky
>>> meta-yocto-bsp = "HEAD:3befe6d7b7fa8c8481519aa8dd0cae52207ad339"
>>> meta-oe
>>> meta-python
>>> meta-networking
>>> meta-webserver = "HEAD:dacfa2b1920e285531bec55cd2f08743390aaf57"
>>> meta-qt5 = "HEAD:cfe02f26de53e5c20e6f9555059cbaaf5ab9b22f"
>>> meta-swupdate = "HEAD:6e4eab4f475b0129d6510815a3bbc4748c97dbbe"
>>> meta-freescale = "HEAD:d6141ea291a1ac9ab8fb1dd1110d408f840fda57"
>>> meta-fsl-demos = "HEAD:0ec6d7e206705702b5b534611754de0787f92b72"
>>> meta-digi-arm
>>> meta-digi-dey = "HEAD:1246ecff2cecea9247d94f36385608ac844d7abb"
>>>
>>> Initialising tasks: 100%
>>> |###################################################################################################|
>>> Time: 0:00:04
>>> NOTE: Executing SetScene Tasks
>>> NOTE: Executing RunQueue Tasks
>>> ERROR: core-image-base-1.0-r0 do_rootfs: Could not invoke dnf.
>>> Command
>>> '/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/recipe-sysroot-native/usr/bin/dnf
>>> -y -c
>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs/etc/dnf/dnf.conf
>>> --setopt=reposdir=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs/etc/yum.repos.d
>>> --repofrompath=oe-repo,/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo
>>> --installroot=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/rootfs
>>> --setopt=logdir=/home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/temp
>>> -x udev-cache --nogpgcheck install locale-base-en-us
>>> locale-base-en-gb packagegroup-core-boot
>>> packagegroup-core-eclipse-debug run-postinsts
>>> packagegroup-dey-bluetooth psplash packagegroup-core-ssh-dropbear
>>> packagegroup-dey-network packagegroup-base-extended
>>> packagegroup-dey-audio packagegroup-dey-wireless rpm dnf' returned 1:
>>> Added oe-repo repo from
>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo
>>>
>>> Last metadata expiration check: 0:00:00 ago on Fri 01 Jun 2018
>>> 03:49:44 PM UTC.
>>> Error:
>>> Problem: package packagegroup-base-1.0-r83.0.ccimx6ulstarter
>>> requires packagegroup-distro-base, but none of the providers can be
>>> installed
>>> - package packagegroup-base-extended-1.0-r83.0.ccimx6ulstarter
>>> requires packagegroup-base, but none of the providers can be installed
>>> - package packagegroup-distro-base-1.0-r83.0.ccimx6ulstarter
>>> requires packagegroup-dey-core, but none of the providers can be
>>> installed
>>> - conflicting requests
>>> - nothing provides busybox-hwclock needed by
>>> packagegroup-dey-core-1.0-r0.0.ccimx6ulstarter
>>>
>>> ERROR: core-image-base-1.0-r0 do_rootfs: Function failed: do_rootfs
>>> ERROR: Logfile of failure stored in:
>>> /home/build/test/tmp/work/ccimx6ulstarter-dey-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.2380
>>>
>>> ERROR: Task
>>> (/usr/local/dey-2.4/sources/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs)
>>> failed with exit code '1'
>>> NOTE: Tasks Summary: Attempted 4106 tasks of which 4105 didn't need
>>> to be rerun and 1 failed.
>>>
>>> Summary: 1 task failed:
>>> /usr/local/dey-2.4/sources/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs
>>>
>>> Summary: There were 2 ERROR messages shown, returning a non-zero
>>> exit code.
>>>
>>> Seems that the introduction of systemd "confuses" dnf. I found a
>>> quite similar description in this bugreport here:
>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12025
>>> only that I'm not using different machines. This bug doesn't seem to
>>> be resolved yet.
>>>
>>> Our Yocto-distribution is DIGI Embedded Yocto. So non-standard.
>>> Shall I still log a bug against oe-core?
>>
>> You could, but without a reproducer on oe-core, chances are that it
>> won't be easily resolved.
>>
>> At a glance, it isn't clear to me how systemd is causing your
>> busybox-hwclock
>> issue. But if you check the list, that particular package has recently
>> been moved around/broken out. So I'd suggest making sure that all of
>> your layers are on the correct branches and that releases aren't being
>> mixed.
> Our hardware supplier told me that systemd is broken in their
> distribution and recommended to try it without systemd. That's what
> I'm doing right now.
>
> Thank you again for all the information already. All the Best,
> Jakob
>>
>> Bruce
>>
>>> If someone here has some more helpful ideas, I would be very thankful.
>>>
>>> All the Best,
>>> Jakob
>>>
>>> On 31.05.2018 13:44, Bruce Ashfield wrote:
>>>> On 2018-05-31 7:00 AM, Jakob Hasse wrote:
>>>>> Hello,
>>>>
>>>> Make sure to cc meta-virtualization on questions like this, since
>>>> that is where you'll get more eyes that are running docker
>>>> all the time.
>>>>
>>>>> I ran into trouble running docker on our target.
>>>>> 1. When I want to start docker, I first have to re-mount cgroups:
>>>>> root at target:~# cgroups-umount
>>>>> root at target:~# cgroups-mount
>>>>> Otherwise docker would produce an error:
>>>>> ERRO[0002] Failed to built-in GetDriver graph btrfs /var/lib/docker
>>>>>
>>>>> 2. When I then start dockerd, it complains about a missing nat table:
>>>>> root at target:~# dockerd
>>>>> INFO[0000] libcontainerd: new containerd process, pid: 929
>>>>> WARN[0000] containerd: low RLIMIT_NOFILE changing to max
>>>>> current=1024 max=4096
>>>>> INFO[0001] [graphdriver] using prior storage driver: overlay2
>>>>> INFO[0001] Graph migration to content-addressability took 0.00
>>>>> seconds
>>>>> WARN[0001] Your kernel does not support cgroup memory limit
>>>>> WARN[0001] Unable to find cpu cgroup in mounts
>>>>> WARN[0001] Unable to find blkio cgroup in mounts
>>>>> WARN[0001] Unable to find cpuset cgroup in mounts
>>>>> WARN[0001] mountpoint for pids not found
>>>>> INFO[0001] Loading containers: start.
>>>>> WARN[0001] Running modprobe nf_nat failed with message: `modprobe:
>>>>> WARNING: Module nf_nat not found in directory
>>>>> /lib/modules/4.9.81-dey+g2c6ae4c`, error: exit status 1
>>>>> WARN[0001] Running modprobe xt_conntrack failed with message:
>>>>> `modprobe: WARNING: Module xt_conntrack not found in directory
>>>>> /lib/modules/4.9.81-dey+g2c6ae4c`, error: exit status 1
>>>>> Error starting daemon: Error initializing network controller:
>>>>> error obtaining controller instance: failed to create NAT chain:
>>>>> iptables failed: iptables --wait -t nat -N DOCKER: iptables
>>>>> v1.6.1: can't initialize iptables table `nat': Table does not
>>>>> exist (do you need to insmod?)
>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>> (exit status 3)
>>>>>
>>>>> Our configuration is as suggested here:
>>>>> https://wiki.yoctoproject.org/wiki/TipsAndTricks/DockerOnImage,
>>>>> except
>>>>
>>>> I've never seen that wiki page before (or at least I don't remember
>>>> seeing it), so I can't confirm or deny the validity of the content :)
>>>>
>>>>> that I don't include the system systemd stuff (it lets my build
>>>>> fail)
>>>>
>>>> If systemd is breaking your build, make sure to log a bugzilla against
>>>> oe-core
>>>>
>>>>> and connman (using NetworkManager).
>>>>> Furthermore, I added the following lines to the kernel bbappend file:
>>>>>
>>>>> # remove old defconfig
>>>>> SRC_URI_remove = " defconfig"
>>>>> # replace with new defconfig
>>>>> SRC_URI_append = " file://defconfig"
>>>>>
>>>>> KERNEL_FEATURES_append = " features/cgroups/cgroups.scc "
>>>>>
>>>>> I also added a lot of configurations manually to the defconfig
>>>>> (mostly via menuconfig) to enable NAT:
>>>>>
>>>>> CONFIG_CGROUP_DEVICE=y
>>>>> CONFIG_IP_MULTICAST=y
>>>>> CONFIG_IP_ADVANCED_ROUTER=y
>>>>> CONFIG_NETFILTER=y
>>>>> CONFIG_NF_CONNTRACK=y
>>>>> CONFIG_NF_TABLES=y
>>>>> CONFIG_NF_NAT=y
>>>>> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
>>>>> CONFIG_NETFILTER_XT_MATCH_COMMENT=y
>>>>> CONFIG_NETFILTER_XT_MATCH_HL=y
>>>>> CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
>>>>> CONFIG_NETFILTER_XT_MATCH_LIMIT=y
>>>>> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
>>>>> CONFIG_NETFILTER_XT_MATCH_RECENT=y
>>>>> CONFIG_IP_VS=y
>>>>> CONFIG_NF_TABLES_IPV4=y
>>>>> CONFIG_IP_NF_IPTABLES=y
>>>>> CONFIG_IP_NF_NAT=y
>>>>> CONFIG_IP_NF_FILTER=y
>>>>> CONFIG_IP_NF_MANGLE=y
>>>>> CONFIG_IP6_NF_IPTABLES=y
>>>>> CONFIG_IP6_NF_FILTER=y
>>>>> CONFIG_IP6_NF_MANGLE=y
>>>>> CONFIG_BTRFS_FS=y
>>>>> CONFIG_OVERLAY_FS=y
>>>>>
>>>>> Apart from that, I added virtualization and aufs as DISTRO_FEATURE
>>>>> in local.conf and also enabled it in menuconfig.
>>>>>
>>>>> But I still keep getting the above mentioned iptables error when
>>>>> trying to start docker. All this hassle makes me suspicious,
>>>>> especially as I'm quite sure that I once had docker running
>>>>> already with an image on our target and it wasn't that hard. So
>>>>> maybe it's just a misconfiguration and I need to add something in
>>>>> local.conf or the kernel recipe? Is systemd necessary? Or am I
>>>>> missing some life-or-death kernel configuration? It would also be
>>>>> nice if I could avoid the cgroup re-mounting before starting docker.
>>>>
>>>> What release branch are you using ?
>>>>
>>>> I'm running docker from meta-virt every day, as are many others,
>>>> but you have several differences in your configuration.
>>>>
>>>> - most use systemd as the init manager, I know that I do. That
>>>> is going to impact how cgroups is set up on your 'host' image.
>>>> You shouldn't need to touch cgroups at all if systemd is used,
>>>> since it is correct out of the box.
>>>>
>>>> - You are using a different kernel and kernel configuration.
>>>> linux-yocto + the configuration fragments in the layer are what
>>>> is routinely tested. Are you using linux-yocto, or something
>>>> different ? If it is different, all you can do is run the various
>>>> checks to make sure that the docker prereqs are in place.
>>>>
>>>> The errors you see in dockerd tells me that the options you are
>>>> turning on, are not making it into the final kernel that is
>>>> running on target.
>>>>
>>>> Cheers,
>>>>
>>>> Bruce
>>>>
>>>>>
>>>>> Thanks for every answer!
>>>>> All the Best,
>>>>> Jakob
>>>>>
>>>>
>>>
>>
>
More information about the meta-virtualization
mailing list