[meta-virtualization] RFT/FYI: docker/containerd/runc uprevs pushed to master
Shakthi Pradeep (tpradeep)
tpradeep at cisco.com
Wed Apr 11 07:50:04 PDT 2018
Hello Bruce,
Could you try running Docker with –p option. After solving the iptables issue, I am hitting another issue.
Now exec of /usr/bin/docker-proxy gives “no such file or directory” error
root at genericx86-64:/mnt# docker run -itd --name svo -p 10000:8080 svo supervisord -n
29bd8de9dd028888f5f060cb42240762e28dd07cdfa91185fa49953c3da28c36
docker: Error response from daemon: driver failed programming external connectivity on endpoint svo (6397cfd4d4e69099c609c8a933e35418a0bf30c1acb4e3b0ebad55668e099fa1): fork/exec /usr/bin/docker-proxy: no such file or directory.
root at genericx86-64:/mnt# which /usr/bin/docker-proxy
/usr/bin/docker-proxy
root at genericx86-64:/mnt# ldd /usr/bin/docker-proxy
linux-vdso.so.1 (0x00007ffd1bbf3000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0000003267a00000)
libc.so.6 => /lib/libc.so.6 (0x0000003267600000)
/lib64/ld-linux-x86-64.so.2 => /lib/ld-linux-x86-64.so.2 (0x00007f0c9e5dc000)
root at genericx86-64:/mnt#
Regards,
Shakthi
On 11/04/18, 8:15 PM, "Shakthi Pradeep (tpradeep)" <tpradeep at cisco.com> wrote:
Hello Bruce,
Since I am having build issues with master branch I am back to rocko branch.
With rocko branch when I use –p option with Docker, I was getting following error “iptables: No chain/target/match by that name.”
root at genericx86-64:~# docker run -itd --name svo -p 10000:8080 svo supervisord -n
52416b288a255361de6e0e09e78055627d898e44cc6b9839430bb2e48b5c5324
docker: Error response from daemon: driver failed programming external connectivity on endpoint svo (b5a9bed9b9825027ce02fa603b5daa3d4220953404f8d3efbda4ab5f0ac49138): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 10000 -j DNAT --to-destination 172.17.0.2:8080 ! -i docker0: iptables: No chain/target/match by that name.
(exit status 1)).
This was due to missing LKM, xt_nat & x_tables. With this bundled in the final iso I could solve above problem. Now I am hitting another error
Below is the complete list of modules
root at genericx86-64:/mnt# lsmod
Module Size Used by
xt_tcpudp 16384 0
ipt_MASQUERADE 16384 1
nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE
iptable_nat 16384 1
nf_conntrack_ipv4 16384 3
nf_defrag_ipv4 16384 1 nf_conntrack_ipv4
nf_nat_ipv4 16384 1 iptable_nat
iptable_filter 16384 1
ip_tables 24576 2 iptable_filter,iptable_nat
bridge 106496 0
stp 16384 1 bridge
llc 16384 2 bridge,stp
xt_nat 16384 0
nf_nat 24576 3 xt_nat,nf_nat_masquerade_ipv4,nf_nat_ipv4
xt_conntrack 16384 1
nf_conntrack 86016 7 xt_nat,nf_conntrack_ipv4,ipt_MASQUERADE,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_nat
libcrc32c 16384 2 nf_conntrack,nf_nat
xt_addrtype 16384 2
x_tables 28672 7 xt_nat,ip_tables,iptable_filter,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
root at genericx86-64:/mnt#
Regards,
Shakthi
On 09/04/18, 7:49 PM, "Bruce Ashfield" <bruce.ashfield at gmail.com> wrote:
Sure, but none of them are docker.
Bruce
On Mon, Apr 9, 2018 at 9:46 AM, Shakthi Pradeep (tpradeep)
<tpradeep at cisco.com> wrote:
> Hello Bruce,
>
> I see pkg_postinst_${PN}() are part of below recipes….
>
> TPRADEEP-M-91HW:meta-virtualization tpradeep$ grep -r postinst *
> recipes-containers/singularity/singularity_git.bb:pkg_postinst_${PN}() {
> recipes-containers/lxc/lxc_2.0.8.bb:pkg_postinst_${PN}() {
> recipes-containers/lxc/lxc_2.0.8.bb:pkg_postinst_${PN}-networking() {
> recipes-extended/xen/xen.inc:pkg_postinst_${PN}-volatiles() {
> recipes-extended/libvirt/libvirt_1.3.5.bb:pkg_postinst_libvirt() {
> recipes-networking/openvswitch/openvswitch.inc:# rdeps. E.g. ovs-pki calls sed in the postinstall. sed may be
> recipes-networking/openvswitch/openvswitch.inc:pkg_postinst_${PN}-pki () {
> recipes-networking/openvswitch/openvswitch.inc:pkg_postinst_${PN}-testcontroller () {
>
> Regards,
> Shakthi
>
> On 09/04/18, 6:47 PM, "Bruce Ashfield" <bruce.ashfield at gmail.com> wrote:
>
> On Mon, Apr 9, 2018 at 4:37 AM, Shakthi Pradeep (tpradeep)
> <tpradeep at cisco.com> wrote:
> > Hello Bruce,
> >
> > Moved to master branch of Yocto. When I build I am getting below error. What does this mean?
>
> There were changes made to the way that postinstall scripts were run
> in the January
> timeframe,
>
> i.e.:
>
> commit 7bc55b29609765904af9f330600229e96cc044cf
> Author: Alexander Kanavin <alexander.kanavin at linux.intel.com>
> Date: Mon Jan 29 14:01:32 2018 +0200
>
> meta/lib/oe/package_manager.py: deprecate 'exit 1' as a way to
> defer to first boot
>
> 'exit 1' is not optimal for two reasons:
>
> 1) Code is hard to read; it is not obvious that it means 'defer
> what follows to first boot'.
> 2) Worse, this hides actual errors in the scriptlets; there is no
> difference between scriptlet
> failing because it's intended to be run on target and scriptlet
> failing because there's a bug or
> a regression somewhere.
>
> The new, supported way is to place the code that has to run on
> target into pkg_postinst_ontarget(),
> or, if a more fine-tuned control is required, call
> 'postinst-intercepts defer_to_first_boot' from
> pkg_postinst() to explicitly request deferral to first boot.
>
> (From OE-Core rev: d12cf56e9ff2a4f13dfbef9290ea5647b52b3f6d)
>
> Signed-off-by: Alexander Kanavin <alexander.kanavin at linux.intel.com>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
>
> :100644 100644 2a07f0e39ad6... f7e013437c91... M
> meta/lib/oe/package_manager.py
>
> commit 6ca669105ff7f405e85142d1aa5375a8430c9606
> Author: Alexander Kanavin <alexander.kanavin at linux.intel.com>
> Date: Mon Jan 29 14:01:31 2018 +0200
>
> package.bbclass: add support for pkg_postinst_ontarget()
>
> This function is a convenient and more readable shortcut for situations
> when the postinst code always needs to run on target. All commands that
> cannot be executed during cross-install and can only be run on target
> should go into this function. They will only be executed on first boot
> (if package was cross-installed) or immediately during package installation
> on target.
>
> Plain pkg_postinst() works as before: it is run during cross-install time,
> it can contain a request to defer to first boot, and it is also run
> during package installation on target.
>
> Also fix the oeqa test for this functionality to use the new function
> where appropriate.
>
> (From OE-Core rev: 229f4e975fb6957f44b5c56735fd6d58564098d7)
>
> Signed-off-by: Alexander Kanavin <alexander.kanavin at linux.intel.com>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
>
> :100644 100644 112aa08c80f8... d4bab6dcc228... M
> meta-selftest/recipes-test/postinst/postinst_1.0.bb
> :100644 100644 7dc759699f4b... 6a7f35a3e787... M
> meta/classes/package.bbclass
>
> -------
>
> But you shouldn't be getting that warning with docker, since it doesn't
> have a pkg_postinst_${PN}(), or related element in the recipe.
>
> So something in your build is adding that postinstall action, that isn't
> part of the meta-virtualization layer.
>
> Bruce
>
>
> >
> > Initialising tasks: 100% |######################################################################################################################################################| Time: 0:00:04
> > NOTE: Executing SetScene Tasks
> > NOTE: Executing RunQueue Tasks
> > WARNING: core-image-full-cmdline-1.0-r0 do_rootfs: Intentionally failing postinstall scriptlets of ['docker'] to defer them to first boot is deprecated. Please place them into pkg_postinst_ontarget_${PN} ().
> > If deferring to first boot wasn't the intent, then scriptlet failure may mean an issue in the recipe, or a regression elsewhere.
> > Details of the failure are in /workdir/poky/build/tmp/work/genericx86_64-poky-linux/core-image-full-cmdline/1.0-r0/temp/log.do_rootfs.
> > WARNING: core-image-full-cmdline-1.0-r0 do_rootfs: [log_check] core-image-full-cmdline: found 2 warning messages in the logfile:
> > [log_check] warning: %post(docker-18.03.0+git708b068d3095c6a6be939eb2da78c921d2e945e2-r0.core2_64) scriptlet failed, exit status 1
> > [log_check] WARNING: Intentionally failing postinstall scriptlets of ['docker'] to defer them to first boot is deprecated. Please place them into pkg_postinst_ontarget_${PN} ().
> >
> > Regards,
> > Shakthi
> >
> > On 06/04/18, 1:09 AM, "Bruce Ashfield" <bruce.ashfield at gmail.com> wrote:
> >
> > On Thu, Apr 5, 2018 at 11:55 AM, Shakthi Pradeep (tpradeep)
> > <tpradeep at cisco.com> wrote:
> > > Hello Bruce,
> > >
> > > I pulled master branch from git://git.yoctoproject.org/poky but build fails
> > >
> > > When I run "bitbake core-image-full-cmdline" or "bitbake core-image-minimal" I get following error
> > >
> > > ERROR: Task (/nobackup/tpradeep/playground/yocto/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.15.7.bb:do_install) failed with exit code '1'
> > > ERROR: Task (/nobackup/tpradeep/playground/yocto/poky/meta/recipes-devtools/autoconf-archive/autoconf-archive_2016.09.16.bb:do_install) failed with exit code '1'
> > >
> > > Any idea why?
> >
> > Nope. But the errors are unrelated to the changes in meta-virt. If the
> > issue persists, asking on the oe-core list is a better bet.
> > I'd suggest that you rm -rf tmp, and start the build again.
> >
> > Bruce
> >
> > >
> > > Regards,
> > > Shakthi
> > >
> > > -----Original Message-----
> > > From: Bruce Ashfield [mailto:bruce.ashfield at gmail.com]
> > > Sent: Thursday, April 05, 2018 7:44 PM
> > > To: Shakthi Pradeep (tpradeep) <tpradeep at cisco.com>
> > > Cc: meta-virtualization at yoctoproject.org
> > > Subject: Re: [meta-virtualization] RFT/FYI: docker/containerd/runc uprevs pushed to master
> > >
> > > On Thu, Apr 5, 2018 at 9:32 AM, Shakthi Pradeep (tpradeep) <tpradeep at cisco.com> wrote:
> > >> Hello Bruce,
> > >>
> > >> I am actually working on a repo from Third Party. So here things are little outdated.
> > >>
> > >> I found some info from http://git.yoctoproject.org
> > >>
> > >> I am currently using Docker 1.13.0 which was committed on 2017-01-20
> > >>
> > >> http://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/commit/r
> > >> ecipes-containers/docker?id=0b631bf01487d3d1be0c91ea7749168fb3c47dcb
> > >>
> > >> I need to migrate to 18.03.0 which you committed 3 days ago
> > >>
> > >> http://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/commit/r
> > >> ecipes-containers/docker?id=a5074cecf18faea1a325c8ac9220d2df41250af5
> > >>
> > >> If I take recipe from above commit and update my workspace, will all the dependencies migrate to latest automatically?
> > >
> > > Nope. You'd need to pull in all the updates that I recently pushed for the dependencies, or you'll end up with runtime errors.
> > > Also note, that due to the different go versions and build infrastructure in oe-core, there's no guarantee that those new recipes will build properly on an older baseline.
> > >
> > > Bruce
> > >
> > >>
> > >> Regards,
> > >> Shakthi
> > >>
> > >> -----Original Message-----
> > >> From: Bruce Ashfield [mailto:bruce.ashfield at gmail.com]
> > >> Sent: Thursday, April 05, 2018 6:46 PM
> > >> To: Shakthi Pradeep (tpradeep) <tpradeep at cisco.com>
> > >> Cc: meta-virtualization at yoctoproject.org
> > >> Subject: Re: [meta-virtualization] RFT/FYI: docker/containerd/runc
> > >> uprevs pushed to master
> > >>
> > >> root at cube-server:~# docker --version
> > >> Docker version 18.03.0, build 0f1bb353423e
> > >>
> > >> If you are building meta-virt master, the update should happen automatically since the PV of the updated recipe is higher than the old one. All the dependencies will follow after that.
> > >>
> > >> Bruce
> > >>
> > >> On Thu, Apr 5, 2018 at 9:02 AM, Shakthi Pradeep (tpradeep) <tpradeep at cisco.com> wrote:
> > >>> Hello Bruce,
> > >>>
> > >>> Which version of Docker are you running.
> > >>>
> > >>> I just noticed I am running Docker 1.13.1 which is too old.
> > >>>
> > >>> How do I migrate to latest version of Docker & also corresponding dependencies.
> > >>>
> > >>> Regards,
> > >>> Shakthi
> > >>>
> > >>> -----Original Message-----
> > >>> From: Bruce Ashfield [mailto:bruce.ashfield at gmail.com]
> > >>> Sent: Wednesday, April 04, 2018 6:45 PM
> > >>> To: Shakthi Pradeep (tpradeep) <tpradeep at cisco.com>
> > >>> Cc: meta-virtualization at yoctoproject.org
> > >>> Subject: Re: [meta-virtualization] RFT/FYI: docker/containerd/runc
> > >>> uprevs pushed to master
> > >>>
> > >>> Excellent news.
> > >>>
> > >>> Yes, everything is in master of meta-virtualization now. That way it will have time to soak a bit before the upcoming release.
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Bruce
> > >>>
> > >>> On Wed, Apr 4, 2018 at 8:32 AM, Shakthi Pradeep (tpradeep) <tpradeep at cisco.com> wrote:
> > >>>> Hello Bruce,
> > >>>>
> > >>>> Yes I was missing the bbappend for kernel config. Got it working after I wrote the mail to you.
> > >>>>
> > >>>> Are your changes committed to git repo? Is it in the master branch of Yocto?
> > >>>>
> > >>>> Regards,
> > >>>> Shakthi
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Bruce Ashfield [mailto:bruce.ashfield at gmail.com]
> > >>>> Sent: Wednesday, April 04, 2018 5:57 PM
> > >>>> To: Shakthi Pradeep (tpradeep) <tpradeep at cisco.com>
> > >>>> Cc: meta-virtualization at yoctoproject.org
> > >>>> Subject: Re: [meta-virtualization] RFT/FYI: docker/containerd/runc
> > >>>> uprevs pushed to master
> > >>>>
> > >>>> On Wed, Apr 4, 2018 at 12:57 AM, Shakthi Pradeep (tpradeep) <tpradeep at cisco.com> wrote:
> > >>>>> Hello Bruce,
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Timing is Perfect !!!
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> I am currently trying to get Docker CE to work with Yocto. I could
> > >>>>> include the Docker executable in ISO but when I run it I get some errors.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> When I boot the image looks like Docker service start is failing
> > >>>>> due to missing kernel modules. Please refer attached screenshot and
> > >>>>> below error log.
> > >>>>
> > >>>> Do you have a bbappend for your 4.8 kernel that adds the docker configuration fragments ?
> > >>>>
> > >>>> That's the most likely reason for the issues.
> > >>>>
> > >>>> I was able to run a whole suite of tests against 4.12, 4.14 and 4.15 so those kernels + fragments are known to work.
> > >>>>
> > >>>> Bruce
> > >>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> * docker.service - Docker Application Container Engine
> > >>>>>
> > >>>>> Loaded: loaded (/lib/systemd/system/docker.service; enabled;
> > >>>>> vendor
> > >>>>> preset: enabled)
> > >>>>>
> > >>>>> Active: failed (Result: exit-code) since Tue 2018-04-03 13:17:51
> > >>>>> UTC; 17min ago
> > >>>>>
> > >>>>> Docs: https://docs.docker.com
> > >>>>>
> > >>>>> Process: 317 ExecStart=/usr/bin/dockerd -H fd:// (code=exited,
> > >>>>> status=1/FAILURE)
> > >>>>>
> > >>>>> Main PID: 317 (code=exited, status=1/FAILURE)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 dockerd[317]:
> > >>>>> time="2018-04-03T13:17:51.035178755Z" level=warning msg="Running
> > >>>>> modprobe xt_conntrack failed with message: `modprobe: WARNING:
> > >>>>> Module xt_conntrack not found in directory
> > >>>>> /lib/modules/4.8.24-WR9.0.0.10_standard`, error: exit status 1"
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 dockerd[317]:
> > >>>>> time="2018-04-03T13:17:51.040727372Z" level=info msg="Firewalld running:
> > >>>>> false"
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 dockerd[317]:
> > >>>>> time="2018-04-03T13:17:51.170575344Z" level=warning msg="Could not
> > >>>>> load necessary modules for IPSEC rules: Running modprobe xfrm_user
> > >>>>> failed with
> > >>>>> message: `modprobe: WARNING: Module xfrm_user not found in
> > >>>>> directory /lib/modules/4.8.24-WR9.0.0.10_standard`, error: exit status 1"
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 dockerd[317]:
> > >>>>> time="2018-04-03T13:17:51.172397913Z" level=info msg="Default
> > >>>>> bridge
> > >>>>> (docker0) is assigned with an IP address 172.17.0.0/16. Daemon
> > >>>>> option --bip can be used to set a preferred IP address"
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 dockerd[317]: Error starting daemon:
> > >>>>> Error initializing network controller: Error creating default "bridge" network:
> > >>>>> Failed to Setup IP tables: Unable to enable ACCEPT INCOMING rule:
> > >>>>> (iptables
> > >>>>> failed: iptables --wait -I FORWARD -o docker0 -m conntrack
> > >>>>> --ctstate RELATED,ESTABLISHED -j ACCEPT: iptables: No chain/target/match by that name.
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 dockerd[317]: (exit status 1))
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 systemd[1]: docker.service: Main
> > >>>>> process exited, code=exited, status=1/FAILURE
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 systemd[1]: Failed to start Docker
> > >>>>> Application Container Engine.
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 systemd[1]: docker.service: Unit
> > >>>>> entered failed state.
> > >>>>>
> > >>>>> Apr 03 13:17:51 intel-x86-64 systemd[1]: docker.service: Failed
> > >>>>> with result 'exit-code'.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Regards,
> > >>>>>
> > >>>>> Shakthi
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> -----Original Message-----
> > >>>>> From: meta-virtualization-bounces at yoctoproject.org
> > >>>>> [mailto:meta-virtualization-bounces at yoctoproject.org] On Behalf Of
> > >>>>> Bruce Ashfield
> > >>>>> Sent: Wednesday, April 04, 2018 8:44 AM
> > >>>>> To: meta-virtualization at yoctoproject.org
> > >>>>> Subject: [meta-virtualization] RFT/FYI: docker/containerd/runc
> > >>>>> uprevs pushed to master
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> After spending a few days de-tangling the
> > >>>>> moby/docker/runc/containerd and oe-core go infrastructure changes,
> > >>>>> I was able to run docker/runc/containerd through a system/stress test and everything seems to be working.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> There were a few regressions that I worked through, as well as
> > >>>>> build/packaging changes, but I'm no longer seeing any issues and
> > >>>>> all the patches/functionality have been carried forward.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> One thing of note is that the docker and open containers containerd
> > >>>>> split/fork is no longer an issue, so I've modified the default to
> > >>>>> be the opencontainers variant. Similarly, the docker and
> > >>>>> opencontainers runc are very similar. I've kept both variants of
> > >>>>> both recipes for now, since I'd like to track things for a bit
> > >>>>> longer before declaring the split unnecessary.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Also for those that care, I created a reference docker-ce recipe
> > >>>>> that tracks the docker-ce repo versus the components themselves.
> > >>>>> Right now it is reference only, since it needs a bit more work, but
> > >>>>> I wanted to get it out there, in case someone really cares about
> > >>>>> docker-ce (I don't really, but someone might!).
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Summary: I just pushed the following changes to master:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> d7d310ae4113 meta-virt: prefer containerd-opencontainers
> > >>>>>
> > >>>>> 935e3d969ef1 containerd: uprev to v1.0.2
> > >>>>>
> > >>>>> f5fbfa8ac4db docker-ce: introduce reference recipe/build
> > >>>>>
> > >>>>> a5074cecf18f docker: uprev to 18.03.0
> > >>>>>
> > >>>>> e3d960f4fcd9 runc: uprev to 1.0.0-rc5
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> If anyone sees regressions, build or architecture issues .. report
> > >>>>> them to me (and the list) and we'll get them fixed up.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Cheers,
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Bruce
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> "Thou shalt not follow the NULL pointer, for chaos and madness
> > >>>>> await thee at its end"
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>>
> > >>>>> meta-virtualization mailing list
> > >>>>>
> > >>>>> meta-virtualization at yoctoproject.org
> > >>>>>
> > >>>>> https://lists.yoctoproject.org/listinfo/meta-virtualization
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
> > >>
> > >>
> > >>
> > >> --
> > >> "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
> > >
> > >
> > >
> > > --
> > > "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
> >
> >
> >
> > --
> > "Thou shalt not follow the NULL pointer, for chaos and madness await
> > thee at its end"
> >
> >
>
>
>
> --
> "Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end"
>
>
--
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"
More information about the meta-virtualization
mailing list