[yocto] Creating meta-ocaml layer - how to handle system-wide OCaml library manager?

Thu Aug 23 13:13:20 PDT 2018

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On August 22, 2018 1:01 PM, Christopher Clark <christopher.w.clark at gmail.com> wrote:

>>> On Tue, 14 Aug 2018 17:23:55 +0000, Wolfgang Tolkien <w at tolkien.email> wrote:
>>>
>>> I need some help and direction:
>>>
>>> I've started working on a 'meta-ocaml' layer to add support for the
>>> OCaml language to Yocto:
>>>
>>> https://gitlab.com/wtolkien/meta-ocaml
>>>
>>> However, many OCaml projects seem to rely on the 'findlib' library
>>> manager (http://projects.camlcity.org/projects/findlib.html) which keeps
>>> all information in a fixed system-global path.
>>>
>>> This seems to conflict with Yocto's sysroot concept. Also, 'findlib'
>>> handles dependencies, which is obviously something Yocto would normally do.
>>>
>>> What's the best way to handle this? Try to make do without findlib?
>>> Ideally I'd like to end up with an 'ocaml.bbclass' file that just needs
>>> to be included, but maybe that's not achievable?
>
> Hi Wolfgang,
>
> I'm happy to see your interest in building an OCaml layer for Yocto and OpenEmbedded. I have some practical experience with that endeavour and I can hopefully provide some guidance. Your observations above of integration challenges are accurate.
>
> I work on the OpenXT Project, which uses the Xen hypervisor. Xen has some associated Open Source tools written OCaml: eg. vhdtool is useful; there is an OCaml implementation of the xenstore daemon; and the xapi toolstack, used by Citrix XenServer, is written in OCaml. The Mirage Project, which enables production of unikernels from OCaml source, is likely to be the strongest motivator for continued interest in using OCaml with Xen. The ability to cross-compile with OE, on x86-64 hardware for ARM IoT targets would be valuable to Mirage.
>
> My view is that there is a need for input and resources from the OCaml community to build a supportable OCaml OpenEmbedded layer. I think that OE's robust support for cross-compilation, and toolchain for building full integrated systems, would be valuable to that community and could motivate engagement with OE.
>
> The OpenXT Project's current OCaml layer is very simple, found here:
> https://github.com/OpenXT/meta-openxt-ocaml-platform
> and is used by recipes such as this:
> https://github.com/OpenXT/xenclient-oe/blob/master/recipes-openxt/xenclient/uid_git.bb
>
> I have a different version of this layer in a branch on my Github repository with additional functionality, built while researching Opam package manager integration into the layer, plus another layer containing a couple of sample tool recipes to exercise it. I haven't worked on this for several months, due to more pressing priorities, so this is a snapshot of incomplete development work.
>
> # opam integration branch of the layer, with build instructions:
> https://github.com/dozylynx/meta-openxt-ocaml-platform/tree/sumo-opam-integration
>
> # layer with vhd-tool and mirage recipes, and their build dependencies:
> https://github.com/dozylynx/meta-ocaml-apps/tree/sumo
>
> Note that vhd-tool and mirage build with different versions of OCaml (4.02.2 and 4.04.2 respectively) via switches, deliberately: that version of vhd-tool actually works, and mirage needs modern libraries.
>
> About this Opam integration research layer:
>
> * This layer provides a "fetcher" to be used by bitbake that integrates opam,
>   enabling SRC_URI in the recipe to specify OCaml packages to build.
>   This means that you can skip having to write a recipe for every dependency
>   of every opam package: specify what you want and it'll run the solver.
>
> * It respects OE's build steps: eg. it only fetches during "do_fetch".
>
> * It does not require installation of any OCaml tools on the build host; it will
>   build the version of opam that it uses. (Possibly OE-blastphemy though...)
>
> * It supports lockdown of package dependencies. The file produced when
>   running the solver can be captured and added to the recipe for future
>   deterministic builds.
>
> * Patches can be applied to individual dependencies, in the typical OE way,
>   listed in the recipe.
>   (eg. to fix simple portability defects in OCaml software build scripts.)
>
> * It handles building software inside specified Opam "switches", so that
>   you don't have to have all software packages in a layer all use the
>   same version of OCaml: each recipe can use the platform version it needs.
>   This can utilize significant amounts of disk space pretty quickly;
>   reuse of switches has been out of scope for this work so far.
>
>   In my view, this is a requirement for a supportable OCaml layer,
>   because the available OCaml software is highly version-sensitive.
>   Once you find a version of a tool that works, it's important to be
>   able to retain the ability to build it, even as the versions of other
>   tools advance independently.
>
> * It can build OCaml software for both native and target.
>   eg. vhd-tool is very useful native to produce Virtual Machine disk images
>   from OE builds.
>   The layer handles the bytecode hashbang carefully, taking switches
>   and recipe-specific-sysroots into account.
>
> * Cross-compilation support is incomplete. A skeleton structure is there
>   but there are real challenges getting dune/jbuilder to cooperate, and
>   also establish an environment to link correctly.
>   Unfortunately it is common for OCaml software packages to assume that
>   software that has been built can then be run in a later stage of the build.
>
> This layer is definitely incomplete, fairly rough, and should be considered as a prototype, proof-of-concept. There are some pretty serious hacks in place, just to get it close to working -- eg. heuristics to switch between compiling for host and cross-compiling for target -- but with that said, my view is that it does demonstrate a viable structure, and that the objective of delivering integration is achievable. The next step would be to work on the OCaml tools to better support the OE environment.
>
> It hasn't had review by the Yocto, OE or OCaml communities and I don't consider it to be ready for that; but since you asked for just this thing, I thought you might find it of interest. Please let me know if you'd like to talk.
>
> Christopher

Hi and thanks for the amazing work you already did! I can learn a lot here!

When I did my initial posting, I was looking at findlib and 'dune/jbuilder', but after digging a bit deeper I also arrived at 'opam' as possibly the easiest way to achieve results quickly. However, my approach so far is a lot more primitive and reading your response I can see its limitations. Chiefly among them is the lack of support for opam 'switches'. Being a total newbie to OCaml, I wasn't aware of the significance of this feature.

Since cross-compiling is essential to me, all my dev work so far has been for an arm target. Here is a quick summary of my approach so far:

* have a single OPAMROOT in Yocto's build/tmp/works-shared directory.

* have separate recipes for the OCaml compiler: -native, -cross and -runtime (ocamlrun for the target). The compilers are also installed in 'work-shared', with links to them in ocaml-native/-cross's sysroot. This has the advantage to allow me to use fixed paths from opam's perspective, while it least keeping some of OE's structure in place. Nevertheless, all this is clearly Yocto blasphemy, however I don't think this issue can ultimately be resolved since opam and OE are competing for similar responsibilities.

* my OE recipes for opam are a lot more primitive and push almost everything into the 'do_install' task where they call 'opam install'

* it's possible to rely on dependencies defined within opam, however I have no 'smart heuristics' in place as to what opam packages are -native vs. -cross. This has to be handled manually using dependencies within OE recipes, making sure that opam never tries to install both -native and -cross packages within a single 'opam install' run.

* OCaml packages that compile something and then run it during the build process are indeed a problem. My current workaround is to have a native 'ocamlrun' as part of the 'ocaml-cross' compiler sysroot. That way, I just need to make sure anything compiled to be used during build-time is in bytecode.

* I haven't had a lot of exposure opam that use dune yet. Maybe there are still some nasty surprises waiting.

* I completely agree with you that work has to be done on the OCaml tools to make all this smoother....

Again, many thanks for your reply! I'm taking a very good look at your work right now and I'll consider switching to your approach. At a minimum I'm learning a lot about OCaml/Opam path setups, etc.

Wolfgang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20180823/75862ba4/attachment-0001.html>