[yocto] Creating meta-ocaml layer - how to handle system-wide OCaml library manager?

[Christopher Clark]

> > On Tue, 14 Aug 2018 17:23:55 +0000, Wolfgang Tolkien <w at tolkien.email>
> wrote:
> >
> > I need some help and direction:
> >
> > I've started working on a 'meta-ocaml' layer to add support for the
> > OCaml language to Yocto:
> >
> > https://gitlab.com/wtolkien/meta-ocaml
> >
> > However, many OCaml projects seem to rely on the 'findlib' library
> > manager (http://projects.camlcity.org/projects/findlib.html) which keeps
> > all information in a fixed system-global path.
> >
> > This seems to conflict with Yocto's sysroot concept. Also, 'findlib'
> > handles dependencies, which is obviously something Yocto would normally
> do.
> >
> > What's the best way to handle this? Try to make do without findlib?
> > Ideally I'd like to end up with an 'ocaml.bbclass' file that just needs
> > to be included, but maybe that's not achievable?
>
>
Hi Wolfgang,

I'm happy to see your interest in building an OCaml layer for Yocto and
OpenEmbedded. I have some practical experience with that endeavour and I
can hopefully provide some guidance. Your observations above of integration
challenges are accurate.

I work on the OpenXT Project, which uses the Xen hypervisor. Xen has some
associated Open Source tools written OCaml: eg. vhdtool is useful; there is
an OCaml implementation of the xenstore daemon; and the xapi toolstack,
used by Citrix XenServer, is written in OCaml. The Mirage Project, which
enables production of unikernels from OCaml source, is likely to be the
strongest motivator for continued interest in using OCaml with Xen. The
ability to cross-compile with OE, on x86-64 hardware for ARM IoT targets
would be valuable to Mirage.

My view is that there is a need for input and resources from the OCaml
community to build a supportable OCaml OpenEmbedded layer. I think that
OE's robust support for cross-compilation, and toolchain for building full
integrated systems, would be valuable to that community and could motivate
engagement with OE.

The OpenXT Project's current OCaml layer is very simple, found here:
https://github.com/OpenXT/meta-openxt-ocaml-platform
and is used by recipes such as this:
https://github.com/OpenXT/xenclient-oe/blob/master/recipes-openxt/xenclient/uid_git.bb

I have a different version of this layer in a branch on my Github
repository with additional functionality, built while researching Opam
package manager integration into the layer, plus another layer containing a
couple of sample tool recipes to exercise it. I haven't worked on this for
several months, due to more pressing priorities, so this is a snapshot of
incomplete development work.

# opam integration branch of the layer, with build instructions:
https://github.com/dozylynx/meta-openxt-ocaml-platform/tree/sumo-opam-integration

# layer with vhd-tool and mirage recipes, and their build dependencies:
https://github.com/dozylynx/meta-ocaml-apps/tree/sumo

Note that vhd-tool and mirage build with different versions of OCaml
(4.02.2 and 4.04.2 respectively) via switches, deliberately: that version
of vhd-tool actually works, and mirage needs modern libraries.

About this Opam integration research layer:

* This layer provides a "fetcher" to be used by bitbake that integrates
opam,
  enabling SRC_URI in the recipe to specify OCaml packages to build.
  This means that you can skip having to write a recipe for every dependency
  of every opam package: specify what you want and it'll run the solver.

* It respects OE's build steps: eg. it only fetches during "do_fetch".

* It does not require installation of any OCaml tools on the build host; it
will
  build the version of opam that it uses. (Possibly OE-blastphemy though...)

* It supports lockdown of package dependencies. The file produced when
  running the solver can be captured and added to the recipe for future
  deterministic builds.

* Patches can be applied to individual dependencies, in the typical OE way,
  listed in the recipe.
  (eg. to fix simple portability defects in OCaml software build scripts.)

* It handles building software inside specified Opam "switches", so that
  you don't have to have all software packages in a layer all use the
  same version of OCaml: each recipe can use the platform version it needs.
  This can utilize significant amounts of disk space pretty quickly;
  reuse of switches has been out of scope for this work so far.

  In my view, this is a requirement for a supportable OCaml layer,
  because the available OCaml software is highly version-sensitive.
  Once you find a version of a tool that works, it's important to be
  able to retain the ability to build it, even as the versions of other
  tools advance independently.

* It can build OCaml software for both native and target.
  eg. vhd-tool is very useful native to produce Virtual Machine disk images
  from OE builds.
  The layer handles the bytecode hashbang carefully, taking switches
  and recipe-specific-sysroots into account.

* Cross-compilation support is incomplete. A skeleton structure is there
  but there are real challenges getting dune/jbuilder to cooperate, and
  also establish an environment to link correctly.
  Unfortunately it is common for OCaml software packages to assume that
  software that has been built can then be run in a later stage of the
build.

This layer is definitely incomplete, fairly rough, and should be considered
as a prototype, proof-of-concept. There are some pretty serious hacks in
place, just to get it close to working -- eg. heuristics to switch between
compiling for host and cross-compiling for target -- but with that said, my
view is that it does demonstrate a viable structure, and that the objective
of delivering integration is achievable. The next step would be to work on
the OCaml tools to better support the OE environment.

It hasn't had review by the Yocto, OE or OCaml communities and I don't
consider it to be ready for that; but since you asked for just this thing,
I thought you might find it of interest. Please let me know if you'd like
to talk.

Christopher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20180822/d32f9f21/attachment.html>