[meta-xilinx] Linux/FreeRTOS AMP vrings

[Edward Wingate]

I'm trying to get the vrings working in Linux/FreeRTOS AMP
configuration on a Zynq 7020.  One thing I noticed different from
ug978 is when I do "modprobe zynq_remoteproc," the FreeRTOS binary
seems to start right away, rather than waiting for "modprobe
rpmsg_freertos_statistic" (I relocated my FreeRTOS binary to
0x1E000000, so it's using the upper 32MB out of 512MB total):

# modprobe zynq_remoteproc dyndbg="+p"
[   51.497726] CPU1: shutdown
[   51.501234] zynq_remoteproc 1e000000.cpu1app: 1: Alloc irq: 69
[   51.507009] zynq_remoteproc 1e000000.cpu1app: 2: Alloc irq: 70
[   51.513189] zynq_remoteproc 1e000000.cpu1app: 3: Alloc irq: 71
[   51.519293] zynq_remoteproc 1e000000.cpu1app: 4: Alloc irq: 59
[   51.525063] zynq_remoteproc 1e000000.cpu1app: 5: Alloc irq: 63
[   51.676541] zynq_remoteproc 1e000000.cpu1app: Using firmware: cpu1app.elf
[   51.683372]  remoteproc0: 1e000000.cpu1app is available
[   51.688551]  remoteproc0: Note: remoteproc is still under
development and considered experimental.
[   51.697485]  remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED,
and backward compatibility isn't yet guaranteed.
[   51.726997]  remoteproc0: rsc: type 0
[   51.730647]  remoteproc0: carveout rsc: da 1e000000, pa 1e000000,
len 800000, flags 0
[   51.750280]  remoteproc0: carveout va 9f000000, dma 1e000000, len 0x800000
[   51.757072]  remoteproc0: rsc: type 3
[   51.760746]  remoteproc0: rsc: type 2
[   51.764362]  remoteproc0: rsc: type 4
[   51.768035]  remoteproc0: rsc: type 4
[   51.771654]  remoteproc0: rsc: type 0
[   51.775298]  remoteproc0: rsc: type 3
[   51.778972]  remoteproc0: rsc: type 2
[   51.782590]  remoteproc0: rsc: type 4
[   51.786235]  remoteproc0: rsc: type 4
[   51.789908]  remoteproc0: rsc: type 0
[   51.793527]  remoteproc0: rsc: type 3
[   51.797175]  remoteproc0: vdev rsc: id 7, dfeatures 1, cfg len 0, 2 vrings
[   51.804060]  remoteproc0: vdev rsc: vring0: da 1e400000, qsz 256, align 4096
[   51.811089]  remoteproc0: vdev rsc: vring1: da 1e404000, qsz 256, align 4096
[   51.818149] virtio virtio0: reset !
[   51.821566] virtio virtio0: status: 1
[   51.830776]  remoteproc0: registered virtio0 (type 7)
[   51.835751]  remoteproc0: rsc: type 2
[   51.839638]  remoteproc0: rsc: type 4
[   51.843221]  remoteproc0: rsc: type 4
[   51.849650] virtio_rpmsg_bus virtio0: status: 3
[   51.854165]  remoteproc0: vring0: va 9f800000 dma 1e800000 size 3000 idr 0
[   51.864283]  remoteproc0: vring0: va 9f800000 qsz 256 notifyid 0
[   51.870465]  remoteproc0: vring1: va 9f804000 dma 1e804000 size 3000 idr 1
[   51.877273]  remoteproc0: vring1: va 9f804000 qsz 256 notifyid 1
[   51.883448]  remoteproc0: powering up 1e000000.cpu1app
[   51.890193]  remoteproc0: Booting fw image cpu1app.elf, size 453505
[   51.896385] zynq_remoteproc 1e000000.cpu1app: iommu not found
[   51.902442]  remoteproc0: rsc: type 0
[   51.906021]  remoteproc0: rsc: type 3
[   51.909857]  remoteproc0: rsc: type 2
[   51.913459]  remoteproc0: trace0 added: va 9f035c10, da 0x1e035c10,
len 0x8000
[   51.920815]  remoteproc0: rsc: type 4
[   51.924397]  remoteproc0: rsc: type 4
[   51.928206]  remoteproc0: phdr: type 1 da 0x1e000000 memsz 0x302d0
filesz 0x1c014
[   51.935937]  remoteproc0: phdr: type 1 da 0x1e0302d0 memsz 0x5940
filesz 0x138
[   51.943320] zynq_remoteproc 1e000000.cpu1app: zynq_rproc_start
[   51.949557]  remoteproc0: remote processor 1e000000.cpu1app is now up
[   51.956618]  remoteproc0: kicking vq index: 0
[   51.961157] zynq_remoteproc 1e000000.cpu1app: KICK Firmware to
start send messages vqid 0
[   51.969365] virtio_rpmsg_bus virtio0: rpmsg host is online
[   51.974771] virtio_rpmsg_bus virtio0: status: 7
[   51.979340] random: nonblocking pool is initialized

Regardless of this difference from ug978, I can see the FreeRTOS
firmware receiving the firmware kick (SGI 2) from Linux and it tries
to send a rpmsg reply, but the RING_TX area is not initialized
properly (it is all zeroes) and so the vring descriptor it gets is
invalid. It seems the RING_TX area should be initialized based on the
debug messages above.

Is it possible Linux is initializing RING_TX at a different place than
where FreeRTOS thinks it should be?  Is the resource table in the elf
the only place where the vrings are defined? The resource table looks
like this:

const struct rproc_resource {
    struct resource_table base;
    struct fw_rsc_carveout code_cout;

    /* rpmsg vdev entry */
    struct fw_rsc_vdev rpmsg_vdev;
    struct fw_rsc_vdev_vring rpmsg_vring0;
    struct fw_rsc_vdev_vring rpmsg_vring1;

    /* trace entry */
    struct fw_rsc_trace trace;
    struct fw_rsc_mmu uart0;
    struct fw_rsc_mmu scu;
} ti_ipc_remoteproc_ResourceTable = {
    .base = {
       .ver = 1,
       .num = 5,
       .reserved = { 0, 0 },
       .offset = {
            offsetof(struct rproc_resource, code_cout),
            offsetof(struct rproc_resource, rpmsg_vdev),
            offsetof(struct rproc_resource, trace),
            offsetof(struct rproc_resource, uart0),
            offsetof(struct rproc_resource, scu)
        },
    },
    .code_cout = {
        .type = TYPE_CARVEOUT,
        .da = 0x1E000000,
        .pa = 0x1E000000,
        .len = 1<<25,
        .flags=0,
        .reserved=0,
        .name="CPU1CODE",
    },

    /* rpmsg vdev entry */
    .rpmsg_vdev = { TYPE_VDEV, VIRTIO_ID_RPMSG, 0,
RPMSG_IPU_C0_FEATURES, 0, 0, 0, 2, { 0, 0 }, /* no config data */ },

    /* the two vrings */
    .rpmsg_vring0 = { RING_TX, 0x1000, VRING_SIZE, 1, 0 },
    .rpmsg_vring1 = { RING_RX, 0x1000, VRING_SIZE, 2, 0 },

    /* Trace buffer */
    .trace = { TYPE_TRACE, TRACE_BUFFER_START, TRACE_BUFFER_SIZE, 0,
"trace_buffer", },

    /* Peripherals */
    .uart0 = { TYPE_MMU, 1, STDOUT_BASEADDRESS, 0, 0xc02, "uart", },
    .scu = { TYPE_MMU, 2, XPS_SCU_PERIPH_BASE, 0, 0xc02, "scu", },
};

It's been adapted from the one in ug978.  I wasn't interested in the
latency demo aspects of ug978 and so didn't add the timer and latency
stuff.  I just want to get the inter-CPU data passing working.

Anybody know where to debug from here, why it seems like the vring
descriptors are not initialized properly?  Thanks for your help.