[meta-xilinx] Linux/FreeRTOS AMP vrings

[Edward Wingate]

Thanks Magnus,

I just got it to work.  Turns out the carveout size in my resource
table should NOT include the vrings.  I had a 32MB carveout size, so I
reduced the size to 4MB so it covers the cpu1 app elf, but not the
vring buffers.  After doing this, the vrings line up with Linux DMA
allocation now.

But running into another problem now.  When my cpu1 app sends the
service announcement to Linux, Linux doesn't see the data (it's all
zeroes):
[   21.750833] zynq_remoteproc 1e000000.cpu1app: KICK Linux because of
pending message
[   21.758851]  remoteproc0: vq index 0 is interrupted
[   21.763659] virtio_rpmsg_bus virtio0: From: 0x0, To: 0x0, Len: 0,
Flags: 0, Reserved: 0
[   21.771925] rpmsg_virtio RX: 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00  ................
[   21.780580] virtio_rpmsg_bus virtio0: msg received with no recipient
[   21.786854] virtio_rpmsg_bus virtio0: Received 1 messages

Also, CPU1 goes off to no man's land after this (PC is pointing to the
vector interrupt table, but doesn't seem able to read code memory
anymore).  I suspect I might have caching/MMU config issue, but not
sure exactly where yet.

My memory allocations:
Linux on CPU0 has first 480MB of DDR RAM (0x0 - 0x1DFFFFFF)
FreeRTOS on CPU1 has last 32MB (0x1E000000 - 0x1FFFFFFF)

The carveout in resource table is defined at (0x1E000000 -
0x1E3FFFFF).  This 4MB is where the cpu1 ELF resides.
The tx/rx vrings are at 0x1E400000 and 0x1E404000, respectively.

In FreeRTOS cpu1 app, 0-480MB are marked as unassigned/reserved in the
MMU table.  480-512MB are marked as inner cached only.  I also tried
marking only 480-484MB as inner cached only, since the cpu1 app elf is
only occupying 4MB with vrings after that, but this didn't help.

Linux is mapping the entire upper 32MB to virtual memory (/proc/vmallocinfo):
0x9f000000-0xa1001000 33558528 dma_declare_coherent_memory+0x58/0xfc
phys=1e000000 ioremap

Could Linux be mucking with the memory allocated to FreeRTOS on CPU1?



On Tue, Jul 14, 2015 at 1:59 AM, Magnus Olsson <Mago at hms.se> wrote:
> Hey,
>
> That's the thing -- there is no automatic way of syncing the AMP application and Linux at the moment. You would expect that the remoteproc subsystem of Linux would honor the da (device address) to the VRINGs specified in the resource table, but it doesn't. This is actually a TODO in the remoteproc code at the moment, see the comment on
> http://lxr.free-electrons.com/source/drivers/remoteproc/remoteproc_core.c#L300
>
> So what happens is, your AMP app declares a VRING, passes its address back to Linux through the resource table, and then Linux ignores it and allocates the VRING using the DMA API.
>
> The only way to make it work (at the moment) is to make sure your AMP application keeps the VRINGs in the same place as Linux DMA API will allocate it. And yes, it is a pain because it can move around. What I've done is to put the VRINGs at a low adress, then my AMP app goes ontop of that and finally Linux runs at offset 16 MB (or something). This makes it more or less static, unless I need to adjust the size of my VRINGs.
>
> If you look at the remoteproc output (some DEBUG might need to be enabled) during a bootup, you'll see it print both the da passed by your AMP app and later on, the da which was the result of the DMA allocation. Unless these match, your rings will be broken.
>
>
> ________________________________________
> From: meta-xilinx-bounces at yoctoproject.org <meta-xilinx-bounces at yoctoproject.org> on behalf of Edward Wingate <edwingate8 at gmail.com>
> Sent: Monday, July 13, 2015 19:14
> To: meta-xilinx at yoctoproject.org
> Subject: Re: [meta-xilinx] Linux/FreeRTOS AMP vrings
>
> On Thu, Jul 2, 2015 at 10:49 AM, Edward Wingate <edwingate8 at gmail.com> wrote:
>> How does this get reconciled with the only addresses that FreeRTOS
>> knows about (1e400000/1e404000)?  It seems I need to somehow get Linux
>> to allocate the vrings at 1e400000/1e404000 instead of where it is
>> currently allocating at.
>
> There's a comment in the FreeRTOS application linker script for vring
> memory placement that says "Linker script has to match Linux dma
> allocation".  So I think I need to modify my linker script somehow,
> but not sure how to go about getting FreeRTOS vring memory allocation
> to match Linux DMA allocation.
>
> The vring memory allocation in the FreeRTOS app linker script puts
> them after the trace buffer and after the end of the ELF:
>
>    /* Trace buffer should be inside carveout */
>    __trace_buffer_start = .;
>    . = . + 0x8000; /* It should be TRACE_BUFFER_SIZE */
>    __trace_buffer_end = .;
>    __elf_end = .; /* This is size of carveout */
>
> /* Linker script has to match Linux dma allocation
> * TX buffer has to start at align address but alignment
> * is done by dma_alloc_from_coherent and get_order and
> bitmap_find_free_region functions
> * Algorithm is easy 1 << (count position the most significant bit from
> ((__elf_end - 1) >> (PAGE_SHIFT -1))
> */
>    . = ALIGN(0x400000);
>
>    __ring_tx_addr = .;
>    . = . + (256 * 16) + (2 *(3+256)); /* vring size macro without
> vring_used_elements */
>
>    . = ALIGN(0x1000); /* Used buffer must be aligned */
>    __ring_tx_addr_used = .;
>    . = . + 4 + (8 * 256); /* vring_used structure */
>    __ring_tx_addr_used_end = .;
>    . = ALIGN(0x2000);
>
>    __ring_rx_addr = .;
>   . = . + (256 * 16) + (2 *(3+256)); /* vring size macro without
> vring_used_elements */
>   . = ALIGN(0x1000);
>   __ring_rx_addr_used = .; /* Used buffer must be aligned */
>   . = . + 4 + (8 * 256); /* vring_used structure */
>   __ring_rx_addr_used_end = .;
>   . = ALIGN(0x2000);
>
> In this case, RING_TX is at 0x1E400000 and RING_RX at 0x1E404000.  I
> tried relocating the vrings to around the memory location Linux wants
> to be allocate them (0x1D000000) and adding another corresponding
> carveout entry in the resource table, but Linux will then allocate the
> vrings at yet another memory location. (And FreeRTOS has problems
> accessing that memory location, probably because MMU settings hasn't
> been set up correctly.)
>
> How do I get FreeRTOS vring memory location (defined in the linker
> script) and the Linux dma allocation for vrings (defined in the
> resource table) to sync up to the same memory location?  The memory
> location for the vrings in the resource table does sync with the
> linker script, but when Linux processes the resource table, it will
> allocate vring at another memory location.  Thanks for your help.
> --
> _______________________________________________
> meta-xilinx mailing list
> meta-xilinx at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-xilinx