ODroid as raplacement for Raspberry Pi

Yes we use cyclictest to compare to other systems than run cyclictest. However, FF + youtubes gives us the worst latency test scores vs any other tests, cyclictest included..

> The real test for latency has been starting Firefox and then playing a youtube.

You have to be careful when doing stress testing. The test needs to look like the load you expect to have for the numbers to be relevant. The workload of this board is only LinuxCNC, no browsing or video decoding ever. And AFAIK the workload of decoding video is nothing like the workload of running LinuxCNC.

If you go by playing video, then you're getting absolute worst case numbers of the board as a video decoder, not the board as a LinuxCNC host. Now if you were going to use the board to watch video while LinuxCNC was running, then I agree that playing a video as a load test would be good. But this board will run LinuxCNC in kiosk mode.

I agree with this Alan. Lets face it. If you are doing things with the PC, you are not (or shouldn't be) running a machine at the same time...
I've been scouring the net for kernel config settings for arm and preempt_rt.
Interesting there is a new preempt mode that lets you select PREEMPT_RT and other modes as you would isolcpus. But not in the 5.10 kernel you use.

> If you are doing things with the PC, you are not (or shouldn't be) running a machine at the same time...

Agreed. I'm planning to run LinuxCNC in kiosk mode so you literally can't do anything else. I won't even have a desktop installed.

> I've been scouring the net for kernel config settings for arm and preempt_rt.

I did the same and came up with the ones you see in that build script. Nothing else seemed to do any good, or actually made the numbers worse.

This is probably a good time to mention that it's critical to start processes with schedtool (like `schedtool -a 2-5 -e nice -n99 latency-histogram`). Giving the process a high priority and binding it to the isolated cpus has a big positive effect on latency.

> Interesting there is a new preempt mode that lets you select PREEMPT_RT

That's interesting. I only knew about the CONFIG_PREEMPT_RT kernel build config setting that gets you an RT kernel.

> Everything looks right but something goes awry with threading

I had about every failure imaginable. I tried Armbian, but the build would fail I think because one of the official RT patches wouldn't apply cleanly.

I also had lots of boot failures with tobetter's Ubuntu 22.04 image. It u-boot appears to be very sensitive to wrong kernel commandline in boot.scr.

The compiler you use also seems to matter a lot. The kernel wouldn't boot when built with Linaro ARM compiler v7 (as HardKernel recommends), but it works perfectly when built with Linaro ARM compiler v12.

And LinuxCNC won't build with Linaro v12. I have to switch to using the cross compiler in the crossbuild-essential-arm64 package.

> Do you have an image you can share somewhere?

Give this a try: drive.google.com/file/d/1wbU988tXvZyPX5V.../view?usp=share_link

(Warning to future readers: that link will go dead once we have a better build option)

> Has Tobetter's image already got it in there?

They maintain Odroid and PREEMPT_RT patches in a separate repo: github.com/tobetter/linux

> If we can migrate the process to Armbian's build system, the process will be a lot easier.

Indeed. It looks like Armbian just pulls from HardKernel's kernel repo. In theory you could just swap in tobetter's repo url. I'm pretty sure there are some other build differences though.

> It would be good to see the output from cyclictest as it speaks a language other RT guys understand.

I'll post those numbers today.

I've forked the Armbian Build process here
github.com/rodw-au/armbianbuild
I'm currently building for Bookworm edge with./compile.sh EXPERT=yes
build an image
don't change the kernel
add a graphical environment xfce
add browser, editor internet
build and burn to an SD (not emmc yet)
I'm not sure how much can be stored in git yet....

I think Armbian just pull the pettitboot (and maybe uboot) from HardKernel.
You can select the image used by editing this file so you can use a matching RT build to your kernel
github.com/armbian/build/blob/master/con...e/meson64_common.inc

These are some of the kernel settings I wanted to review


CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_KVM=n
CONFIG_VIRTUALIZATION=n
CONFIG_LOCALVERSION_AUTO=y
CONFIG_ARCH_SUPPORTS_RT=y
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_HZ_PERIODIC=y
CPU_IDLE_GOV_LADDER =y
EXPERT=y ?????????
CONFIG_NO_HZ_FULL=y

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_IDLE=n
CONFIG_EMBEDDED=y
I also wanted to review toebetters commits to see what is changing in his kernel.

To the best of my knowledge,

Master switch to build RT kernel
CONFIG_PREEMPT_RT=y

Needed to enable CONFIG_PREEMPT_RT (on >= 5.15 ?)
CONFIG_VIRTUALIZATION=n
CONFIG_ARCH_SUPPORTS_RT=y

Doesn't matter due to CONFIG_PREEMPT_RT
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y

Alias for CONFIG_EXPERT (lwn.net/Articles/421304/)
CONFIG_EMBEDDED=y

Off because we don't need virtualized guests
CONFIG_KVM=n

Automatically append version information to the version string
CONFIG_LOCALVERSION_AUTO=y

Mutually exclusive. We want PERFORMANCE so we're always at max frequency
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y

menuconfig Menu only
CONFIG_CPU_FREQ=y

menuconfig Menu only (github.com/torvalds/linux/blob/master/drivers/cpuidle/Kconfig)
CONFIG_CPU_IDLE=n
CPU_IDLE_GOV_LADDER =y

Thanks, I will try and set some things tonight.
Currently am using kernel 6.2 rc 3  with Armbian.edge.
Thinking about it I will revert to 6.1.12 and Armbian Current. This should boot on an eMMC and align with the bookworm kernel (and htere is a RT patch)
In my PREEMPT_RT branch, there is a file called command.txt which is a single command that the build envionment gives me at the end which should replicate what I have run. I'll keep it current. It will take a couple of days to get this sorted out before my repo will do much.

It will be something like
Give it a go and let me know how it goes (burn on SD card not eMMC and boot with switch to the right). Image will be in the output folder.
I use Balena Etcher on Linux whjich is super cool but eventually, we can make it burn as part of the script if we want.






 

> You have to be careful when doing stress testing. The test needs to look like the load you expect to have for the numbers to be relevant.

We have little control over how the applications and drivers are developed so we look for worst case in testing. If you wish to gamble on what the lowest latency might be to run a base thread for software stepping on your hobby machine that is up to you. There is no latency police enforcing anything involved here

If any of the ODroids end up testing reliable enough to use in production machines they will be be running servo threads anyway with external FPGAs for steppers, encoders and servos.

Bari wrote:

If any of the ODroids end up testing reliable enough to use in production machines they will be be running servo threads anyway with external FPGAs for steppers, encoders and servos.

I don't think there is any doubt Odroid's  are reliable as I've used several of their  x86 devices and they work fine once you get the NIC drivers installed. (Hint: the 5.10 kernel -  Bullseye and above includes the driver). One of them runs Bullseye and Ethercat. Its just their user base is not really worried about Real time so we are not catered for.

You see ignorant comments like this one whxih was well retorted!
forum.armbian.com/topic/13250-preempt-rt...omment&comment=96666
 Some people have no idea. 20 ms is real time what crap!
Nice reply
With the Odroid N2, you get a 6 core CPU (4 BIG, 2 little for power saving)

I think if you look at kernel development both normal and real time, there is a lot of effort happening to support the ARM environment, some of it is directed at the N2. You can't use this stuff on the Buster 4.x kernel......