meta-erlang · joaohf · Oct 14, 2024 · Oct 14, 2024 · Oct 14, 2024 · Oct 14, 2024
diff --git a/blog/2023-01-29/index.md b/blog/2023-01-29/index.md
@@ -20,6 +20,8 @@ The [Yocto Project](https://www.yoctoproject.org/) has been made to create
 specific embedded Linux distributions. And still does a great job on this field
 making it possible to run softwares in a range of target architectures.
 
+<!-- truncate -->
+
 But, the term _embedded Linux distribution_ should be expanded a bit more; it's
 not only about creating tiny filesystem within cross-compiled software, scripts
 and configurations targeting platforms with small footprint, CPU and memory

diff --git a/blog/2023-02-04/index.md b/blog/2023-02-04/index.md
@@ -24,6 +24,8 @@ post has shown the basic idea for how would be possible (and in fact it is
 possible) to build a custom Linux distribution and run it on Graviton AWS
 instances.
 
+<!-- truncate -->
+
 Now, it's time to refactor out the previous result. Removing what are not
 necessary in order to create a slim meta-erlang cloud ready image.
 

diff --git a/blog/2023-09-02/index.md b/blog/2023-09-02/index.md
@@ -21,6 +21,8 @@ So, I'm wondering if it would be possible to enable x32 support in Erlang/OTP
 build. That way, I could make a Yocto image for x32 that runs on x86-64
 machines.
 
+<!-- truncate -->
+
 Here is some references about the subject:
 
 - [Yocto, Using x32 psABI](https://docs.yoctoproject.org/dev-manual/x32-psabi.html)

diff --git a/blog/2023-11-02/index.md b/blog/2023-11-02/index.md
@@ -20,6 +20,8 @@ From [atomvm.net](https://www.atomvm.net/) website, atomvm is:
 One of the atomvm goals is the possibility to run BEAM code on really small
 systems like MCUs.
 
+<!-- truncate -->
+
 For a while, I was wondering what could be the benefits of integrating atomvm
 into meta-erlang recipes. It didn't look to make any sense for me. Then, I
 started playing with that just to see if I was able to run an atomvm program in

diff --git a/blog/2024-04-28/index.md b/blog/2024-04-28/index.md
@@ -13,6 +13,8 @@ From [livebook.dev](https://livebook.dev) website:
 Livebook is getting famous for modern notebooks. It helps a lot for several
 tasks and to solve issues where documentation and code have to walk together.
 
+<!-- truncate -->
+
 Live book has been added to the standard beamtools SDK. That is very handy
 because one can start livebook quickly and it will work with all SDK tools. That
 means, we don't need to install anything else. It's all integrated.

diff --git a/blog/2024-09-24/index.md b/blog/2024-09-24/index.md
@@ -7,6 +7,8 @@ tags: [meta-erlang, fwup]
 This blog post is a tutorial about how to make A/B software updates with fwup
 tool and Yocto Project.
 
+<!-- truncate -->
+
 ## Objectives and Tools
 
 A short sentence as requirement for this tutorial is:

diff --git a/blog/2024-10-06/index.md b/blog/2024-10-06/index.md
@@ -135,11 +135,13 @@ buildhistory is a shortcut because we can inspect how files would be installed
 into the final image, without having to burn the image and boot it on a real (or
 virtual) target.
 
+<!-- prettier-ignore-start -->
 :::note
 Check out the
 [Enabling and Disabling Build History](https://docs.yoctoproject.org/5.0.3/dev-manual/build-quality.html?highlight=buildhistory#enabling-and-disabling-build-history)
 instructions for how to enable buildhistory.
 :::
+<!-- prettier-ignore-end -->
 
 In this post we are interested in check the size of erlang and elixir packages.
 
@@ -250,12 +252,14 @@ automatically installed:
 
 These packages are the basic ones needed in order to run Erlang programs.
 
+<!-- prettier-ignore-start -->
 :::note
 Actually, your application is in charge of making a proper Erlang/OTP
 release. Installing the package `erlang` is an option if you need it installed
 into standard locations. Otherwise, your application should produce a valide
 release with a builtin ERTS inside.
 :::
+<!-- prettier-ignore-end -->
 
 The following charts are listed below to get a view about the size of those
 basic erlang packages.
@@ -317,6 +321,7 @@ xychart-beta
 bitbake -c build multiconfig:x86_64_x32-poky-linux-gnux32:core-image-minimal
 ```
 
+<!-- prettier-ignore-start -->
 :::note
 Enabling x32 psABI should be considered if your application needs to
 handle 64-bit features, but you can't afford wasting space with addressing. See
@@ -325,6 +330,7 @@ the blog post [Exploring x32 psABI for Erlang/OTP](/blog/2023/09/02/index).
 The package size difference in bytes between x32 build and a normal x86-64
 is 4615556.
 :::
+<!-- prettier-ignore-end -->
 
 ```mermaid
 ---
@@ -400,10 +406,12 @@ automatically installed:
 That is because elixir package needs erlang (erts, kernel, stdlib and sasl)
 installed. And, the erlang compiler is also a requirement.
 
+<!-- prettier-ignore-start -->
 :::note
 Again, your application should manage a release for best results.
 Installing the package `elixir` only makes sense for testing purposes.
 :::
+<!-- prettier-ignore-end -->
 
 The following table summarizes the packages sizes per processor architecture.
 
@@ -417,4 +425,3 @@ The following table summarizes the packages sizes per processor architecture.
 
 There are slight size differences. It should not as there is no architecture
 dependency on elixir beam files.
-
diff --git a/blog/2024-10-12/index.md b/blog/2024-10-12/index.md
@@ -0,0 +1,206 @@
+---
+title: Running Erlang/OTP test suite on target with ptest
+authors: [joaohf]
+tags: [meta-erlang]
+---
+
+ptest (package test) means packaging and installing runtime tests that are
+included in many upstream packages. At the end of the day it runs Erlang/OTP
+cross compiled test suite on the target hardware. The aim is to detect a range
+of problems when running test suites on different combinations of architecture
+processors and libc.
+
+<!-- truncate -->
+
+The Erlang/OTP howto about
+[TESTING.md](https://github.com/erlang/otp/blob/master/HOWTO/TESTING.md)
+explains most of the details when running tests. While the Yocto's documentation
+about
+[Testing Packages With ptest](https://docs.yoctoproject.org/dev-manual/packages.html#testing-packages-with-ptest)
+explains the ptest configuration.
+
+So, this post is about how to run those tests using ptest approach on Yocto
+builds.
+
+## How does ptest work ?
+
+Recipes with ptest enabled includes a shell script called _run-ptest_ with the
+steps necessary to start the test suite provided by the target software (e.g.:
+Erlang/OTP).
+
+The run-ptest script is responsible for starting the test suite and
+consolidating the test results into a single format. The format is the
+[automake simple test](https://www.gnu.org/software/automake/manual/automake.html#Simple-Tests)
+format:
+
+```
+result: testName
+```
+
+Where: _result_ is one of PASS, FAIL or SKIP and _testName_ can be anything.
+
+The erlang recipe from meta-erlang layer supports ptest. And, when activated,
+the package erlang-ptest is installed into the final image. It has the erlang
+test suite cross compiled and ready to be executed.
+
+When ptest for erlang recipe gets executed, it finishes the erlang test suite
+configuration and starts to run each test configured in run-ptest script.
+
+The Erlang/OTP requires several hours to finish. Because of that, ptest for
+erlang just executes a small set of all tests available (currently only
+_emulator_ and _kernel_ tests are enabled).
+
+:::note
+
+To get a better idea about how to run Erlang/OTP test suites the documentation
+[TESTING.md](https://github.com/erlang/otp/blob/master/HOWTO/TESTING.md),
+specially the session Running the tests.
+
+:::
+
+The erlang-ptest package installs the test suite and also the run-ptest script
+at _/usr/lib/erlang/ptest_ folder.
+
+One point very important is that Erlang/OTP test suites need some tools
+installed into the target image in order to execute some tests. For example, to
+run emulator and kernel tests the target OS needs tmux and openssh packages.
+However the erlang recipe takes care of these details.
+
+Each test suite might require additional packages. As there is no documentation
+describing all the requirements the best strategy is to inspect the test suite
+for each Erlang/OTP application
+([lib/\*/test](https://github.com/erlang/otp/tree/master/lib) and
+[erts/test](https://github.com/erlang/otp/tree/master/erts/test) folders).
+
+## Running Erlang/OTP ptest
+
+Yocto provides some configuration when enabling ptest. As it is only relevant
+for testing purposes.
+
+A specific image feature exists that enables ptest. So, in the local.conf
+configuration file:
+
+```
+# ptest image
+EXTRA_IMAGE_FEATURES += "ptest-pkgs"
+```
+
+This is all that is necessary to do in order to enable ptest build.
+
+As we want to install erlang in the final image, we also need to add it to the
+final image. On build/local.conf configuration file:
+
+```
+# installing erlang and elixir
+IMAGE_INSTALL:append:pn-core-image-minimal = " erlang erlang-modules-dev elixir elixir-modules-dev links"
+```
+
+:::note
+
+The _links_ package here is necessary to open HTML common test reports from
+console.
+
+:::
+
+It's ready to build the core-image-minimal image:
+
+```
+bitbake core-image-minimal
+```
+
+And, for testing purposes, run the generated image using qemu emulator:
+
+```
+runqemu slirp kvm nographic serialstdio core-image-minimal
+```
+
+Logging into the running system and execute ptest for erlang:
+
+```
+Poky (Yocto Project Reference Distro) 5.1 qemux86-64 ttyS0
+
+qemux86-64 login: root
+
+WARNING: Poky is a reference Yocto Project distribution that should be used for
+testing and development purposes only. It is recommended that you create your
+own distribution for production use.
+
+root@qemux86-64:~# ptest-run erlang
+```
+
+The script /usr/lib/erlang/ptest/run-ptest starts execute controlled by the
+application called [ptest runner](https://git.yoctoproject.org/ptest-runner2/).
+After several hours we get the following results:
+
+```
+START: ptest-runner
+2024-10-14T17:15
+BEGIN: /usr/lib/erlang/ptest
+
+FAIL: emulator
+... lots of console messages ...
+FAIL: kernel
+... lots of console messages ...
+
+=== common test Summary ===
+Testing tests.emulator_test: TEST COMPLETE, 2112 ok, 4 failed, 82 skipped of 2198 test cases
+Testing tests.kernel_test: TEST COMPLETE, 1338 ok, 20 failed, 293 skipped of 1651 test cases
+=== Test Summary ===
+TOTAL: 2
+PASSED: 0
+FAILED: 2
+DURATION: 5822
+END: /usr/lib/erlang/ptest
+2024-10-14T01:39
+STOP: ptest-runner
+TOTAL: 1 FAIL: 0
+```
+
+Well, we got 4 fails for emulator tests and 20 fails for kernel tests. Not bad,
+but I don't know if it is also good.
+
+For further inspection, it's possible to also check output messages for each
+test. The run-ptest scripts writes a specific log console for each test
+executed:
+
+```
+root@qemux86-64:~# ls /usr/lib/erlang/ptest/ -l
+total 14680
+-rw-r--r--  1 root root   236964 Oct 14 00:40 erlang_ptest_emulator_20241014-000215.log
+-rw-r--r--  1 root root 14770546 Oct 14 01:39 erlang_ptest_kernel_20241014-004016.log
+-rwxr-xr-x  1 root root      807 Oct 14 00:01 run-ptest
+drwxr-xr-x 39 root root     4096 Apr  5  2011 tests
+
+```
+
+As usual when running common tests (remember Erlang/OTP test suites are just
+common tests), the HTML reports were written at
+_/usr/lib/erlang/ptest/tests/test_server/index.html_. Opening it with links
+command to check the results:
+
+```
+links /usr/lib/erlang/ptest/tests/test_server/index.html
+```
+
+ptest produces a lot of data to inspect.
+
+## Why running ptest for Erlang/OTP on Yocto builds is important ?
+
+Now that we know what is ptest. Given the following question:
+
+> How do we know that a cross compiled Erlang/OTP build really works on the
+> target hardware ?
+
+The answer could be: testing it using tests provided by Erlang/OTP source code.
+
+So, ptest is great for detecting problems.
+
+As ptest has the ability of running Erlang/OTP test suites on the final image
+and reporting the results, we can get a baseline when comparing tests executed
+on different combinations of processor architectures and C standard libraries
+(libc) or any other aspect like different linux kernel versions. It's possible
+to detect regressions or find weak spots not covered by tests in a specific
+platform.
+
+Also, the mechanism of executing those tests is simple. The user does not need
+to be an Erlang expert to check that something is not ok on some platform.