Since our last update in May we have continued working on hardware-testing automation. We are happy to announce that we have reached an important milestone: we are now testing the OnePlus 6T as part of our regular development workflow! If you are not sure why this is important, or want to read more about the design decisions and the different parts that compromise the system, you can find all the information in the previous blog post.
Work done and current status
Since May, the system that we designed has been put in place. To make it possible, the following work has been executed:
- 4 QEMU runners available, 2 for x86_64, 1 for aarch64, and 1 for armv7 are now available in GitLab, and can be used in any project. This can be done importing the corresponding tags. Thanks Oliver and Martin R. for setting these up!
- A OnePlus 6T runner is now also available in GitLab, which means that if selecting this device, tests will run on real hardware. You can see a full demo video of the device booting up in here. Thanks Martin R. and Clayton for setting it up!
- Thanks to Martin R. and Pablo, support for integration testing on the device-runners is now available in our GitLab (!17, !19, !20) and integrated in:
- Thanks to Federico and ncorna, the PCB to control the hardware went through 3 revisions:
- The first revision (v0.1) included just 2 assembled boards, and other empty ones for debugging. The PCBs were received, and although non-functional, most issues could be debugged successfully.
- The second version (v0.2) included the fixes from the first revision, and consisted already of 5 fully assembled PCBs. This version is already good enough for being used to power the OnePlus 6T in Martin R.'s farm.
- The third revision (v0.3), has some small fixes, and potentially support for buttons for Samsung (and maybe other) devices. This order is a bigger batch of 20 boards to distribute to Trusted Contributors, and should be flying to Europe as you read this blog post!
- The firmware !1 for the PCB to control the device, power it on and off, press buttons, get access to power consumption reads, etc. (#6) has been written. Thanks Martin R. and Casey!
- CI-tron added support for the postmarketOS PCB, and for the OnePlus 6(T). Thanks Martin!
With all these things in place, it is now possible to run tests directly on hardware using our regular developer workflows. For example, whenever changes to the kernel used by the OnePlus 6T are done, CI will build the new kernel, and boot it on the device, to make sure that the device keeps booting!
As the description shows, moving from the previous architectural idea to something real has taken a considerable amount of effort and money. In addition to the countless volunteer hours that everybody has put into this project, the money from OpenCollective has allowed us to pay for hardware and development that has considerably sped up the process of getting here. Overall, the project costs can be summarize as:
- ~1630€ in hardware costs for PCB revisions v0.1, v0.2, v0.3 and associated hardware (cables, chargers, etc.), with some more being expected in the next weeks.
- 10082€ in pure development costs.
- 1285.2€ in project management to coordinate everybody involved, do this communication, provide reports to the team, and the keep the project from stalling.
In total, the cost for the paid work done on this project this year has been around 13 000€, which is close to what we expected in early May.
Looking ahead
With the conclusion of this initial part of the project, we have all the logic in place to do integration testing on any device, and run arbitrary tests for the low-layers of the operating system. Overall, we have a "Minimum Viable Product": something that shows that the approach is correct, and can already deliver some small results.
Moving forward, the goal is to scale-up this project. Our goal is to add more devices and to write tests to cover many other use-cases. Eventually, having devices in a hardware CI should be a requirement for the main category, but should also help make sure that devices stay in a good shape. During FOSDEM and the now-traditional subsequent hackathon we expect to test the whole setup on more devices, and to introduce more people in the team to CI-tron. We also expect to start writing more comprehensive documentation on how to add devices to our farm.
Overall, we expect that as this setup matures we can start to extract more benefits from automated testing. We hope that this can help us reduce the workload on maintainers, while at the same time reducing the number of bugs that make it into releases, overall improving the general reliability of the whole project.
We are extremely happy to have finally reached this stage. It has taken our community many years, multiple attempts, and quite some money to get to this stage. We are happy and thankful for everybody that helped along the way.
If like us, you would like to see postmarketOS become more reliable and appreciate the work we continue doing towards this, consider donating via our OpenCollective!