Hello! Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1]. ## What is Laniakea? Laniakea is a suite of tools and adapters which originated in the Tanglu Debian derivative. Its purpose is to bring together data and configuration from the multitude of separate tools that are required to run a Debian derivative, as well as adding missing functionality in an integrated way. Tools which are wrapped by Laniakea include Britney, dose3, germinator, dak and more. Due to Laniakea's database being the authoritative source of information for a derivative, there is always only one place to change configuration, individual configuration for the wrapped tools is automatically generated from database information. Also, the centralization of data in a database and well-defined directory structure allows Laniakea modules to cross-reference information and make decisions based on data from other tools, e.g. the debcheck information is used to determine whether a package is actually buildable in the distribution. Using the database, all information Laniakea knows about is accessible (currently read-only) via modern web interfaces. Laniakea also contains a global worker pool to execute arbitrary long-running tasks. These tasks currently are image builds and package builds. All tasks are run on separate machines in systemd-nspawn containers via debspawn[2], for build tasks static-analysis reports are generated in the Firehose XML format[3]. Laniakea also contains a message-bus based on ZeroMQ and JSON to let modules running on other machines know about changes and react to events when running in a more isolated environment. This also allows notifications via email or Matrix. ## What changed now? Laniakea previously was written entirely in the D programming language. Since the project was a fun experiment in Tanglu, using the language also wasn't much of an issue. With our new requirements in PureOS and limited manpower though, it became apparent that the choice of D was hindering progress in this case, simply due of D's much smaller ecosystem of existing tools and libraries. Time spent at improving the database connection code in D could be better spent at implementing a new feature we need. So, Laniakea was mostly ported to Python 3.6+ with some code still remaining in D. Since D has great bindings to Python, writing a module to run existing D code from Python was very easy. This also allows us to reuse a lot of the more complex logic in Laniakea without changes (and the chance to introduce new bugs). Porting progress can be followed via this document[4]. Since Monday last week, PureOS' infrastructure is now running on the Python-ified version of Laniakea, removing the extra maintenance burden of developing two versions of the same software in parallel. ## How does the change affect me? Ideally, a user or developer of PureOS should not see any changes. The new implementation should work just as the previous one. The web UI on master.pureos.net looks slightly different, but that is it. Since the new Laniakea database has no indices yet, certain queries could take a bit longer. One notable change is the absence of software.pureos.net - this module could not be easily ported and needs to be rewritten entirely. I am working on that, and we will definitely get it back, better than before. The Laniakea message piblishing code also has not been ported yet, so modules can not communicate with each other via ZeroMQ. This should not ceuase any issues, but when it is finally implemented, we will get a new Matrix bot to notify about archive changes immediately. ## What's next? ### Create an integration test framework for Laniakea We need better testing, or rather, any testing for some Laniakea modules and larger pieces of the codebase. Since a large chunk of Laniakea is interfacing with existing code or with its own modules, we need some way to do intergration testing properly. ### More unit testing! The Python code doesn't have many tests written yet, so this needs to be added as well. ### Bring back software.po.n (webswview) The web application just needs to be rewritten. This is a work in progress at time. ### Reimplement Laniakea messaging The Lighthouse module of Laniakea needs to get its message publication support back. I intend to make use of signedjson[5] for message signing this time. When implementing this feature, the very crude current Lighthouse implementation also needs a complete overhaul. Additionally, we can write a Matrix bot for Laniakea again at this stage. ### Better dak integration The Debian Archive Kit (dak) and Laniakea basically exist alongside each other, not interacting all that much. That should be changed to make dak and Laniakea talk to each other better to reduce the reliance on cronjobs. ### Better performance There is a lot of potential for performance improvements in Laniakea, most notably in the module that imports archive data into the database. Making these improvements will allow for many nice things, like faster package processing. ### Package for Debian Laniakea should be available from withing Debian. This is a long-term goal as it requires at least some stability of the database schemas and more and better testing. Also, the JavaScript dependencies of the web modules need to be sorted out properly. Ultimately though, at some point you can just apt install Laniakea to create your own Debian derivative. ## What can I do? Contribute! Either by writing code for Laniakea (documentation is currently updated to reflect the Python changes) or filing bugs. If you see anything weird happening in PureOS infrastructure that hasn't happened before, please file a bug in the PureOS bugtracker[6] or against Laniakea directly if you know it is to blame already. Happy hacking! Matthias [1]: https://github.com/lkorigin/laniakea [2]: https://github.com/lkorigin/debspawn [3]: https://github.com/fedora-static-analysis/firehose [4]: https://github.com/lkorigin/laniakea/blob/master/docs/porting-status.md [5]: https://github.com/matrix-org/python-signedjson [6]: https://tracker.pureos.net/