- PureOS-project - Purism Mailing Lists

Re: [PureOS] Laniakea Python version is now online!
by Matthias Klumpp 21 May '19

21 May '19

Am Mo., 20. Mai 2019 um 19:44 Uhr schrieb Chris Lamb <chris.lamb(a)puri.sm>: > > Matthias Klumpp wrote: > > > > > Since last week, all of PureOS infrastructure is now switched to the > > > > latest, mostly Python-based version of Laniakea[1]. > > > > > > This is great news. Congratulations Matthias! > > Indeed, congratulations and thank you for posting about it here too. > My apologies for this late reply getting lost between various cracks > in my inbox. :( Don't worry, there is no expiry date on replies ;-) > > > If so, perhaps we use a separate Laniakea channel for Matrix so as > > > not to flood the PureOS channel. > > I completely +1 the idea of a #pureos-changes channel. > > I use this pattern in many projects; having that "soft" visibility is > extremely helpful for all sorts of reasons (both technical and non- > technical) in terms of encouraging more of a team feel, avoiding any > feeling of being isolated from others. Indeed, I even wrote the > #debian-devel- changes bot back in 2008 for these very reasons and it > has been remarkably successful in these regards. Yes, we had this in Tanglu as well and it was really useful :-) > Can I thus resubmit my request for this to be re-prioritised? :) It's definitely a high priority, but giving people access to easy information about the archive is even more important I would say. That being said, software.pureos.net is available again now, so you can view build logs easily again. Next I will also re-add the debcheck and migration information to the package details pages, as that's important as well for diagnosing problems. Then maybe also refine the pages to sort by suite as well, and also improve the application/component search a bit (as that's the neat thing our users may be interested to use, and it's easy to do). Than I would actually be moving on to reinstate the messaging system between Laniakea components, however I don't feel at ease doing that without a proper testing framework in place, so that also has a slightly higher priority. But then, the next thing definitely is the message bus, Matrix bot and possibly Fedmsg bridge. > Oh, one adjunct question; how can we contribute/learn more about this > codebase? Whilst justifiable for many reasons, as it is on Github it > definitely feels a little separate from the rest of the PureOS code. > Can we ameliorate this somehow? I don't have any concrete suggestions > at this time, alas. On why it's on a different platform: That is simply because the project started outside of PureOS before I joined Purism, and was for a while shared between two derivatives. Nowadays I actually prefer it to be not as tightly coupled to the PureOS project, because I hope that as soon as it's mature enough more derivatives may consider using it. That's very often easier when the infrastructure for a project is at a more neutral place. The README file of the project as well as some PureOS wiki pages definitely tell people who want to know who was responsible for moving the project forward though. On contributing: There is outdated documentation on https://lkorigin.github.io/ which I should update soon to reflect what the Python version has to offer (and to elaborate on some points). The code is also documented, but generating some API documentation from it would also be helpful, I guess. I also gave a talk about the project at Debconf, but that's probably not a good starting point. The talk slides however may be useful, I'd have to modernize them as well though, just like the rest of the documentation. To build the project itself, one can use - rather unusual for a Python project - Meson. Meson/ninja will build the binary parts and then install all pieces into the right location, then one can use the `lk-admin` tool to initialize the database and configure the system. Currently, to make Laniakea really useful, one needs to combine it with either dak or reprepro for archive management. I'll certainly answer any questions anyone trying to understand any of the components may have, and hopefully also write a nice quickstart guide. The problem to get all the tasks done is - as always - time ;-) Cheers, Matthias

2 1

Laniakea Python version is now online!
by Matthias Klumpp 20 May '19

20 May '19

Hello! Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1]. ## What is Laniakea? Laniakea is a suite of tools and adapters which originated in the Tanglu Debian derivative. Its purpose is to bring together data and configuration from the multitude of separate tools that are required to run a Debian derivative, as well as adding missing functionality in an integrated way. Tools which are wrapped by Laniakea include Britney, dose3, germinator, dak and more. Due to Laniakea's database being the authoritative source of information for a derivative, there is always only one place to change configuration, individual configuration for the wrapped tools is automatically generated from database information. Also, the centralization of data in a database and well-defined directory structure allows Laniakea modules to cross-reference information and make decisions based on data from other tools, e.g. the debcheck information is used to determine whether a package is actually buildable in the distribution. Using the database, all information Laniakea knows about is accessible (currently read-only) via modern web interfaces. Laniakea also contains a global worker pool to execute arbitrary long-running tasks. These tasks currently are image builds and package builds. All tasks are run on separate machines in systemd-nspawn containers via debspawn[2], for build tasks static-analysis reports are generated in the Firehose XML format[3]. Laniakea also contains a message-bus based on ZeroMQ and JSON to let modules running on other machines know about changes and react to events when running in a more isolated environment. This also allows notifications via email or Matrix. ## What changed now? Laniakea previously was written entirely in the D programming language. Since the project was a fun experiment in Tanglu, using the language also wasn't much of an issue. With our new requirements in PureOS and limited manpower though, it became apparent that the choice of D was hindering progress in this case, simply due of D's much smaller ecosystem of existing tools and libraries. Time spent at improving the database connection code in D could be better spent at implementing a new feature we need. So, Laniakea was mostly ported to Python 3.6+ with some code still remaining in D. Since D has great bindings to Python, writing a module to run existing D code from Python was very easy. This also allows us to reuse a lot of the more complex logic in Laniakea without changes (and the chance to introduce new bugs). Porting progress can be followed via this document[4]. Since Monday last week, PureOS' infrastructure is now running on the Python-ified version of Laniakea, removing the extra maintenance burden of developing two versions of the same software in parallel. ## How does the change affect me? Ideally, a user or developer of PureOS should not see any changes. The new implementation should work just as the previous one. The web UI on master.pureos.net looks slightly different, but that is it. Since the new Laniakea database has no indices yet, certain queries could take a bit longer. One notable change is the absence of software.pureos.net - this module could not be easily ported and needs to be rewritten entirely. I am working on that, and we will definitely get it back, better than before. The Laniakea message piblishing code also has not been ported yet, so modules can not communicate with each other via ZeroMQ. This should not ceuase any issues, but when it is finally implemented, we will get a new Matrix bot to notify about archive changes immediately. ## What's next? ### Create an integration test framework for Laniakea We need better testing, or rather, any testing for some Laniakea modules and larger pieces of the codebase. Since a large chunk of Laniakea is interfacing with existing code or with its own modules, we need some way to do intergration testing properly. ### More unit testing! The Python code doesn't have many tests written yet, so this needs to be added as well. ### Bring back software.po.n (webswview) The web application just needs to be rewritten. This is a work in progress at time. ### Reimplement Laniakea messaging The Lighthouse module of Laniakea needs to get its message publication support back. I intend to make use of signedjson[5] for message signing this time. When implementing this feature, the very crude current Lighthouse implementation also needs a complete overhaul. Additionally, we can write a Matrix bot for Laniakea again at this stage. ### Better dak integration The Debian Archive Kit (dak) and Laniakea basically exist alongside each other, not interacting all that much. That should be changed to make dak and Laniakea talk to each other better to reduce the reliance on cronjobs. ### Better performance There is a lot of potential for performance improvements in Laniakea, most notably in the module that imports archive data into the database. Making these improvements will allow for many nice things, like faster package processing. ### Package for Debian Laniakea should be available from withing Debian. This is a long-term goal as it requires at least some stability of the database schemas and more and better testing. Also, the JavaScript dependencies of the web modules need to be sorted out properly. Ultimately though, at some point you can just apt install Laniakea to create your own Debian derivative. ## What can I do? Contribute! Either by writing code for Laniakea (documentation is currently updated to reflect the Python changes) or filing bugs. If you see anything weird happening in PureOS infrastructure that hasn't happened before, please file a bug in the PureOS bugtracker[6] or against Laniakea directly if you know it is to blame already. Happy hacking! Matthias [1]: https://github.com/lkorigin/laniakea [2]: https://github.com/lkorigin/debspawn [3]: https://github.com/fedora-static-analysis/firehose [4]: https://github.com/lkorigin/laniakea/blob/master/docs/porting-status.md [5]: https://github.com/matrix-org/python-signedjson [6]: https://tracker.pureos.net/

3 3

Updates on reproducible builds in PureOS
by Jeremiah C. Foster 18 May '19

18 May '19

Hi, I've received and email recently from the folks at the FSF about PureOS reproducible builds. When we were at LibrePlanet was asked if the FSF would be kind enough to 'verify' our reproducible builds and they replied via mail saying; > Yes, we are interested in doing this. Generally, we will want > the build and verification process to be publicly documented > and mostly automated for an initial setup To which I replied, here is how we build our ISOs; https://tracker.pureos.net/w/development/howto_build/ I've set up debspawn to build in ISO from cron as well and I'll automated diffoscope to create and html report for us and the FSF to review. I also have created this document to explain a bit about reproducible builds in PureOS. This document pertains to PureOS and it refers to upstream source to find out more about the upstream project. https://tracker.pureos.net/w/development/reproducible_isos/ Currently the work being done is around automation; - having image building being done by cron - running diffoscope The ISOs that get built today are around 1.5 gigs which makes me wonder if they're too large to put in Debian's Jenkins machine lest they fill up disk. Happy to hear feedback on all the above. Regards, Jeremiah

2 1

Please assume all issues require tracking unless told otherwise
by Jonas Smedegaard 02 May '19

02 May '19

[ posting again, to correct non-lists address this time ] Hi all, There seems to be confusion about which issues need tracking and which are adequate to just discuss casually on Matrix. If in doubt, please read the topic line of the matrix room you're in. PureOS matrix room currently offers this guidance: > Create new issue at https://tracker.pureos.net/ (or locate and add to > existing issue) for all the things unless explicitly told it needs no > tracking or followup Let's please help each other remember to track all that needs tracking. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private _______________________________________________ Pureos-project mailing list Pureos-project(a)lists.puri.sm https://lists.puri.sm/listinfo/pureos-project

1 0

Please assume all issues require tracking unless told otherwise
by Jonas Smedegaard 02 May '19

02 May '19

Hi all, There seems to be confusion about which issues need tracking and which are adequate to just discuss casually on Matrix. If in doubt, please read the topic line of the matrix room you're in. PureOS matrix room currently offers this guidance: > Create new issue at https://tracker.pureos.net/ (or locate and add to > existing issue) for all the things unless explicitly told it needs no > tracking or followup Let's please help each other remember to track all that needs tracking. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private

1 0

Let's stablize PureoS Green
by François Téchené 22 Mar '19

22 Mar '19

Hi, I am joining this discussion to expose the vision from the UI/UX design side of PureOS. In term of UI/UX design, we are conforming to the Ethical Design manifesto (https://2017.ind.ie/ethical-design/) . Based on that we have defined a few guidelines that we want to make sure are respected. - One of them is that we target the majority of the population in term of user experience. This means that we should think in term of "democratic" choice. The question we keep asking ourselves is : Will the defaults work best for the majority of human beings? The idea here is to avoid proposing different flavors for different tastes but to focus on defaults that targets the majority. - Another guideline we would like to be respected is that we should contribute upstream. No matter which project we want to adopt, we shouldn't put our effort in forking it but instead in joining the community in that global effort for the sake of everyone. In that regard, we are currently contributing to the GNOME project as the default PureOS DE on both the laptop and the phone. We have been working for the past year, along with the GNOME design team in designing the GNOME mobile interfaces as well as bringing convergence to the desktop (laptop) platform. This is a work in progress that is being implemented by both the Purism devs and the community. Some of that work has already been implemented as part of GNOME 3.32. My point here is that GNOME, as a whole, along with GTK+ will dramatically evolve in the next few years in order match the user experience that we are designing and developing for the Librem products. Our goal is to define a tight integration and consistent experience between Librem Laptops, Librem Phone, PureOS, Librem One and the Librem Key. We need to experience the experience that we design and that the devs are implementing, as well as delivering to the people in a not too long frequency. So if we decide to go for a stable long term release of the core OS, one solution for us would be : - To have a "dev" PureOS repository that would let us test the latest GNOME and GTK updates (along with the stable packages of the core system in order to test the stability of a release within the current core packages?) - To be able to release GNOME upgrades to our users in a not too long frequency (maximum 1 year) ideally the 6 months GNOME release cycle. I think this is a critical time for us in term of user experience so we must not rush in the decisions and make sure we find a good compromise. Thank you! -- https://puri.sm

6 26

Building my own PureOS ISO image?
by Chris Lamb 19 Mar '19

19 Mar '19

Hi pureos-project, I'm trying to build a PureOS ISO image on my own system but can't seem to locate a script/repo/documentation, etc. I'm probably just missing something obvious. To clarify, I'm trying to build an ISO like: https://www.pureos.net/download/ ... rather than the OEM image. I'm guess Matthias is the person to specifically ask here? (Ping?) Just to underline it really doesn't matter how ugly/manual the process is or that it could potentially be improved — everything always can be and I won't "judge" anyone on the status quo, especially for non-daily processes; I'd just like to build one myself. Best wishes, -- Chris Lamb https://puri.sm

4 20

Let's stablize PureoS Green
by Jeremiah C. Foster 18 Mar '19

18 Mar '19

Hi, In the few months I've been with Purism, I've been astonished by the ability of the team to build PureOS with the resources at hand. One of the reasons for this is likely the use of Laniakea as CI but of course the bulk of the reason is the quality of the base distro and the experience of those who're developing PureOS. There are naturally some pain points that come with new products like the Purism Librem laptops. I think that some of these rough edges might be smoothed by following Debian Testing as it becomes Debian Stable and keeping Debian Stable as our "Green" repository for PureOS packages. I've floated this idea internally in Purism and there has been a great deal of support for the idea. Others have also identified Debian Testing as being too fast moving for the Librem. While I mentioned this in another thread, I'd like to use this thread for discussion on the idea's merits as well as what is required technically to do this. What do those folks on this mailing list think? Should we keep PureOS Green on Debian (Buster) Stable? Thanks, Jeremiah

5 8

Buster Freeze
by Jeremiah C. Foster 14 Mar '19

14 Mar '19

https://release.debian.org/buster/freeze_policy.html So Buster is Frozen today if I'm not mistaken. Is there anything we ought to do in our Laniakea for us to track the Freeze? My assumption is that we'll see fewer updates as fewer packages will be accepted into testing. Also, if Debian removes packages from testing, will those get remove from PureOS? I'm thinking about things like gksu, do we have to manually remove packages that are removed from Debian? Regards, Jeremiah

3 7

Re: [PureOS] Building my own PureOS ISO image?
by Jeremiah C. Foster 14 Mar '19

14 Mar '19

On Mon, 2019-03-11 at 19:50 +0100, Matthias Klumpp wrote: > Am Mo., 11. März 2019 um 18:47 Uhr schrieb Jeremiah C. Foster > <jeremiah.foster(a)puri.sm>: > > [...] > > > > Re-reading this question the answer should be "no". I'm running > > Clear > > Linux on the host, I boot into PureOS via systemd-nspawn. So PureOS > > is > > running in a container. > > Ah! So, you are running debspawn in a PureOS container created with > systemd-nspawn? Yes, exactly. > That's probably not a good idea, because it means you are nesting > containers, as debspawn will create another ephemeral container on > its > own as well. I think this is where the problem is as well. pagemap file appears to map process memory usage from virtual to physical. I guess some process is filling it up. Having a container on a container on a host makes the debugging process a bit difficult. > So I guess this behavior might be some kind of bug related to nesting > containers (I am actually very surprised that nspawn lets you do that > - I never tried that but always thought that it wouldn't work anyway. > > Cheers, > Matthias > > P.S: Debspawn will probably work on any distribution that has > debootstrap, but I don't think anyone has tried that yet. debootstrap is on my Clear Linux system. I used it to create the PureOS container and for various Debian chroots. The machine itself is quite powerful; 8 core i7-8705G CPU @ 3.10GHz with hyperthreading. It makes for a speedy build machine and I like the Clear Linux use of OStree, but I think I need to move the machine to Debian or, alternatively, use the other build script. There is no substantive difference between the two build approaches is there, I mean aside from environment. Cheers, Jeremiah > --- > I welcome VSRE emails. See http://vsre.info/ <-- Interesting. :-)

2 2