- PureOS-project - Purism Mailing Lists

Lollypop in byzantium has a bug that should be fixed
by Andreas Ronnquist 23 Sep '21

23 Sep '21

Bullseye is released, and byzantium seems to get closer and closer to being released - congratulations! In the Lollypop package there's a bug in the package in byzantium/bullseye which affects the adaptive mode of the application (which is bad for the phone interface of course). Upstream bug #2778 [1] (fixed in 1.4.19-2 in debian, in upstream in 1.4.20, while 1.4.14-1 is in byzantium/bullseye). It's two commits upstream [2][3] merged to one patch in the debian fix [4]. If I can, I would like to fix this in byzantium - what's the preferred way to do this? Cherry-picking the fix and adding to 1.4.14-1, or simply backporting the latest (or maybe just the first one with the fix) to byzantium? I would gladly provide an updated package. regards -- Andreas Rönnquist mailinglists(a)gusnan.se andreas(a)ronnquist.net [Please don't CC me, if I mail to a mailinglist, I am subscribed to it.] 1: https://gitlab.gnome.org/World/lollypop/-/issues/2778 2: https://gitlab.gnome.org/World/lollypop/-/commit/5e0cbc686e3754a5a3c6e05315… 3: https://gitlab.gnome.org/World/lollypop/-/commit/8cd6117a597be5ee276dca7b99… 4: https://salsa.debian.org/python-team/packages/lollypop/-/commit/bdad22d541d…

3 4

Re: [PureOS] Next steps for Laniakea: The archive problem
by Matthias Klumpp 02 Mar '21

02 Mar '21

Am Di., 2. März 2021 um 12:35 Uhr schrieb Guido Günther <guido.gunther(a)puri.sm>: > > Hi Matthias, > On Mon, Feb 01, 2021 at 08:47:43PM +0100, Matthias Klumpp wrote: > [..snip..] > > So, what do you think? Am I crazy, or is this a good thing to finally attempt? > > Thanks for the explanations, your reasoning makes sense to me. Two > things kept me wondering: > > 1.) Could that be built so we can share more things with other Debian > downstreams to put long term maintenance on broader shoulders? Absolutely, I think Laniakea itself could be extremely useful for other downstreams as well. However, there's one catch: At the moment, Laniakea can theoretically work with Aptly/reprepro/dak etc. for repository management. After this change though, you would pretty much need to use its built-in archive service. I do absolutely think that that's a price worth paying, but it may make Lk harder to adopt. > 2.) How would that help (or delay) other discussed developments in PureOS like > > - being able to block packages from migration by bug reports (as in > Debian) (https://tracker.pureos.net/T872) That one is unrelated to this proposal, so would neither be blocked nor helped by it (even though it may benefit from more accurate database information). > - automatic QA via autopkgtests, reproducibility tests (and > rejection of packages when those fail) The autopkgtests/reproducibility stuff itself wouldn't be impacted, any automation based on that would be easier to implement though (so automatic package rejection etc.), except for things touching migration control, which is unrelated (again). > (https://tracker.pureos.net/T649, https://tracker.pureos.net/T1013) > - rebuilding packages on import > (https://tracker.pureos.net/T1016) The changes would help, but we could actually do this right now as well - *but* rebuilding all packages needs a lot more maintenance from the PureOS team as we would need to handle all transitions on our own. At the moment, we don't have tooling set up for this task to make it easier to achieve. > - building modified packages from git tags > (https://tracker.pureos.net/T1014)) Would be far easier with the changes implemented. > - automatic forward porting of downstream changes > (https://tracker.pureos.net/T1015) That would probably be unrelated, as handled by a separate tool. > and how would that fit in the overall PureOS roadmap. (i filed > bugs for each bullet point above if my search didn't bring one up > already). Many of these features could already be implemented, but would actually be harder to write, because most need some synchronization with dak's state (which is the #1 problem source). My first priority would be to streamline the migration process and also get feature-parity with what we have today. Then, the other features could be implemented on top, or even in parallel in case of the QA bits. > As daily PureOS consumer (and user) i'm curious how it will affect these > developments? Ideally you shouldn't notice anything. To make sure we don't break installations, we would likely need to run a copy of the archive in parallel that's created by the new system, and then at some point make the switch. When that's done, user shouldn't notice any changes, if things are done right, with possibly two exceptions: I'm not planning to implement index diffs or by-hash initially, just to keep the scope of this manageable. So, and apt update would have a bigger download size. On the upside though, we would have a Contents file again for those people who need it, and software.pureos.net would always reflect the current state of the archive exactly. Cheers, Matthias

1 0

Next steps for Laniakea: The archive problem
by Matthias Klumpp 02 Mar '21

02 Mar '21

Hi! I would like to share a part of the plan for future Laniakea improvements which might likely bind a significant amount of my time, but may also provide a huge amount of improvements for the PureOS tooling. What is Laniakea? ------------------------- In order to build a Debian derivative, many tools are needed - besides managing the archive, you also want to do QA, migrate packages between suites, sync packages from Debian, show some web UI about packages, etc. If all tools are separate and don't know of each other, you have to manage them separately, which is a lot of work, but also coordinate them somehow and manage their configuration (e.g. if a suite stabilizes, you need to adjust the new development target 5 times for 5 different tools). Laniakea is a suite of tools which either provides features for archive management on its own, or wraps existing tools and manages their configuration. All tools share one database for configuration which is the single source of truth in the system, and individual Laniakea modules can also communicate directly via a ZeroMQ-based messaging protocol. The Problem ----------------- Laniakea was originally created as part of the Tanglu Debian derivative, and then adapted for PureOS. One of the original ideas of Laniakea was to work with any archive management software, bei it dak (Debian Archive Kit), reprepro, aptly, ... and just wrap it like it wraps any other module. Because that's what it's design was based on: Be very modular and make each module easily replaceable in case something better comes along. This also enabled Laniakea to for a while host two ways to get packages built, while nowadays this has pretty much been collapsed into one singular job management system, which handles package builds as well as image builds. The problem with making the archive manager pluggable is that it goes against Laniakea's principle of having a single source of truth in its database. Currently, only dak is properly supported as archive manager for Laniakea, which keeps its own package database, handles all incoming packages and generates data which Laniakea then imports into its own copy of the archive database. This is terrible for multiple reasons: * Laniakea and dak can - even if just briefly - desynchronize. In reality this makes people wonder which version of a package currently is in what suite, as the web UI does not reflect the actual archive status. * Due to potentially desynchronized states, Laniakea can not trigger certain actions, or can only trigger them in a static sequence of operation when it *knows* that the archive data is currently the same as the data in its database. This results in a lot of archive operations being executed in strict sequences and some changes will not take effect until those were run (like package removals, migrations, syhfornizations, ...) * Laniakea does not know of certain actions taken by dak: For example, if dak rejects a package from upload, Laniakea can only guess that this has happened as there is no direct communication channel that would be race-free. This results in heuristics for the package build system to figure out whether an upload actually reached its destination suite. * The constant interactions between Laniakea and dak are a huge performance hit - Laniakea needs to constantly synchronize its database with the dak data, which can occasionally take more than 20min. That combined with the sequences that Laniakea has to run anyway destroys performance and the "immediateness" that archive actions could have, as everything runs at timed intervals by cron jobs. The Solution ----------------- The solution seems simple: Just implement archive management in Laniakea itself! I did shy away from this for years though. Reasons are that creating the apt archive actually is not an easy task, and a lot of work went into dak for a lot of nice features that would all have to be implemented again. Also, the Debian archive is constantly evolving, and by detaching from dak we would not be able to benefit from them unless we put in work from our side to implement them again. Another reason why this is a bad idea is that it would bind my time, so other features which I consider important would get realized much later. There's also personal reasons for not attempting that rewrite: Dak has some insane SQL to generate indices performantly directly from database tables, and me being no experienced database developer would not help with achieving a better result. However, despite all of these concerns I think I should attempt to write the Laniakea archive module, with the aim to replace dak mid-term. Reasons being the following: * The issues due to interfacing with dak are touching every area of our archive, and it is honestly a pain to work with. Also, 90% of feature requests brought forth by users directly tie in with this issue and could be easily addressed if Laniakea could control the archive directly, and would be the only entity doing so (instead of having some external thing to synchronize with). * Ubuntu's Launchpad is a good example of what can be built if the archive is integrated a bit more tightly with other distribution management features. * Launchpad also uses apt-ftparchive for some metadata read actions. Performance and flexibility is a huge worry for me in an endeavour like this, so I looked into using apt-ftparchive for a "Laniakea archive" prototype. It's not a tool that makes sense to run directly on the archive, but it is great to obtain exactly the metadata we need to build archive index files. Those can be transformed into JSON and stored in Postgres JSON table elements, so we can very easily extend the extra metadata elements in future, if we need to. Also, reading a large quantity of data elements from Postgres tables with SQLAlchemy's ORM works very quickly, and converting and writing them with C++ code is almost instant. Building on top of apt-ftparchive and APT's libapt-pkg also ensures we benefit from improvements made on these tools in Debian. * We don't actually need to support all features initially. Some features we would certainly lose initially are pdiffs, by-hash etc, but a functioning APT archive, without any legacy support but also without any of the very fancy new features, could actually be built in a reasonable amount of time. Of course that would be annoying for some users, but the audience for PureOS would unlikely mind these changes much. And we could gradually add them back later. So, due to that I think it makes sense to spend some serious time to look into implementing this module. I would build on top of apt-pkg and apt-ftparchive, using C++ code where necessary for performance, and Python code everywhere else. I would also make heavy use of json/jsonb entries in Postgres for many of the metadata elements, as it's a convenient, performant but most of all easily extensible way to store most of the auxiliary metadata. For actions such as dependency-resolution I would also use the existing APT library. There are a few open questions on design details that I haven't nailed down, but unlike a year ago, I think a project like this is actually doable and will not take years to complete. But as always with these things, what may look easily achievable now can turn out to be much harder in future ;-) So, what do you think? Am I crazy, or is this a good thing to finally attempt? Cheers, Matthias P.S: To address the obvious question "Why not adjust dak?" - I looked into this, but the amount of changes we would need is quite big, and since there is no chance to merge these upstream, we would essentially be forking dak and still having a separate database and separate-ish tooling. Also, dak serves its own purpose and serves it well for Debian - trying to make it something that it isn't just to work for Laniakea is not a great plan.

2 1

Byzantium versioning
by Guido Günther 15 Dec '20

15 Dec '20

Hi, since we're dropping phone specific package for byzantium we also ought to drop `+librem5.X' in the version numbers. Matthias, what would you suggest here so that we generate higher version numbers in byzantium when backporting a package to both amber and byzantium to be consistent whith the rest of PureOS but also have a higher number than the current <upstream>-<debian-rev)pureos+librem5.<revision> Cheers, -- Guido

3 4

Bits from PureOS | opening the firehose
by Jeremiah C. Foster 04 Dec '20

04 Dec '20

Hello, Some bits from PureOS; 1. Byzantium on all the things 2. End of life for Amber? 3. EFI 1. Due to the fact that there is a new SoC / CPU in the Mini and the Librem 14, we need to move to a new kernel for these new devices. The good news is that the Byzantium kernel appears to fit our needs. The Mini that I have here is a Intel Coffee Lake i7-8565U @ 1.80GHz with 8 Cores. uname -a; "Linux 5.9.0-1-amd64 #1 SMP Debian 5.9.1-1 (2020-10-17) x86_64 GNU/Linux" /etc/os-release; VERSION_ID=10.0 VERSION_CODENAME=byzantium Everything has been working quite smoothly except for some issues around accessing our repos; https://tracker.pureos.net/T960 I have an ISO with the installer that Matthias has fixed up and we hope to have a Go / No Go decision with that image for tomorrow (Wednesday). 2. Amber should likely have a EoL date. I feel like that should be a sort of LTS style approach and say we'll support Amber for two more years. We still sell some products (Librem 15, server) with Amber still on it and enterprises expect longer support so something along those lines seems sensible. Feedback most welcome. 3. Can we deliver an installer that supports EFI? I realize this has been a goal for a while and that there is work still needed to be done for this and we have the usual resource constraints. What more needs to be done? Thank you!

2 3

PureOS Phone images
by Jeremiah C. Foster 12 Sep '20

12 Sep '20

Hi, [Feel free to reply on-list, I just wanted to make sure everyone saw this.] [Please find attached an invite to a mumble call at 16:00 CEST tomorrow.] Welcome to a thread about PureOS images for our Librem 5 phone. Let's start with Guido who knows a thing or two about the Librem 5; "I'd like to (slowly) start filling in enough of byzantium so we can build images for the phone. First candidates would be a kernel and flash-kernel. We currently keep the sources at https://source.puri.sm/Librem5/flash-kernel and https://source.puri.sm/Librem5/linux-next - i just wanted to check that there aren't any objections when the number of packages with patches increases quite a bit." Matthias writes; "First of all, thank you for communicating this early :-) I have no fundamental objections to adding it, you would just need to target "byzantium" or "landing". - Any chance that the kernel changes would break the desktop flavor of PureOS? Or would that be a completely separate kernel package and not merged with the desktop kernel? - You'd need to make sure phone things work on amd64 as well as on arm64 in case they have a convergent base and are used on both form factors. And 3) the PureOS team is tiny so we can't maintain lots and lots of PureOS<->Debian delta. But with the phone team's help I think that's absolutely doable there's also secondary questions, like whether it makes sense to just move shared sources to the pureos group(s) in Gitlab also, I'll have to go through with improving Laniakea's sync overview page to be less cluttered - at the moment it's quite hard to know what needs merging where ;-) (that's actually the thing I am working on at the moment, since that also gets rid of the last bit of D code and makes deploying Laniakea easier). - Laniakea can theoretically also build packages from Git repos - you'll need a list of repos and signed tags of a certain format in them, LK will then verify the tag signature and send the repo to a worker to create a source package out of that's then auto-uploaded. It's quite neat in concept... ....in practice I would need to get rid of the manual list and actually verify signatures (at the moment anything would be built, so that's really not deployable) Anyway, that's $future_stuff, let's work on getting the phone pieces into byzantium easily first :-)" Jeremiah writes; - I would like to understand more about the increase in kernel patches; how will they be maintained? What are the expectations of the Librem 5 team, is there going to be a transfer of responsibility for the patches from the Librem 5 team to PureOS team? Our resources are currently thinly stretched across our CI / CD, QA, and support teams, which means adding devices is a heavy lift that needs co-ordination. We don't really have a process for additional devices yet either and there needs to be some thought as we grow given that our Librem 5 traffic might significantly increase load and bandwidth on existing infrastructure. Do we have any data on; - Currently Librem 5 CI (I believe we run Munin on the ARM 02 server no? Do we have output from that which might show load, disk, CPU, etc? - How about the current instance of Laniakea? Are we running Munin? If not, do we have data on the server from a capacity planning standpoint? - I'd like to know a bit more about the plan - do we intend to use your new process to build from git repos with signed tags Matthias? Or are we going to continue to build on ARM 02 and then upload to Laniakea? Best regards, Jeremiah

3 6

Fwd: Bug#969321: transition: GNOME 3.38 (mutter, evolution-data-server, etc.)
by Jeremiah C. Foster 02 Sep '20

02 Sep '20

Hello! Please find attached an email you may already have been aware of but has a bearing on our work in PureOS. Namely that GNOME 3.38 is likely coming into Debian soon (and then to PureOS Byzantium) and will likely be in Bullseye when that's released. Cheers, Jeremiah

1 0

New PureOS images are ready for testing! (2020-07-18)
by Matthias Klumpp 18 Jul '20

18 Jul '20

Hi! We have new images ready for testing for both PureOS 9 (Amber) and the Byzantium development version. For the former, please give feedback on the updated images on their tracking bug: https://tracker.pureos.net/T919 The Byzantium development images were also generated per-month as usual and can be found at https://downloads.pureos.net/byzantium/ for testing. Cheers, Matthias

1 0

Bits from PureOS | Happy hypoxia averted (so far)
by Jeremiah C. Foster 03 Jul '20

03 Jul '20

Hello all! Admit it, you've missed this mailing list haven't you? :-) Firstly, I wanted to ask a question about backporting a 5.6 kernel for Amber. This would be used on the new mini that's been announced. ( https://shop.puri.sm/shop/librem-mini/) While the currently Amber kernel 4.19 works, according to Mr. Chromebox there are a couple of issues; - 4.19 doesn't have great support for the Comet Lake chipset - Bringing up a coreboot port with the mini, one can't use PureOS as it's not bootable from the vendor UEFI firmware What is the easiest way to create a bootable OEM image for Mr. Chromebox to boot the new mini? My guess is that we ought to be able to just build a Byzantium image and things should "just work". I say that because Debian 10 apparently supports EFI - https://wiki.debian.org/UEFI. Can we build an OEM image for Byzantium? How is this done for amber, is there a particular script I can use or a set of steps we use to do this? We ought to likely document this process for posterity. :-) Getting a new image for the factory floor will help a lot, we have more than minis in the pipeline as you may have seen. More to come, specifically some notes about buildstream and reprotest as well as variables for gitlab-ci.yml. Regards, Jeremiah

1 0

Fwd: [AppStream] AppStream 0.12.11 released!
by Jeremiah C. Foster 12 May '20

12 May '20

Relevant for our interests. -- -- Jeremiah C. Foster <jeremiah.foster(a)puri.sm> Gpg key id: 798DB834436A7BE38C97422D0DC062205451931B

2 1