- PureOS-project - Purism Mailing Lists

Bits from PureOS | Life in the time of novel corona virus
by Jeremiah C. Foster 20 Mar '20

20 Mar '20

Hi everyone, A somber note today mostly to try to express solidarity for everyone on the list and anyone involved who may be going through hard times. Let me know personally if there's anything I can do. With regard to PureOS it is onward and upward. Matthias has been working on integrating flatpak support into Laniakea. This is a big deal and I'm really excited to see it. We've been working on a CI pipeline that builds flatpaks and then they get uploaded to the CD system, but that has been problematic because a lot of the technolgy is new and it's shifted around a lot. Flatpaks also do a sort of nested virtualization or nested isolation and that can be difficult to handle with many of the tools we use. Having some native functionality in Laniakea will be very helpful. I've been doing some poking at OpenQA from the SuSE folks. It is not packaged for Debian (though it's core package os-autoinst is). Trying to get a log in to Debian's instance of OpenQA https://openqa.debian.net/ leads to some weird site called microfocus.com. I wonder what relationship that holds to Debian's infrastructure? I'll have to dig a little more but I believe that having PureOS tested there might be a good thing. Kali is tested there as well so it may save us some disk space and CPU time to have it done there. We'll need to make sure that it meets the Free Software policies we have and while OpenQA itself surely does (its GPLv2), we shouldn't have to give info to a third party to access testing services. The RYF application for the Librem Key is still in process. We've updated the source code and it's README and nearly completed our response to the FSF so hopefully we'll receive some good news when we resubmit. That's it for now! Stay well and we'll get through this together! Jeremiah PS - I really mean it when I say you can reach out to me personally, I'm happy to talk to anyone on this list about PureOS, Purism, or just personal stuff. -- Jeremiah C. Foster <jeremiah.foster(a)puri.sm> Gpg key id: 798DB834436A7BE38C97422D0DC062205451931B

2 2

January 2020 in Reproducible Builds
by Chris Lamb 10 Feb '20

10 Feb '20

==================================================================== o ⬋ ⬊ January 2020 in Reproducible Builds o o ⬊ ⬋ https://reproducible-builds.org/reports/2020-01/ o ==================================================================== Welcome to the January 2020 report from the Reproducible Builds project. In this month's issue, we cover: * Upstream news & event coverage — Reproducing the Telegram messenger, etc. * Software development — Updates and improvements to our tooling * Distribution work — More work in Debian, openSUSE & friends * Misc news — From our mailing list & how to get in touch (Q: But what are reproducible builds...?) Whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. If you are interested in contributing, please visit the Contribute [1] page on our website. [1] https://reproducible-buids.org/contribute/ Upstream news & event coverage ============================== The Telegram [2] messaging application has documented full instructions [3] for verifying that its original source code is exactly the same code that is used to build the versions available on the Apple App Store and Google Play. [2] https://telegram.org/ [3] https://core.telegram.org/reproducible-builds Reproducible builds were mentioned in a panel on Software Distribution with Sam Hartman, Richard Fontana, & Eben Moglen [4] at the Software Freedom Law Center's 15h Anniversary Fall Conference [6] (at ~35m21s). [4] https://www.youtube.com/watch?v=rMinFopJMW0&t=2121s [6] https://www.softwarefreedom.org/events/2019/annual-conference/ Vagrant Cascadian will present a talk at SCALE 18x [7] in Pasadena, California on March 8th titled "There and Back Again, Reproducibly" [8]. [7] https://www.socallinuxexpo.org/scale/18x [8] https://www.socallinuxexpo.org/scale/18x/presentations/there-and-back-again… Matt Graeber [9] (@mattifestation) posted on Twitter that: > If you weren't aware of the reason Portable Executable [11] timestamps > in Win 10 binaries were nonsensical, Raymond's post explains the reason: > to support reproducible builds. ... referencing an article by Raymond Chen from January 2018 [12] which, amongst other things, mentions: > One of the changes to the Windows engineering system begun in Windows > 10 is the move toward reproducible builds. [9] https://medium.com/@mattifestation [11] https://en.wikipedia.org/wiki/Portable_Executable [12] https://devblogs.microsoft.com/oldnewthing/20180103-00/?p=97705 Jan Nieuwenhuizen announced the release of GNU Mes 0.22 [13] which produced a bit-for-bit identical "mescc-mes-static" binary with the mes-rb5 package within the GNU Guix operating system. Vagrant Cascadian subsequently uploaded this version of Mes to Debian. [13] https://lists.reproducible-builds.org/pipermail/rb-general/2020-January/001… Software development ==================== diffoscope ---------- diffoscope [17] is our in-depth and content-aware diff-like utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure [18] and is essential for identifying fixes and causes of nondeterministic behaviour. [17] https://diffoscope.org [18] https://tests.reproducible-builds.org/debian/reproducible.html This month, diffoscope versions 135 and 136 were uploaded to Debian unstable by Chris Lamb. He also made the following changes to diffoscope itself, including: * New features: * Support external difference tools such as Meld [19], etc. similar to git-difftool(1). [20] * Extract resources.arsc files as well as classes.dex from Android .apk files to ensure that we show differences there. [21] * Fallback to the regular .zip container format for .apk files if apktool is not available. [22][23][24][25] * Drop --max-report-size-child and --max-diff-block-lines- parent; scheduled for removal in January 2018. [26] * Append a comment to a difference if we fallback to a less- informative container format but we are missing a tool. [27][28] * Bug fixes: * No longer raise a KeyError exception if we request an invalid member from a directory container. [29] * Documentation/workflow improvements: * Clarify that "install X" in various outputs actually refers to system-level packages. [30] * Add a note to the Contributing documentation to suggest enable concurrency when running the tests locally. [31] * Include the CONTRIBUTING.md file in the PyPI.org [32] release. [33][34] * Logging improvements: * Log a debug-level message if we cannot open a file as container due to a missing tool to assist in diagnosing issues. [35] * Correct a debug message related to compare_meta calls to quote the arguments correctly. [36] * Add the current PATH environment variable to the "Normalising locale..." debug-level message. [37] * Print the Starting diffoscope $VERSION line as the first line of the log as we are, well, starting diffoscope. [38] * If we don't know the HTML output name, don't emit an enigmatically truncated HTML output for debug message. [39] * Tests: * Don't exhaustively output the entire HTML report when testing the regression for #875281; parsing the JSON and pruning the tree should be enough. [41] * Refresh and update the fixtures for the .ico tests to match the latest version of Imagemagick [42] in Debian unstable. [43] * Code improvements: * Add a .git-blame-ignore-revs file to improve the output of git-blame(1) by ignoring large changes when introducing the Black [44] source code reformatter and update the CONTRIBUTING.md guide on how to optionally use it locally. [45] * Add a noqa line to avoid a false-positive Flake8 [46] "unused import" warning. [47] * Move logo.svg to under the doc/ directory [48] and make setup.py executable [49]. * Tidy diffoscope.main's configure method. [50][51][52][53] * Drop an assertion that is guaranteed by parallel if conditional [54] and an unused "Difference" import from the APK comparator. [55] * Turn down the volume for a recommendation in a comment. [56] * Rename the diffoscope.locale module to diffoscope.environ as we are modifying things beyond just the locale (eg. calling tzset [57], etc.) [58] * Factor-out the generation of foo not available in path comment messages into the exception that raises them [59] and factor out running all of our many zipinfo into a new method [60]. * trydiffoscope [61] is the web-based version of diffoscope. This month, Chris Lamb fixed the PyPI.org [62] release by adding the trydiffoscope script itself to the MANIFEST file and performing another release cycle. [63] In addition, Marc Herbert adjusted the cbfstool [64] tests to search for expected keywords in the output, rather than specific output [65], fixed a misplaced debugging line [66] and added a "Testing" section to the CONTRIBUTING.rst [67] file. Vagrant Cascadian updated to diffoscope 135 [68] in GNU Guix [69]. [19] https://meldmerge.org/ [20] https://salsa.debian.org/reproducible-builds/diffoscope/issues/87 [21] https://salsa.debian.org/reproducible-builds/diffoscope/issues/27 [22] https://salsa.debian.org/reproducible-builds/diffoscope/commit/bbbb322 [23] https://salsa.debian.org/reproducible-builds/diffoscope/commit/6710868 [24] https://salsa.debian.org/reproducible-builds/diffoscope/commit/85c2100 [25] https://salsa.debian.org/reproducible-builds/diffoscope/commit/b1b9977 [26] https://salsa.debian.org/reproducible-builds/diffoscope/commit/fee25e5 [27] https://salsa.debian.org/reproducible-builds/diffoscope/commit/0615036 [28] https://salsa.debian.org/reproducible-builds/diffoscope/commit/88bc142 [29] https://salsa.debian.org/reproducible-builds/diffoscope/commit/7765669 [30] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c99cd9b [31] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9d88cda [32] https://pypi.org/ [33] https://salsa.debian.org/reproducible-builds/diffoscope/commit/0e5a0c5 [34] https://salsa.debian.org/reproducible-builds/diffoscope/commit/aa9db34 [35] https://salsa.debian.org/reproducible-builds/diffoscope/commit/5757c8b [36] https://salsa.debian.org/reproducible-builds/diffoscope/commit/3e3a18d [37] https://salsa.debian.org/reproducible-builds/diffoscope/commit/600831b [38] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9a200bc [39] https://salsa.debian.org/reproducible-builds/diffoscope/commit/b709d89 [41] https://salsa.debian.org/reproducible-builds/diffoscope/issues/84 [42] https://imagemagick.org/ [43] https://salsa.debian.org/reproducible-builds/diffoscope/commit/614b0d1 [44] https://black.readthedocs.io/en/stable/ [45] https://salsa.debian.org/reproducible-builds/diffoscope/commit/4e40f75 [46] https://flake8.pycqa.org/en/latest/ [47] https://salsa.debian.org/reproducible-builds/diffoscope/commit/8993168 [48] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f8f0d14 [49] https://salsa.debian.org/reproducible-builds/diffoscope/commit/efb7bdc [50] https://salsa.debian.org/reproducible-builds/diffoscope/commit/d377d3b [51] https://salsa.debian.org/reproducible-builds/diffoscope/commit/d2e283c [52] https://salsa.debian.org/reproducible-builds/diffoscope/commit/d658583 [53] https://salsa.debian.org/reproducible-builds/diffoscope/commit/8eb852a [54] https://salsa.debian.org/reproducible-builds/diffoscope/commit/722d735 [55] https://salsa.debian.org/reproducible-builds/diffoscope/commit/026ff74 [56] https://salsa.debian.org/reproducible-builds/diffoscope/commit/412fef1 [57] https://docs.python.org/3/library/time.html [58] https://salsa.debian.org/reproducible-builds/diffoscope/commit/786ff48 [59] https://salsa.debian.org/reproducible-builds/diffoscope/commit/3a5f46d [60] https://salsa.debian.org/reproducible-builds/diffoscope/commit/3b94c0c [61] https://try.diffoscope.org [62] https://pypi.org/ [63] https://salsa.debian.org/reproducible-builds/trydiffoscope/issues/1 [64] https://www.coreboot.org/CBFS [65] https://salsa.debian.org/reproducible-builds/diffoscope/commit/d59dd2a [66] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c150ae8 [67] https://salsa.debian.org/reproducible-builds/diffoscope/commit/0beddf1 [68] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=a3bf43481902ff10d0293… [69] https://guix.gnu.org/ reprotest --------- reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, versions 0.7.11 and 0.7.12 were uploaded to Debian unstable [70] by Holger Levsen. This month, Iñaki Malerba improved the version test to split on the + character [71] and Ross Vandegrift updated the code to allow the user to override timeouts from the surrounding environment [72]. Holger Levsen also made the following additionally changes: * Drop the short timeout and use the install timeout instead. (#897442) * Use "real" reStructuredText [74] comments instead of using the raw directive. [75] * Update the PyPI [76] classifier to express we are using Python 3.7 now. [77] [70] https://tracker.debian.org/pkg/reprotest [71] https://salsa.debian.org/reproducible-builds/reprotest/commit/dc67985 [72] https://salsa.debian.org/reproducible-builds/reprotest/commit/7f645f9 [74] https://docutils.sourceforge.io/rst.html [75] https://salsa.debian.org/reproducible-builds/reprotest/commit/9eb1017 [76] https://pypi.org/ [77] https://salsa.debian.org/reproducible-builds/reprotest/commit/19c6d5a Other tools ----------- * disorderfs is our FUSE [78]-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues. This month, Chris Lamb fixed an issue by ignoring the return values of fsyncdir to ensure (for example) dpkg(1) can "flush" /var/lib/dpkg correctly [79] and merged a change from Helmut Grohne to use the build architecture's version of pkg-config [80] to permit cross- architecture builds [81]. * strip-nondeterminism is our tool to remove specific non- deterministic results from a completed build. This month, version 1.6.3-2 was uploaded to Debian unstable [82] by Holger Levsen to bump the Standards-Version. [78] https://en.wikipedia.org/wiki/Filesystem_in_Userspace [79] https://salsa.debian.org/reproducible-builds/disorderfs/commit/07c6e80 [80] https://www.freedesktop.org/wiki/Software/pkg-config/ [81] https://salsa.debian.org/reproducible-builds/disorderfs/commit/520937a [82] https://tracker.debian.org/news/1095745/accepted-strip-nondeterminism-163-2… Upstream development -------------------- The Reproducible Builds project detects, dissects and attempts to fix as many unreproducible packages as possible. Naturally, we endeavour to send all of our patches upstream. This month, we wrote another large number of such patches, including: * Arnout Engelen (for the NixOS distribution [83]): * bash [84] (enable PGRP_PIPE regardless of build-time kernel version) * jitterentropy [85] (remove timestamps from Gzip[86]-compressed manpages, already filed upstream [87]) * ms-sys [88] (remove timestamps from .gz manpages, already upstream [89]) * Bernhard M. Wiedemann (for the openSUSE [90] distribution): * ImageMagick [91] (toolchain, .png date) * brickv [92] (sort a Python glob/readdir(3)) * cpython [93] (.pyc reproducibility) * doxygen [94] (merged a toolchain patch to prevent nondeterminism from ASLR) * fastjet-contrib [95] (sort find / readdir) * openjfx [96] (Java date) * ruby [97] (Reopen unsorted Ruby glob issue) * rubygem-sassc [98] (sort a Ruby readdir(3)) * Chris Lamb: * #948279 filed against python-gmusicapi. * #948582 filed against bochs. * #948872 filed against pcbasic. * #949379 filed against vmatch. * #949580 filed against pkg-js-tools. * #949684 filed against mcomix. * #949817 filed against shotcut (forwarded upstream [113]). * #950138 filed against pikepdf (forwarded upstream [116]). * Jelle van der Waa (Arch Linux [117]): * ardour [118] (hash ordering) * drumkv1 [119] (remove timestamp from .gz manpage) * frotz [120] (drop date) * frotz-dumb [121] * gnutls [122] (remove timestamps from .gz manpages) * samplv1 [123] (remove timestamp from .gz manpage) * sane [124] (date) * shards [125] (date) * Martin Liška: * gcc [126] (toolchain, fixing randomness in some .o files, with Alexander Monakov & Richard Biener) * Vagrant Cascadian submitted a large number patches via the Debian bug tracking system targeting the packages Civil Infrastructure Platform [127] as identified by the CIP package set [128] including: * #948757 & #948759 filed against apache2. * #948771 filed against guile-2.2. * #949114 & #949115 filed against alsa-tools. * #949270 & #949271 filed against libtool. * #949273 & #949275 filed against geoip. * #949324 filed against groff. * #949338 filed against gettext. * #949341 filed against sqlite3. * #949342 & #949343 filed against flex. * #949346 & #949348 filed against libnet. [83] https://nixos.org [84] https://github.com/NixOS/nixpkgs/pull/77196 [85] https://github.com/NixOS/nixpkgs/pull/77202 [86] https://www.gnu.org/software/gzip/manual/gzip.html [87] https://github.com/smuellerDD/jitterentropy-library/commit/0eca18fbd30318f5… [88] https://github.com/NixOS/nixpkgs/pull/77096 [89] https://sourceforge.net/p/ms-sys/patches/7/ [90] https://www.opensuse.org/ [91] https://github.com/ImageMagick/ImageMagick/pull/1270 [92] https://github.com/Tinkerforge/brickv/pull/23 [93] https://bugs.python.org/issue34033 [94] https://github.com/doxygen/doxygen/issues/7474 [95] https://github.com/alisw/fastjet/pull/6 [96] https://github.com/openjdk/jfx/pull/99 [97] https://bugs.ruby-lang.org/issues/8709 [98] https://github.com/sass/sassc-ruby/pull/178 [113] https://github.com/mltframework/shotcut/pull/824 [116] https://github.com/pikepdf/pikepdf/pull/76 [117] https://archlinux.org [118] https://github.com/ardour/ardour/pull/464/ [119] https://github.com/rncbc/drumkv1/pull/31 [120] https://gitlab.com/DavidGriffith/frotz/issues/189 [121] https://bugs.archlinux.org/task/65149 [122] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/gnu… [123] https://github.com/rncbc/samplv1/pull/30 [124] https://gitlab.com/sane-project/backends/issues/228 [125] https://github.com/crystal-lang/shards/pull/314 [126] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93274 [127] https://www.cip-project.org/ [128] https://tests.reproducible-builds.org/debian/bullseye/amd64/pkg_set_CIP.html Distribution work ================= openSUSE -------- In openSUSE [156], Bernhard M. Wiedemann published his monthly Reproducible Builds status update [157] and submitted the following bugs and patches: * doxygen [158] (toolchain, ASLR [159]; already merged upstream) * frotz [160] (version update & date) * gcc9 [161] (report unreproducible .o files, forwarded upstream [162]) * mingw* [163] (report random filename in .a files) * perl-TimeDate [164] (fix a "year 2020" bug, forwarded upstream [165]) * python-sherpa [166] (CPU-detection via --mtune=native) * qpress [167] (make PGO reproducible) * rubygem-sassc [168] (CPU & readdir, partially submitted upsteam [169]) * stgit [170] (recreate unreproducible .pyc files with fixed filesystem readdir(3) order) * xmvn [171] (report nondeterminism from filesystem order and randomness) Many Python packages were updated to avoid writing .pyc files with an embedded random path, including jupyter-jupyter-wysiwyg [172], jupyter- jupyterlab-latex [173], python-PsyLab [174], python-hupper [175], python-ipyevents [176] (don't rewrite .zip file), python-ipyleaflet [177], python-jupyter-require [178], python-jupyter_kernel_test [179], python-nbdime [180] (do not rewrite .zip, avoid time-based .pyc), python-nbinteract [181], python-plaster [182], python-pythreejs [183], python-sidecar [184] & tensorflow [185] (use pip install --no- compile). [156] https://www.opensuse.org/ [157] https://lists.opensuse.org/opensuse-factory/2020-01/msg00296.html [158] https://build.opensuse.org/request/show/766399 [159] https://en.wikipedia.org/wiki/Address_space_layout_randomization [160] https://build.opensuse.org/request/show/765634 [161] https://bugzilla.opensuse.org/show_bug.cgi?id=1160986 [162] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93274 [163] https://bugzilla.opensuse.org/show_bug.cgi?id=1160672 [164] https://build.opensuse.org/request/show/762957 [165] https://rt.cpan.org/Public/Bug/Display.html?id=124509 [166] https://build.opensuse.org/request/show/760838 [167] https://build.opensuse.org/request/show/767311 [168] https://build.opensuse.org/request/show/763080 [169] https://github.com/sass/sassc-ruby/pull/178 [170] https://build.opensuse.org/request/show/765317 [171] https://bugzilla.opensuse.org/show_bug.cgi?id=1162112 [172] https://build.opensuse.org/request/show/763685 [173] https://build.opensuse.org/request/show/763683 [174] https://build.opensuse.org/request/show/763260 [175] https://build.opensuse.org/request/show/763287 [176] https://build.opensuse.org/request/show/763292 [177] https://build.opensuse.org/request/show/763285 [178] https://build.opensuse.org/request/show/763266 [179] https://build.opensuse.org/request/show/763282 [180] https://build.opensuse.org/request/show/763281 [181] https://build.opensuse.org/request/show/763263 [182] https://build.opensuse.org/request/show/763261 [183] https://build.opensuse.org/request/show/763259 [184] https://build.opensuse.org/request/show/763257 [185] https://build.opensuse.org/request/show/763522 Debian ------ There was yet more progress towards making the Debian Installer [187] images reproducible. Following-on from last months' efforts, Chris Lamb requested a status update [188] on the Debian bug in question. [186] https://debian.org/ [187] https://www.debian.org/devel/debian-installer/ [188] https://bugs.debian.org/926242#157 Daniel Schepler posted to the debian-devel [190] mailing list to ask whether "running dpkg-buildpackage manually from the command line" [191] is supported, particularly with respect to having extra packages installed during the package was built either resulted in a failed build or even broken packages (eg. #948522, #887902, etc.). Our .buildinfo files could be one solution to this as they record the environment at the time of the package build. Holger disabled scheduling of packages from the "oldstable" stretch release on tests.reproducible-builds.org [194]. This is the first time since stretch's existence that we are no longer testing this release. [190] https://lists.debian.org/debian-devel/ [191] https://lists.debian.org/debian-devel/2020/01/msg00263.html [194] https://tests.reproducible-builds.org/ OpenJDK [195], a free and open-source implementation of the Java Platform was updated in Debian [196] to incorporate a number of patches from Emmanuel Bourg, including: * Make the generated character data source files reproducible. (#933339) * Make the generated module-info.java files reproducible. (#933342) * Make the generated copyright headers reproducible. (#933349) * Make the build user reproducible. (#933373) [195] https://jdk.java.net [196] https://tracker.debian.org/news/1094089/accepted-openjdk-11-110610-1-source… 83 reviews of Debian packages were added, 32 were updated and 96 were removed this month adding to our knowledge about identified issues [201]. Many issue types were updated by Chris Lamb, including: * timestamp_in_casacore_tables * random_identifiers_in_epub_files_generated_by_asciidoc * nondeterministic_ordering_in_casacore_tables * captures_build_path_in_golang_compiler * captures_build_path_via_haskell_adddependentfile * png_generated_by_plantuml_captures_kernel_version_and_builddate [201] https://tests.reproducible-builds.org/debian/index_issues.html Lastly, Mattia Rizzolo altered the permissions and shared the notes.git repository [202] which underpins the aforementioned package classifications with the entire "Debian" group on Salsa [203], therefore giving all DDs write access to it. This is an attempt to invite more direct contributions instead of merge requests. [202] https://salsa.debian.org/reproducible-builds/reproducible-notes [203] https://salsa.debian.org/ Other distributions ------------------- The FreeBSD Project [204] Tweeted that: > Reproducible builds are turned on by default for -RELEASE [205] ... which targets the next released version of this distribution [206]. Daniel Ebdrup [207] followed-up to note that this option: > Used to be turned on in -CURRENT when it was being tested, but it > has been turned off now that there's another branch where it's used, > whereas -CURRENT has more need to have the revision printed in uname > (which is one of the things that make a build unreproducible). [208] [204] https://www.freebsd.org/ [205] https://twitter.com/debdrup/status/1217488583503892480 [206] https://svnweb.freebsd.org/base?view=revision&revision=338642 [207] https://nullrouted.org/ [208] https://twitter.com/debdrup/status/1217515991267184645 For Alpine Linux [209], Holger Levsen disabled the builders run by the Reproducible Builds project as our patch to the abuild utility (see December's report [210] doesn't apply anymore and thus all builds have become unreproducible again. Subsequent to this, a patch was merged upstream. [211] [209] https://alpinelinux.org/ [210] https://reproducible-buids.org/reports/2019-12/ [211] https://github.com/alpinelinux/abuild/pull/110#issuecomment-580195097 In GNU Guix [212], on January 14th, Konrad Hinsen posted a blog post entitled "Reproducible computations with Guix" [213] which, amongst other things remarks that: > The [guix time-machine command] machine actually downloads the > specified version of Guix and passes it the rest of the command line. > You are running the same code again. Even bugs in Guix will be > reproduced faithfully! [209] https://alpinelinux.org/ [210] https://reproducible-buids.org/reports/2019-12/ [211] https://github.com/alpinelinux/abuild/pull/110#issuecomment-580195097 [212] https://guix.gnu.org/ [213] http://guix.gnu.org/blog/2020/reproducible-computations-with-guix/ The Yocto Project [214] reported that they have reproducible cross-built binaries that are independent of both the underlying host distribution the build is run on and independent of the path used for the build. This is now being continually tested on the Yocto Project's automated infrastructure to ensure this state is maintained in the future. [214] https://www.yoctoproject.org/ Project website & documentation ------------------------------- There was more work performed on our website [215] this month, including: * Chris Lamb: * Python SOURCE_DATE_EPOCH [216] documentation, clarifying that the second example generates a Python str-type, not a datetime.datetime [217] * Correct word omissions in the report template. [218] * Link to to our mailing list overview page [219] (and not the archives). [220] * Apply the Black [221] source code reformatter to the draft generation script. [222] * Move continuous tests heading level to <h1> (vs. <h2>) to match the other pages. [223] * Calculate the report authors dynamically. [224] * Holger Levsen: * Add Alpine Linux to our projects [226] and testing [227] pages. [228] * Add links to our list of projects being tested [229] [230] and mark Fedora [231] as being disabled at this time [232]. In addition, Arnout Engelen added a Scala [233] programming language example for the SOURCE_DATE_EPOCH environment variable [235], David del Amo updated the link to the Software Freedom Conversancy [236] to remove some double parentheses [237] and Peter Wu added a Debian example for the -ffile-prefix-map argument to support Clang [238] version 10 [239]. [215] https://reproducible-builds.org/ [216] https://reproducible-buids.org/docs/SOURCE_DATE_EPOCH [217] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8e… [218] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7b… [219] https://lists.reproducible-builds.org/listinfo/rb-general [220] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6e… [221] https://black.readthedocs.io/en/stable/ [222] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9f… [223] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/e1… [224] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/21… [226] https://reproducible-buids.org/who/ [227] https://reproducible-buids.org/citests/ [228] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/45… [229] https://reproducible-buids.org/citests/ [230] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3f… [231] https://getfedora.org/ [232] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/19… [233] https://www.scala-lang.org/ [235] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/49… [236] https://sfconservancy.org/ [237] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f4… [238] https://clang.llvm.org/ [239] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a9… Testing framework ----------------- We operate a full-featured and comprehensive Jenkins [240]-based testing framework that powers tests.reproducible-builds.org [241]. This month, the following changes were made: * Adrian Bunk: * Use the et_EE locale/language instead of fr_CH. In Estonian, the "z" character is sorted between "s" and "t" which is contrary to common incorrect assumptions about the sorting order of ASCII [242] characters. [243] * Add ffile_prefix_map_passed_to_clang to the list of issues filtered as these build failures should be ignored. [244] * Remove the ftbfs_build_depends_not_available_on_amd64 from the list of filtered issues as this specific problem no longer exists. [245] * Holger Levsen: * Debian [246]: * Always configure apt to ignore expired release files on hosts running in the future. [247] * Create an "oldsuites" page, showing suites we used to test in the past. [248][249][250][251][252] * Schedule more old packages from the buster distribution. [253] * Deal with shell escaping and other options. [254][255][256] * Reverse the suite ordering on the packages page. [257][258] * Show bullseye statistics on dashboard page, moving away from buster [259] and additionally omit stretch [260]. * F-Droid [261]: * Document the increased diskspace requirements; we require over 700 GiB now. [262] * Misc: * Gracefully deal with umount problems. [263][264] * Run code to show "todo" entries locally. [265] * Use mmdebstrap instead of debootstrap. [266][267][268] * Jelle van der Waa (Arch Linux [269]): * Set the PACKAGER variable to a valid string to avoid noise in the logging. [270] * Add a link to the Arch Linux-specific package page in the overview table. [271] * Mattia Rizzolo: * Fix a hard-coded reference to the current year. [272] * Ignore "No server certificate defined" warning messages when automatically parsing logfiles. [273] * Vagrant Cascadian special-cased u-boot [274] on the armhf architecture: First, do not build the all architecture as the dependencies are not available on this architecture [275] and also pass the --binary-arch argument to pbuilder too [276]. The usual node maintenance was performed by Mattia Rizzolo [277][278], Vagrant Cascadian [279][280][281][282] and Holger Levsen. [240] https://jenkins.io/ [241] https://tests.reproducible-builds.org [242] https://en.wikipedia.org/wiki/ASCII [243] https://salsa.debian.org/qa/jenkins.debian.net/commit/d1f68a80 [244] https://salsa.debian.org/qa/jenkins.debian.net/commit/ea55e4bc [245] https://salsa.debian.org/qa/jenkins.debian.net/commit/64a062e3 [246] https://debian.org/ [247] https://salsa.debian.org/qa/jenkins.debian.net/commit/30a71a97 [248] https://salsa.debian.org/qa/jenkins.debian.net/commit/ffc13a29 [249] https://salsa.debian.org/qa/jenkins.debian.net/commit/c1accbfb [250] https://salsa.debian.org/qa/jenkins.debian.net/commit/13550757 [251] https://salsa.debian.org/qa/jenkins.debian.net/commit/2545eba2 [252] https://salsa.debian.org/qa/jenkins.debian.net/commit/d9b11c34 [253] https://salsa.debian.org/qa/jenkins.debian.net/commit/e5ebb1fa [254] https://salsa.debian.org/qa/jenkins.debian.net/commit/a35cc960 [255] https://salsa.debian.org/qa/jenkins.debian.net/commit/872453bb [256] https://salsa.debian.org/qa/jenkins.debian.net/commit/b5091f70 [257] https://salsa.debian.org/qa/jenkins.debian.net/commit/d27e8d3f [258] https://salsa.debian.org/qa/jenkins.debian.net/commit/d495b88f [259] https://salsa.debian.org/qa/jenkins.debian.net/commit/db5cf9c0 [260] https://salsa.debian.org/qa/jenkins.debian.net/commit/5516f56f [261] https://f-droid.org/ [262] https://salsa.debian.org/qa/jenkins.debian.net/commit/a6a0a691 [263] https://salsa.debian.org/qa/jenkins.debian.net/commit/a555f19e [264] https://salsa.debian.org/qa/jenkins.debian.net/commit/9f58a8df [265] https://salsa.debian.org/qa/jenkins.debian.net/commit/cdad9995 [266] https://salsa.debian.org/qa/jenkins.debian.net/commit/6ce1a757 [267] https://salsa.debian.org/qa/jenkins.debian.net/commit/cab70212 [268] https://salsa.debian.org/qa/jenkins.debian.net/commit/464508f1 [269] https://archlinux.org/ [270] https://salsa.debian.org/qa/jenkins.debian.net/commit/66c8eee0 [271] https://salsa.debian.org/qa/jenkins.debian.net/commit/95c9006e [272] https://salsa.debian.org/qa/jenkins.debian.net/commit/d61c0338 [273] https://salsa.debian.org/qa/jenkins.debian.net/commit/1919aa19 [274] https://www.denx.de/wiki/U-Boot/ [275] https://salsa.debian.org/qa/jenkins.debian.net/commit/08ef4bae [276] https://salsa.debian.org/qa/jenkins.debian.net/commit/0c838739 [277] https://salsa.debian.org/qa/jenkins.debian.net/commit/0c6b7024 [278] https://salsa.debian.org/qa/jenkins.debian.net/commit/e60be6d4 [279] https://salsa.debian.org/qa/jenkins.debian.net/commit/79dbfa03 [280] https://salsa.debian.org/qa/jenkins.debian.net/commit/d2fcbfab [281] https://salsa.debian.org/qa/jenkins.debian.net/commit/d3d94fcc [282] https://salsa.debian.org/qa/jenkins.debian.net/commit/19baf9e9 Misc news --------- On our mailing list [283] this month: * Chris Lamb responded in-depth to a thread on "Reproducible system images" [284] that was started in December [285] by Lars Wirzenius. This then led to a sub-thread regarding reproducible Docker images [286]. * Holger Levsen posted a brief request for help [287] regarding the bot that lives on our #reproducible-builds IRC channel that interfaces with our Twitter handle [288]. [283] https://lists.reproducible-builds.org/listinfo/rb-general/ [284] https://lists.reproducible-builds.org/pipermail/rb-general/2020-January/001… [285] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/00… [286] https://lists.reproducible-builds.org/pipermail/rb-general/2020-January/001… [287] https://lists.reproducible-builds.org/pipermail/rb-general/2020-January/001… [288] https://twitter.com/ReproBuilds § If you are interested in contributing to the Reproducible Builds project, please visit our "Contribute" [289] page on our website. However, you can also get in touch with us via: * IRC: #reproducible-builds on irc.oftc.net. * Twitter: @ReproBuilds [290] * Reddit: /r/ReproducibleBuilds [291] * Mailing list: rb-general(a)lists.reproducible-builds.org [292] [289] https://reproducible-builds.org/contribute/ [290] https://twitter.com/ReproBuilds [291] https://reddit.com/r/reproduciblebuilds [292] https://lists.reproducible-builds.org/listinfo/rb-general § This month's report was written by Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, heinrich5991, Holger Levsen, Jelle van der Waa, Mattia Rizzolo and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list. Best wishes, -- Chris Lamb https://puri.sm

1 0

December 2019 in Reproducible Builds
by Chris Lamb 06 Jan '20

06 Jan '20

==================================================================== o ⬋ ⬊ December 2019 in Reproducible Builds o o ⬊ ⬋ https://reproducible-builds.org/reports/2019-12/ o ==================================================================== Welcome to the December 2019 report from the Reproducible Builds project. In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. In this report for December, we cover: * Media coverage — A Google whitepaper, The Update Framework graduates within the Cloud Native Computing Foundation, etc. * Reproducible Builds Summit 2019 — What happened at our recent meetup? * Distribution work — The latest reports from Arch, Debian and openSUSE, etc. * Software development — Patches, patches, patches... * Mailing list summary * Contact — How to contribute. If you are interested in contributing to our project, please visit the Contribute [1] page on our website. [1] https://reproducible-builds.org/contribute/ Media coverage ============== Google published Binary Authorization for Borg [2], a whitepaper on how they reduce exposure of user data to unauthorised code as well as methods for verifying code provenance using their Borg [3]) cluster manager. In particular, the paper notes how they attempt to limit their "insider risk", ie. the potential for internal personnel to use organisational credentials or knowledge to perform malicious activities. The Linux Foundation [4] announced that The Update Framework [5] (TUF) has graduated within [6] the Cloud Native Computing Foundation (CNCF) and thus becomes the first specification and first security- focused project to reach the highest maturity level in that group. TUF is a technology that secures software update systems initially developed by Justin Cappos [8] at the NYU Tandon School of Engineering [9]. Andrew "bunnie" Huang published a blog post asking "Can We Build Trustable Hardware?" [11]. Whilst it concludes pessimistically that "open hardware is precisely as trustworthy as closed hardware" it does mention that reproducible builds can: > Enable any third-party auditor to download, build, and confirm > that the program a user is downloading matches the intent of the > developers. At the 36th Chaos Communication Congress [12] (36C3) in Leipzig, Hannes Mehnert from the MirageOS [13] project gave a presentation called *Leaving legacy behind* [14] which talks generally about *Mirage* system offering a potential alternative and minimalist approach to security but has a section on reproducible builds (at 38m41s. [ 2] https://cloud.google.com/security/binary-authorization-for-borg/ [ 3] https://en.wikipedia.org/wiki/Borg_(cluster_manager [ 4] https://www.linuxfoundation.org/ [ 5] https://theupdateframework.io/ [ 6] https://www.cncf.io/announcement/2019/12/18/cloud-native-computing-foundati… [ 8] https://engineering.nyu.edu/faculty/justin-cappos [ 9] https://engineering.nyu.edu/ [11] https://www.bunniestudios.com/blog/?p=5706 [12] https://events.ccc.de/congress/2019/wiki/index.php/Main_Page [13] https://mirage.io/ [14] https://media.ccc.de/v/36c3-11172-leaving_legacy_behind Reproducible Builds Summit 2019 =============================== We held our fifth annual Reproducible Builds summit [16] between the 1st and 8th December at Priscilla, Queen of the Medina [17] in Marrakesh, Morocco. The aim of the meeting was to spend time dicussing and working on Reproducible Builds with a widely diverse agenda and the event was a huge success. During our time together, we updated and exchanged the status of reproducible builds in our respective projects, improved collaboration between and within these efforts, expanded the scope and reach of reproducible builds to yet more interested parties, established and continued strategic long-term thinking in a way not typically possible via remote channels, and brainstormed designs for tools to enable end- users to get the most benefit from reproducible builds. Outside of these achievements in the hacking sessions kpcyrd made a breakthrough in Alpine Linux [18] by producing the first reproducible package — specifically, py3-uritemplate [19] — in this operating system. After this, progress was accelerated and by the denouement of our meeting the reproducibility status in Alpine reached 94%. In addition, Jelle van der Waa, Mattia Rizzolo and Paul Spooren discussed and implemented substantial changes to the database that underpins the testing framework that powers tests.reproducible-builds.org in order to abstract the schema in a distribution agnostic way, for example to allow submitting the results of attempts to verify officially distributed Arch Linux packages. Lastly, Jan Nieuwenhuizen, David Terry and Vagrant Cascadian used three entirely-separate distributions (GNU Guix, NixOS and Debian) to produce a bit-for-bit identical GNU Mes [26] binary despite using three different major versions of GCC and other toolchain components to build an initial binary, which was then used to build a final, bit-for-bit identical, binary of Mes. The event was held at Priscilla, Queen of the Medina [27] in Marrakesh, a location «sui generis» that stands for gender equality, female empowerment and the engagement of vulnerable communities locally through cultural activism. The event was open to anybody interested in working on Reproducible Builds issues, with or without prior experience. A number of reports and blog posts have already been written, including for: * openSUSE [28] * OCaml, "opam" and MirageOS [29] * GNU Guix [30] [16] https://reproducible-builds.org/events/Marrakesh2019/ [17] https://www.queenscollective.org/artistryasactivism [18] https://alpinelinux.org/ [19] https://tests.reproducible-builds.org/alpine/main/py3-uritemplate/py3-urite… [26] https://www.gnu.org/software/mes/ [27] https://www.queenscollective.org/artistryasactivism [28] https://lizards.opensuse.org/2019/12/13/opensuse-on-reproducible-builds-sum… [29] https://hannes.nqsb.io/Posts/ReproducibleOPAM [30] https://guix.gnu.org/blog/2019/reproducible-builds-summit-5th-edition/ Distribution work ================= Within Debian, Chris Lamb categorised a large number of packages and issues in the Reproducible Builds notes.git [34] repository, including identifying and creating markdown_random_email_address_html_entities and nondeterministic_devhelp_documentation_generated_by_gtk_doc. [34] https://salsa.debian.org/reproducible-builds/reproducible-notes/activity In openSUSE, Bernhard published his monthly Reproducible Builds status update [38] and filed the following patches: * hidviz [39] (use convert -strip) * python-ipydatawidgets [40] (make pip install reproducible, avoid trouble with Zip order & mtime [41]) * python-jupyterlab-templates [42] (make pip install reproducible) * python-jupyterlab [43] (make pip install reproducible) * python-mox3 [44] (drop Sphinx [45] environment.pickle file) * rpmlint-mini [46] (sort Python compile file list) * rubygem-ronn [47] ( Ruby date, submitted upstream [48] with updated patch) * syslinux6 [49] (sort find / readdir; already upstream) Bernhard also filed bugs against: * libhugetlbfs [50] (unreproducible .ldscript file) * libmicro [51] (Link-Time Optimisation [52] causing unreproducible object files; fix by Martin Pluskal [53]) * python-swifter [54] (report failure to build on single-core CUPs) * tesseract-ocr [55] (report variations via their build machine's CPU) [38] https://lists.opensuse.org/opensuse-factory/2019-12/msg00174.html [39] https://build.opensuse.org/request/show/754485 [40] https://build.opensuse.org/request/show/760182 [41] https://en.wikipedia.org/wiki/Mtime [42] https://build.opensuse.org/request/show/757375 [43] https://build.opensuse.org/request/show/755664 [44] https://build.opensuse.org/request/show/760190 [45] http://www.sphinx-doc.org/ [46] https://build.opensuse.org/request/show/754705 [47] https://build.opensuse.org/request/show/757287 [48] https://github.com/kamontat/ronn/pull/3 [49] https://build.opensuse.org/request/show/759820 [50] https://bugzilla.opensuse.org/show_bug.cgi?id=1159558 [51] https://bugzilla.opensuse.org/show_bug.cgi?id=1159556 [52] https://en.wikipedia.org/wiki/Interprocedural_optimization [53] https://build.opensuse.org/request/show/758238 [54] https://bugzilla.opensuse.org/show_bug.cgi?id=1158578 [55] https://bugzilla.opensuse.org/show_bug.cgi?id=1159231 The Yocto Project announced that it is running continuous tests on the reproducibility of its output [57] which can observed through the oe-selftest runs on their build server [58]. This was previously limited to just the mini images but this has now been extended to the larger graphical images. The test framework is available for end users to use against their own builds. Of particular interest is the production of binary identical results despite arbitrary build paths to allow more efficient builds through reuse of previously built objects, a topic covered in more-depth in a recent LWN article [59]. [56] https://www.yoctoproject.org/ [57] http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases… [58] https://autobuilder.yoctoproject.org/typhoon/#/console [59] https://lwn.net/Articles/804640/ In Arch Linux, the database structure on tests.reproducible- builds.org was changed and the testing jobs updated to match and work has been started on a verification test job which rebuilds the officially released packages and verifies if they are reproducible or not. In the "hacking" time after our recent summit, several key packages were made reproducible, raising the amount of reproducible packages by approximately 1.5%. For example libxslt [62] was patched with the patch originating from Debian and openSUSE. [62] https://www.archlinux.org/packages/extra/x86_64/libxslt/ Software development ==================== diffoscope ---------- diffoscope [64] is our in-depth and content-aware diff-like utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure [65] and is essential for identifying fixes and causes of non-deterministic behaviour. This month, diffoscope version 134 was uploaded to Debian unstable by Chris Lamb. He also made the following changes to diffoscope itself, including: * Always pass a filename with a .zip extension to zipnote otherwise it will return with an UNIX exit code [66] of 9 and we fallback to displaying a binary difference for the entire file. [67] * Include the libarchive [68] file listing for ISO images to ensure that timestamps — and not just dates — are visible in any difference. [69] * Ensure that our autopkgtests [70] are run with our pyproject.toml [71] present for the correct black source code formatter settings. [72] * Rename the text_option_with_stdiout test to text_option_with_stdout [73] and tidy some unnecessary boolean logic in the ISO9660 tests [75]. [64] https://diffoscope.org [65] https://tests.reproducible-builds.org/debian/reproducible.html [66] https://en.wikipedia.org/wiki/Exit_status [67] https://salsa.debian.org/reproducible-builds/diffoscope/commit/a93aa33 [68] https://www.libarchive.org/ [69] https://salsa.debian.org/reproducible-builds/diffoscope/issues/81 [70] https://ci.debian.net/ [71] https://snarky.ca/clarifying-pep-518/ [72] https://bugs.debian.org/945993 [73] https://salsa.debian.org/reproducible-builds/diffoscope/commit/cb1c732 In addition, Eli Schwartz fixed an error in the handling of the progress bar [76] and Vagrant Cascadian added external tool reference for the zstd [77] compression format for GNU Guix [79] as well as updated the version to 133 in that distribution [80][81]. [75] https://salsa.debian.org/reproducible-builds/diffoscope/commit/341b98a [76] https://salsa.debian.org/reproducible-builds/diffoscope/commit/8706b87 [77] https://github.com/facebook/zstd [79] https://salsa.debian.org/reproducible-builds/diffoscope/commit/8c1b357 [80] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6a65185ee46babca0630d… [81] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=5de06b9dfb7e8fa5e3218… Project website & documentation ------------------------------- There was more work performed on our website this month, including: * Bernhard M. Wiedemann: * Add an OCaml example to our SOURCE_DATE_EPOCH documentation [84] and simplify the POSIX shell and date format usage [85][86] * Add a few "logo only" variations of our logo. [87] * Chris Lamb: * Add a link to the Tails [88] privacy-related operating system's instructions on how to verify a downloaded image. [90] * Add a link to the Reproducible Builds subreddit [91] to the page footer. [92] * Correct a "name" typo [93], add a missing "to" [94] and correct capitalisations of "OCaml" throughout the site [95]. * Jelle van der Waa: * Update the GNU Guix logo to the new design. [97] * Fix "signed tarballs are available" link on our Tools [98] page. [99] * Mattia Rizzolo: * Add an explicit robots.txt [100] file. [101] * Add a Google "site verification" [102] token. (Also added to the diffoscope website). [104][105] [88] https://tails.boum.org/ [90] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/fe… [91] https://www.reddit.com/r/reproduciblebuilds/ [92] https://salsa.debian.org/reproducible-builds/reproducible-website/issues/20 [93] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f0… [94] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/02… [95] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3b… [97] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/87… [98] https://reproducible-builds.org/docs/jvm/ [99] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/87… [100] https://www.robotstxt.org/ [101] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/63… [102] https://support.google.com/webmasters/answer/9008080?hl=en [104] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1b… [105] https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/875e… In addition, Paul Spooren added a new page overviewing our Continuous Tests [106] overview [107], Hervé Boutemy made a number of improvements to our Java and JVM documentation [108] expanding and clarifying various definitions as well as adding external links [109][110][111][112] and Mariana Moreira added a .jekyll-cache entry to the .gitignore file [114]. [106] https://reproducible-builds.org/citests/ [107] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1c… [108] https://reproducible-builds.org/docs/jvm/ [109] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/79… [110] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/93… [111] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f3… [112] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/fd… [114] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/eb… Upstream patches ---------------- The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Arnout Engelen: * sbt [115] (timestamps and file order in generated archives) * NixOS [116] installer/iso-image [117] (timestamps in ISO installer image) * Generated an updated NixOS reproducibility report [118] for nixos-unstable's iso_minimal installer image. * Bernhard M. Wiedemann: * bowtie [119] (date) * charybdis [120] (shell date & time) * coq [121] (report that .vo files vary from build order) * coq [122] (OCaml date) * kismet [123] (date) * libcec [124] (CMake: use TIMESTAMP variable instead of build date) * lifelines [125] (date) * OpenStack Python packages [126] (don't package a .pickle file) * orthanc [127] (sort Python readdir) * perl [128] (fix documentation-related build failure in 2020) * php7-pear [129] (sort a PHP-based readdir) * pmix [130] (date, time, host & user) * pw3270 [131] (make date & convert -strip) * python-autobahn [132] (report stuck tests on single CPU machine) * python-psychtoolbox [133] (sort Python readdir) * python-python-crfsuite [134] (sort Python glob [135] / readdir) * ripgrep [136] (report variations from CPU) * rubygem-ronn [137] (updated date patch) * vpp [138] (shell date, regression fix) * Multiple patches to the grass Geographic Information System. [140][141][142] * Jelle van der Waa: * tbb [143] (hostname, date & time) * pcp [144] (date & time) * libcec [145] (date & time) * cgdb [146] (date & time) * cloc [147] (date & time) * dlang [148] (please add support SOURCE_DATE_EPOCH in the D programming language [149] compiler, dlang) * dlang [150] (date & time in the D dtools library) * Chris Lamb: * #857454 re-opened against qtltools * #946315 filed against infernal (forwarded upstream [155]). * #946330 filed against usb-modeswitch-data (applied upstream). * #946331 filed against gtk-doc (forwarded upstream [160]). * #946332 filed against nftables. * #946333 filed against node-chart.js (forwarded upstream [165]). * #946335 filed against parsinsert. * #947608 filed against markdown. * #947708 filed against libtext-markdown-perl. [115] https://github.com/sbt/sbt/pull/5344 [116] https://github.com/NixOS/nixpkgs/pulls?q=is%3Apr+label%3A%226.topic%3A+repr… [117] https://github.com/NixOS/nixpkgs/pull/75484 [118] https://arnout.engelen.eu/nixos-r13y/report/ [119] https://github.com/BenLangmead/bowtie/pull/99 [120] https://github.com/charybdis-ircd/charybdis/pull/297 [121] https://github.com/coq/coq/issues/11229 [122] https://github.com/coq/coq/pull/11227 [123] https://github.com/kismetwireless/kismet/pull/195 [124] https://github.com/Pulse-Eight/libcec/pull/487 [125] https://github.com/lifelines/lifelines/pull/389 [126] https://review.opendev.org/700810 [127] https://bitbucket.org/sjodogne/orthanc/pull-requests/12/sort-file-lists/diff [128] https://github.com/Perl/perl5/pull/17390 [129] https://github.com/pear/pear-core/pull/105 [130] https://github.com/openpmix/openpmix/pull/1560 [131] https://github.com/PerryWerneck/pw3270/pull/2 [132] https://github.com/crossbario/autobahn-python/issues/1275 [133] https://github.com/Psychtoolbox-3/Psychtoolbox-3/pull/614 [134] https://github.com/scrapinghub/python-crfsuite/pull/115 [135] https://docs.python.org/3/library/glob.html [136] https://github.com/BurntSushi/ripgrep/issues/1441 [137] https://github.com/kamontat/ronn/pull/3 [138] https://gerrit.fd.io/r/c/vpp/+/23819 [140] https://github.com/OSGeo/grass/pull/247 [141] https://github.com/OSGeo/grass/pull/251 [142] https://github.com/OSGeo/grass/pull/263 [143] https://github.com/intel/tbb/issues/202 [144] https://github.com/performancecopilot/pcp/pull/805 [145] https://github.com/Pulse-Eight/libcec/issues/485 [146] https://github.com/cgdb/cgdb/pull/215 [147] https://github.com/AlDanial/cloc/pull/438 [148] https://issues.dlang.org/show_bug.cgi?id=20444 [149] https://dlang.org/ [150] https://issues.dlang.org/show_bug.cgi?id=20445 [155] https://github.com/EddyRivasLab/infernal/pull/19 [160] https://gitlab.gnome.org/GNOME/gtk-doc/merge_requests/37 [165] https://github.com/chartjs/Chart.js/pull/6817 Test framework -------------- We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, the following changes were made: * Holger Levsen: * Alpine: * Indicate where Alpine is being built on the node overview page. [175] * Turn off debugging output. [176] * Sleep longer if no packages are to be built. [177] * Misc: * Add some help text to our script to powercycle IONOS [178] (*neé* Profitbricks) nodes. [179] * Install mosh [180] everywhere. [181] * Only install ripgrep [182] on Debian nodes. [183] * Mattia Rizzolo: * Arch Linux: * Normalise the suite names in the database. [185][186][187][188][189] * Drop an unneeded line in the scheduler. [190] * Debian: * Fix a number of SQL errors. [192][193][... [194][195] * Use the debian.debian_support Python library over apt_pkg to perform version comparisons. [196] * Misc: * Permit other distributions to use our web-based package scheduling script. [197 * Reformat our power-cycling script using Black [198] and use the Python logging [199] module. [200] * Introduce a dsources database view to simplify some queries [201] and add a build_type field to support both "doublerebuilds" and verification rebuilds [202]. * Move (almost) all the timestamps in the database schema from raw strings to "real" timestamp data types. [203] * Only block bots on jenkins.debian.net [204] and tests.reproducible-builds.org [205], not any other sites. [206] * kpcyrd (for Alpine Linux): * Patch/install the abuild utility to one that is reproducible. [208][209][210][211] * Bump the number of build workers and collect garbage more frequently. [212][213][214][215] * Classify and display build results consistently. [216][217][218] * Ensure that tmux [219] and ripgrep [220] is installed. [221][222] * Support building packages in the future. [223][224][225] [175] https://salsa.debian.org/qa/jenkins.debian.net/commit/4af96f16 [176] https://salsa.debian.org/qa/jenkins.debian.net/commit/6a461023 [177] https://salsa.debian.org/qa/jenkins.debian.net/commit/f1d3c700 [178] https://www.ionos.com/ [179] https://salsa.debian.org/qa/jenkins.debian.net/commit/23442fc2 [180] https://mosh.org/ [181] https://salsa.debian.org/qa/jenkins.debian.net/commit/25e3d43b [182] https://github.com/BurntSushi/ripgrep [183] https://salsa.debian.org/qa/jenkins.debian.net/commit/f3a3ce6b [185] https://salsa.debian.org/qa/jenkins.debian.net/commit/7a0295e8 [186] https://salsa.debian.org/qa/jenkins.debian.net/commit/231884e8 [187] https://salsa.debian.org/qa/jenkins.debian.net/commit/62750403 [188] https://salsa.debian.org/qa/jenkins.debian.net/commit/d8473a13 [189] https://salsa.debian.org/qa/jenkins.debian.net/commit/77d3b173 [190] https://salsa.debian.org/qa/jenkins.debian.net/commit/035f6170 [192] https://salsa.debian.org/qa/jenkins.debian.net/commit/cd4ee15d [193] https://salsa.debian.org/qa/jenkins.debian.net/commit/e380dad1 [194] https://salsa.debian.org/qa/jenkins.debian.net/commit/8c515b2d [195] https://salsa.debian.org/qa/jenkins.debian.net/commit/528f3bce [196] https://salsa.debian.org/qa/jenkins.debian.net/commit/7677b378 [197] https://salsa.debian.org/qa/jenkins.debian.net/commit/cb775560 [198] https://black.readthedocs.io/ [199] https://docs.python.org/3/library/logging.html [200] https://salsa.debian.org/qa/jenkins.debian.net/commit/325b9f57 [201] https://salsa.debian.org/qa/jenkins.debian.net/commit/95eb84e6 [202] https://salsa.debian.org/qa/jenkins.debian.net/commit/86160814 [203] https://salsa.debian.org/qa/jenkins.debian.net/commit/6e7a475c [204] https://jenkins.debian.net/ [205] http://tests.reproducible-builds.org/ [206] https://salsa.debian.org/qa/jenkins.debian.net/commit/e09cda74 [208] https://salsa.debian.org/qa/jenkins.debian.net/commit/3b55b4d3 [209] https://salsa.debian.org/qa/jenkins.debian.net/commit/b4cfe3d3 [210] https://salsa.debian.org/qa/jenkins.debian.net/commit/2d81fa1a [211] https://salsa.debian.org/qa/jenkins.debian.net/commit/6c3c15e0 [212] https://salsa.debian.org/qa/jenkins.debian.net/commit/35a3dd33 [213] https://salsa.debian.org/qa/jenkins.debian.net/commit/a97cb13c [214] https://salsa.debian.org/qa/jenkins.debian.net/commit/83cc9dca [215] https://salsa.debian.org/qa/jenkins.debian.net/commit/30138aa1 [216] https://salsa.debian.org/qa/jenkins.debian.net/commit/21026d76 [217] https://salsa.debian.org/qa/jenkins.debian.net/commit/70a8fe35 [218] https://salsa.debian.org/qa/jenkins.debian.net/commit/9eeb3a5a [219] https://tmux.github.io/ [220] https://github.com/BurntSushi/ripgrep [221] https://salsa.debian.org/qa/jenkins.debian.net/commit/332f2549 [222] https://salsa.debian.org/qa/jenkins.debian.net/commit/3b43b4f9 [223] https://salsa.debian.org/qa/jenkins.debian.net/commit/912f3126 [224] https://salsa.debian.org/qa/jenkins.debian.net/commit/71380c9a [225] https://salsa.debian.org/qa/jenkins.debian.net/commit/5ee25a02 Lastly, Paul Spooren removed the project overview from the bottom-left of the generated pages [226] and the usual node maintenance was performed by Holger Levsen [227] and Mattia Rizzolo [228][229], etc. [226] https://salsa.debian.org/qa/jenkins.debian.net/commit/23eb5845 [227] https://salsa.debian.org/qa/jenkins.debian.net/commit/dea04259 [228] https://salsa.debian.org/qa/jenkins.debian.net/commit/7587e568 [229] https://salsa.debian.org/qa/jenkins.debian.net/commit/6d8111ce Mailing list summary ==================== There was considerable activity on our mailing list [230] this month. Firstly, Bernhard M. Wiedemann posted a thread asking "What is the goal of reproducible builds?" [231] in order to encourage refinements, extra questions and other contributions to what an end-user experience of reproducible builds should or even could look like. Eli Schwartz then resurrected a previous thread titled "Progress in rpm and openSUSE in 2019" [232] to clarify some points around Arch Linux and Python package installation. Hans-Christoph Steiner followed- up to a separate thread [234] originally started by Hervé Boutemy announcing the status of .buildinfo file support in the Java ecosystem, and Paul Spooren then informed the list [235] that Google Summer of Code is now looking for projects for the latest cohort. Lastly, Lars Wirzenius enquired about the status of Reproducible system images [237] which resulted in a large number of responses [238]. [230] https://lists.reproducible-builds.org/listinfo/rb-general/ [231] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/00… [232] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/00… [234] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/00… [235] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/00… [237] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/00… [238] https://lists.reproducible-builds.org/pipermail/rb-general/2019-December/th… Contact ======= If you are interested in contributing to the Reproducible Builds project, please visit the "Contribute" page on our website: https://reproducible-builds.org/contribute/ However, you can get in touch with us via: * IRC: #reproducible-builds on irc.oftc.net. * Twitter: https://twitter.com/@ReproBuilds * Reddit: https://reddit.com/r/reproduciblebuilds * Mailing list: https://lists.reproducible-builds.org/listinfo/rb-general This month's report was written by Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, Hervé Boutemy, Holger Levsen, Jelle van der Waa, Lukas Puehringer and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list. -- o ⬋ ⬊ Chris Lamb o o reproducible-builds.org ⬊ ⬋ o

1 0

Bits from PureOS | 2020 edition
by Jeremiah C. Foster 04 Jan '20

04 Jan '20

Hello faithful readers! Your attention, replies, and hardwork made 2019 a great year for this mailing list. There's been a huge increase in communication here and I'm grateful for the discussion, feedback, and guidance. I'm dropping this link here to let those who read this list know that we've had a summary of the past year published; https://puri.sm/posts/2019-year-in-review-pureos/ With that posted, it is time to look towards 2020 and layout what the priorities are for the new year. Without question I think the number one priority from Purism management is the PureOS Store. In my discussions with Todd about the Store, it is clear not only that it is a high priority, but that the store can be a way to bolster PureOS and its large archive and get Free Software into the hands of users. This, I feel, is hugely important to our mission and so I'm going to prioritize it at the top of the list. Fortunately, this has been an idea that has already been floated and there is work that's been done towards realizing a store or easily accessible archive in some form or another. I hope to be able to pull the disparate threads together; work done in Laniakea, frontend work from Rodolfo and Francois, as well as other infrastrucutre bits to realize the goal. I hope to organize this work in the coming weeks and try and set out a schedule so we can track and measure our progress and, most importantly, communicate the work that's been done by the team because I think it's quite amazing. More regular releases for the folks who install PureOS on machines on the factory floor is also a high priority. We've come so far now with our releases - amber and byzantium - that it's time to support them with a bit more predictable release cycle and more Quality Assurance. To do that let's pick up the thread on releases, I'll try and revive that. Matthias K. has mentioned OpenQA as a potentially useful tool for doing QA on our releases and I agree, I hope to set that up for us before FOSDEM so we can use it. I have a couple other small tasks to complete before that work can be started as well as some administrative housekeeping, but I'm really eager to get more QA for our releases since we've come so far the last year or so. And yes, FOSDEM! I'm going to be there for about a week. I plan on attending the MiniDebCamp: https://wiki.debian.org/DebianEvents/be/2020/MiniDebCamp It would be great if all got a chance to meet up in Brussels, I would really appreciate it. Just having the opportunity to speak face to face would benefit me tremendously and having a chance to outline all of our goals and plans for 2020 will be an added bonus. Thanks for reading this far! Best, Jeremiah

4 5

policy for level of stability of PureOS stable
by Jonas Smedegaard 20 Dec '19

20 Dec '19

How stable is PureOS stable a.k.a. amber? Which criteria for permitting new packages to get included? Which criteria for permitting existing packages to get updated? Do I just use my gut feeling? Concretely, David Seaward wants ldh-gui-suite added, but makes sense to me to address this generally, eary on. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private

4 6

November 2019 in Reproducible Builds
by Chris Lamb 12 Dec '19

12 Dec '19

==================================================================== o ⬋ ⬊ November 2019 in Reproducible Builds o o ⬊ ⬋ https://reproducible-builds.org/reports/2019-11/ o ==================================================================== Welcome to the November 2019 report from the Reproducible Builds project. As a summary of our project, whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is therefore to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. In this month's report, we cover: * Media coverage and events — A Reproducibility Challenge, etc. * Upstream news — OCaml, Mes, Maven, etc. * Distribution work — The latest reports from Arch, Debian and openSUSE, and friends * Software development — Holiday bonanza of patches, work on diffoscope, etc. * Contributing — How to get in touch, etfc. If you are interested in contributing to our project, please visit the Contribute [1] page on our website. Media coverage and events ========================= We held our fifth annual Reproducible Builds summit [2] between the 1st and 8th December in Marrakesh, Morocco. A full, in-depth report will be posted next month… On November 16th, Vagrant Cascadian presented *There and Back Again, Reproducibly* [3] at the SeaGL [4] in Seattle, Washington. Chris Lamb was featured on The Manifest [5] package management podcast in an episode called *Reproducible Builds project and Debian package management* [6]. ReScience C [7] is an open-access journal that targets computational research and encourages the explicit replication of already published research. This month they announced their *Ten Years Reproducibility Challenge* [8] which promotes the idea that old code — in this instance, a "scientific article [published] before January 1st 2010" — should also run on modern hardware and software in order to check one can obtain the same scientific results in the future. [ 1] https://reproducible-builds.org/contribute/ [ 2] https://reproducible-builds.org/events/Marrakesh2019/ [ 3] https://osem.seagl.org/conferences/seagl2019/program/proposals/671 [ 4] https://seagl.org [ 5] https://manifest.fm [ 6] https://manifest.fm/14 [ 7] http://rescience.github.io/ [ 8] https://rescience.github.io/ten-years/ Upstream news ============= Mike Hommey pushed a change to Mozilla build system [9] to add and print error messages when differences are found between builds as requested in bug #1597903 [10]. There was fresh activity on an old pull request for the OCaml [11] programming language regarding the usage and adoption of the BUILD_PATH_PREFIX_MAP environment variable [12] that is used to ensure that software packages do not embed build-time paths into generated files. On the pull request [13] in question, Gabriel Scherer [14] was kind enough to provide many helpful examples on how to use the rewrite rules [15]. Jan Nieuwenhuizen announced the release of GNU Mes 0.21 [16] and Jeremiah Orians announced [17] the release of mescc-tools-seed [18] version 1.1: > Capable of bootstrapping from a simple hex assembler all the way > to a cross-platform C compiler Work is still ongoing [to] result > in a full bootstrap from a 357 byte bootstrap binary all the way > to GCC. Hervé Boutemy [19] announced the release of three base Apache Maven [20] plugins (source, .jar, and assembly) to get reproducible Builds as a "direct output" from this build system. For more information, please see the "Configuring for Reproducible Builds" [21] section of their documentation. Eli Schwartz reported a bug against the GNU groff [22] typesetting system for incomplete SOURCE_DATE_EPOCH [23] environment variable support; the output files appeared to be embedding the build timezone [24]. [ 9] https://hg.mozilla.org/integration/autoland/rev/cd6b79c74b23 [10] https://bugzilla.mozilla.org/show_bug.cgi?id=1597903 [11] https://ocaml.org/ [12] https://reproducible-builds.org/specs/build-path-prefix-map/ [13] https://github.com/ocaml/ocaml/pull/1515 [14] http://gallium.inria.fr/~scherer/ [15] https://github.com/ocaml/ocaml/pull/1515#issuecomment-559035723 [16] https://lists.reproducible-builds.org/pipermail/rb-general/2019-November/00… [17] https://lists.reproducible-builds.org/pipermail/rb-general/2019-November/00… [18] https://github.com/oriansj/mescc-tools-seed [19] http://people.apache.org/~hboutemy/ [20] https://maven.apache.org/ [21] https://maven.apache.org/guides/mini/guide-reproducible-builds.html [22] https://www.gnu.org/software/groff/ [23] https://reproducible-builds.org/specs/source-date-epoch/ [24] https://savannah.gnu.org/bugs/?57218 Distribution work ================= Arch Linux ---------- A slight but temporary decline in the Arch Linux reproducibility status [25] was determined to be due to a bug in the continuous integration framework where one build was building with --nocheck whilst the other did not, resulting in the test dependencies being installed on one build. This led to differences in the BUILDINFO file which records the build dependencies. Morten Linderud (*Foxboron*) wrote a blog post [26] on the progress of reproducible builds for Arch packages, including how to reproduce packages and a roadmap of future of work. The standard Arch development tools package (devtools) now contains a new tool called makerepropkg which can reproduce a package from the Arch repositories given a seed PKGBUILD file. A lot of work has been put into getting the "[core]" system more reproducible; every package has been rebuilt [27] with a new version of pacman [28] which resolved a previous issue with storing the package size. Build failures and download issues have also been resolved which have lead to an increase of reproducible packages in this distributions continuous integration setup [29]. [25] https://tests.reproducible-builds.org/archlinux/archlinux.png [26] https://linderud.dev/blog/reproducible-arch-linux-packages/ [27] https://lists.archlinux.org/pipermail/arch-dev-public/2019-November/029721.… [28] https://wiki.archlinux.org/index.php/pacman [29] https://tests.reproducible-builds.org/archlinux/archlinux.html openSUSE -------- Bernhard M. Wiedemann posted a summary of openSUSE updates for 2019 [30] including rpm, a high level openSUSE status [31] and fixing problems with .pyc files [32] which is also relevant to Arch Linux. The report also summarises the current reproducibility statu and bernhard also published his monthly Reproducible Builds status update too [33]. [30] https://lists.reproducible-builds.org/pipermail/rb-general/2019-November/00… [31] https://bugzilla.opensuse.org/show_bug.cgi?id=1133809 [32] https://bugzilla.opensuse.org/show_bug.cgi?id=1094323 [33] https://lists.opensuse.org/opensuse-factory/2019-11/msg00370.html Debian ------ Thorsten Glaser [34] filed a bug against the debhelper packaging library [35] to request that it sets and exports a umask [36] of "022" for all operations as a possible "harmonisation potential". A varying umask can result in unreproducible packages as the file permissions on the build system can be embedded into archives generated by the build system. Chris Lamb categorised a large number of packages and issues in the Reproducible Builds "notes [37]" repository, including adding a new ocaml_dune_captures_build_path toolchain issue [38]. Vagrant Cascadian filed a bug against the Lintian [39] Debian static analyser for Debian packages to request that it checks for missing and/or unsigned .buildinfo files [40]. He also uploaded the latest version [41] of GNU Mes [42] to the *unstable* distribution. [34] https://www.mirbsd.org/ [35] https://bugs.debian.org/944691 [36] https://en.wikipedia.org/wiki/Umask [37] https://salsa.debian.org/reproducible-builds/reproducible-notes/activity [38] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/c12c… [39] https://lintian.debian.org/ [40] https://bugs.debian.org/944707 [41] https://tracker.debian.org/news/1080851/accepted-mes-020-1-source-amd64-int… [42] https://www.gnu.org/software/mes/ Other ----- Natanael Copa (@n_copa) posted on Twitter that he was finally able to make a fully reproducible package [44] for Alpine Linux [45]. The NixOS [46] distribution announced that they plan to run a Christmas Hackathon [47] hosted by Smarkets [48] in London, England on 9th December. [44] https://twitter.com/n_copa/status/1192447768855482369 [45] https://alpinelinux.org/ [46] https://nixos.org/ [47] https://www.meetup.com/NixOS-London/events/266848767 [48] https://smarkets.com/ Software development ==================== Upstream patches ---------------- The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Arnout Engelen: * OpenSC [49] (generate consistent DocBook [50] identifiers) [49] https://github.com/OpenSC/OpenSC/pull/1839 [50] https://docbook.org * Bernhard M. Wiedemann: * abseil-cpp [51] (sort the output of find/readdir(2)) * afl [52] (date) * brp-check-suse [53] (to strip link-time optimisation (LTO) [54] data from .o object files) * buzztrax [55] (report a parallelism/nondeterminism issue from GTK-Doc [56]) * cardpeek [57] (fix a previous patch) * cecilia [58] (strip date and time in a .png image file) * lib3270 [59] (merged; date) * maven-plugin-bundle [60] (fix a Java date) * nulloy [61] (.zip issue, already filed upstream [62]) * opencensus-cpp [63] (sort the result of find/readdir(2)) * OpenSC [64] (generate consistent DocBook [65] identifiers) * pcc [66] (fix a build failure from LTO [67] in .a archive files) * perl-HTTP-Cookies [68] (fix a build failure in 2025) * pocl [69] (report compile-time CPU detection) * python-oslo.reports [70] (drop unnecessary files with randomness) * sql-parser [71] (sort find/readdir(2)) * vim [72] (report a build failure when built without parallelism) * Various updates to the RPM package manager [73]: * #931 [74] — Keep at least one changelog entry to set SOURCE_DATE_EPOCH [75]. * #933 [76] — Regression fix so we can override the Build Date header again. * #936 [77] — Sort to avoid ordering issues from parallel header generation. * #944 [78] — Provide a cleaner solution for bug #936 [79] listed above. [51] https://build.opensuse.org/request/show/750468 [52] https://github.com/vanhauser-thc/AFLplusplus/pull/122 [53] https://github.com/openSUSE/brp-check-suse/pull/29 [54] https://en.wikipedia.org/wiki/Interprocedural_optimization [55] https://github.com/Buzztrax/buzztrax/issues/89 [56] https://www.gtk.org/gtk-doc/ [57] https://build.opensuse.org/request/show/746648 [58] https://build.opensuse.org/request/show/750463 [59] https://github.com/PerryWerneck/lib3270/pull/3 [60] https://github.com/apache/felix/pull/209 [61] https://build.opensuse.org/request/show/746033 [62] https://github.com/nulloy/nulloy/pull/149 [63] https://build.opensuse.org/request/show/751817 [64] https://github.com/OpenSC/OpenSC/pull/1839 [65] https://docbook.org [66] https://build.opensuse.org/request/show/745529 [67] https://en.wikipedia.org/wiki/Interprocedural_optimization [68] https://github.com/libwww-perl/HTTP-Cookies/pull/56 [69] https://github.com/pocl/pocl/issues/793 [70] https://review.opendev.org/693327 [71] https://github.com/hyrise/sql-parser/pull/134 [72] https://bugzilla.opensuse.org/show_bug.cgi?id=1157623 [73] https://rpm.org/ [74] https://github.com/rpm-software-management/rpm/pull/931 [75] https://reproducible-builds.org/specs/source-date-epoch/ [76] https://github.com/rpm-software-management/rpm/pull/933 [77] https://github.com/rpm-software-management/rpm/pull/936 [78] https://github.com/rpm-software-management/rpm/pull/944 [79] https://github.com/rpm-software-management/rpm/pull/936 * Chris Lamb: * #943954 for against tm-align [81]. * #943956 for snakemake [83] (forwarded upstream [84]). * #944131 for splitpatch [86] (forwarded upstream [87]). * #944214 filed against libaqbanking [89]. * #944520 for isbg [91] (forwarded upstream [92]). * #944782 for python-sybil [94] (forwarded upstream [95]). * #945105 filed against intel-gpu-tools [97]. * #945576 filed against superlu-dist [99]. * #945822 filed against liblopsub [101]. * genpy [102] [81] https://tracker.debian.org/pkg/tm-align [83] https://tracker.debian.org/pkg/snakemake [84] https://github.com/snakemake/snakemake/pull/80 [86] https://tracker.debian.org/pkg/splitpatch [87] https://github.com/jaalto/splitpatch/pull/8 [89] https://tracker.debian.org/pkg/libaqbanking [91] https://tracker.debian.org/pkg/isbg [92] https://github.com/isbg/isbg/pull/139 [94] https://tracker.debian.org/pkg/python-sybil [95] https://github.com/cjw296/sybil/pull/18 [97] https://tracker.debian.org/pkg/intel-gpu-tools [99] https://tracker.debian.org/pkg/superlu-dist [101] https://tracker.debian.org/pkg/liblopsub [102] https://github.com/ros/genpy/pull/110#event-2768597322 * Vagrant Cascadian: * #944694 filed against resource-agents [104] (forwarded upstream [105]). [104] https://tracker.debian.org/pkg/resource-agents [105] https://github.com/ClusterLabs/resource-agents/commit/088707c81b7ddfc117490… diffoscope ---------- diffoscope [106] is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure and is essential for identifying fixes and causes of non-deterministic behaviour. This month versions 131, 132 and 133 was uploaded to Debian unstable by Chris Lamb. He also made the following changes: * New features / improvements: * Allow all possible .zip file variations to return from external tools with non-zero exit codes, not just known types we can identify (e.g. Java .jmod and .jar files). (#78 [107]) * Limit .dsc and .buildinfo file matching to files in ASCII or UTF-8 format. (#77 [108]) * Bump the previous max_page_size limit from 400 kB to 4 MB. [109] * Clarify in the HTML and text outputs that the limits are per- format, not global. (#944882 [110]) * Don't use line-based buffering when communicating with subprocesses in "binary" mode. (#75 [111]) * Regression fixes: * Correct the substitution/filtering of paths in ELF output to avoid unnecessary differences depending on the path name provided and commandline. (#945572 [112]) * Silence/correct a Python SyntaxWarning [113] message due to incorrectly comparing an integer by identity vs. equality. (#945531 [114]) * Testsuite improvements: * Refresh the OCaml [115] test fixtures to support versions greater than 4.08.1. [116] * Update an Android manifest [117] test to reflect that parsed XML attributes are returned in a new/sorted manner under Python 3.8. [118] * Dramatically Truncate the tcpdump [119] expected diff to 8KB from ~600KB to reduce the size of the release tarball. [120] * Add a self-test to encourage that new test data files are generated dynamically or at least no new ones are added without an explicit override. [121] * Add a comment that the text_ascii1 and text_ascii2 fixture files are used in multiple tests so is not trivial to remove/replace them. [122] * Drop two more test fixture files for the directory tests. [123] * Don't run our self-test against the output of the Black source code reformatter [124] with versions earlier than "ours" as it will generate different results. [125] * Update an XML test for Python 3.8. [126] * Drop unused an unused BASE_DIR global. [127] * Code improvements: * Rework a long string of or statements into a loop with a break. [128] * Update code to reflect the latest version of the Black [129] source code reformatter. [130] * Correct a reference to the .rdx extension suffix in a comment. [131] [106] https://diffoscope.org [107] https://salsa.debian.org/reproducible-builds/diffoscope/issues/78 [108] https://salsa.debian.org/reproducible-builds/diffoscope/issues/77 [109] https://salsa.debian.org/reproducible-builds/diffoscope/commit/7d6daf7 [110] https://bugs.debian.org/944882 [111] https://salsa.debian.org/reproducible-builds/diffoscope/issues/75 [112] https://bugs.debian.org/945572 [113] https://docs.python.org/3/library/exceptions.html#SyntaxWarning [114] https://salsa.debian.org/reproducible-builds/diffoscope/commit/ecef7e6 [115] https://ocaml.org/ [116] https://salsa.debian.org/reproducible-builds/diffoscope/commit/4754277 [117] https://developer.android.com/guide/topics/manifest/manifest-intro [118] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c01dc80 [119] https://www.tcpdump.org/ [120] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9c66d5f [121] https://salsa.debian.org/reproducible-builds/diffoscope/commit/1c4b127 [122] https://salsa.debian.org/reproducible-builds/diffoscope/commit/69a8cc7 [123] https://salsa.debian.org/reproducible-builds/diffoscope/commit/a0f7a11 [124] https://black.readthedocs.io/en/stable/ [125] https://salsa.debian.org/reproducible-builds/diffoscope/commit/aefa5a3 [126] https://salsa.debian.org/reproducible-builds/diffoscope/commit/06d11dd [127] https://salsa.debian.org/reproducible-builds/diffoscope/commit/02497c9 [128] https://salsa.debian.org/reproducible-builds/diffoscope/commit/477584e [129] https://black.readthedocs.io/en/stable/ [130] https://salsa.debian.org/reproducible-builds/diffoscope/commit/75ea5c3 [131] https://salsa.debian.org/reproducible-builds/diffoscope/commit/4a00369 Other contributions were also made from: * Jelle van der Waa: * Add support for comparing .zst files created by Zstandard [132] compression algorithm. (#34) [133] * Mattia Rizzolo: * Install python3-all whilst running the autopkgtests [134] as we want to run the tests against all supported Python versions. [135] * Use apt-get instead of apt in our Dockerfile. [136] * Add zstd to our test dependencies after the resolution of issue #34. [138] [132] https://github.com/facebook/zstd [133] https://salsa.debian.org/reproducible-builds/issues/34 [134] https://ci.debian.net/ [135] https://salsa.debian.org/reproducible-builds/diffoscope/commit/cb74600 [136] https://salsa.debian.org/reproducible-builds/diffoscope/commit/a5c1364 [137] https://salsa.debian.org/reproducible-builds/issues/34 [138] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2c87e96 strip-nondeterminism -------------------- strip-nondeterminism [139] is our tool to remove specific non- deterministic results from a completed build. This month, Chris Lamb added file as a dependency for libfile-stripnondeterminism-perl (#945212 [140]) and moved away from deprecated $ADTTMP variable [... [141] and made two uploads in total (1.6.2-1 & 1.6.3-1). [139] https://tracker.debian.org/pkg/strip-nondeterminism [140] https://bugs.debian.org/945212 [141] https://salsa.debian.org/reproducible-builds/strip-nondeterminism.git/commi… Project website --------------- There was yet more effort put into our our website [142] this month, including: * Chris Lamb dropped a duplicated use the term "community" and other words [143][144], correcting the capitalisation of GitHub [145] & GitLab [146] [147] and corrected the use of an "an" [148]. * Daniel Edgecumbe added a section on initramfs and .cpio files [149] to our Archive Metadata [150] page. [151] * Hervé Boutemy added a link to Maven Guide to Configuring for Reproducible Builds [152] to our JVM page [153]. [154] * Jelle van der Waa added Arch Linux [155]-specific links for diffoscope [156] and friends to our *Tools* [157] page. [158] [142] https://reproducible-builds.org/ [143] https://salsa.debian.org/reproducible-builds/reproducible-website.git/commi… [144] https://salsa.debian.org/reproducible-builds/reproducible-website.git/commi… [145] https://github.com/ [146] https://gitlab.org/ [147] https://salsa.debian.org/reproducible-builds/reproducible-website.git/commi… [148] https://salsa.debian.org/reproducible-builds/reproducible-website.git/commi… [149] https://reproducible-builds.org/docs/archives/#initramfs-images [150] https://reproducible-builds.org/docs/archives/ [151] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/c5… [152] https://maven.apache.org/guides/mini/guide-reproducible-builds.html [153] https://reproducible-builds.org/docs/JVM/ [154] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/93… [155] https://archlinux.org/ [156] https://diffoscope.org/ [157] https://reproducible-builds.org/docs/tools/ [158] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0c… Test framework --------------- We operate a comprehensive Jenkins [159]-based testing framework that powers tests.reproducible-builds.org [160]. This month, the following changes were made: * Alexander Couzens (OpenWrt [161]): Fix a typo in the kirkwood architecture. [162] * Holger Levsen: * Debian: * Display newer suites first on pages showing the oldest build results. [163] * Use the fully qualified-domain name (FQDN) when specifying hostnames in our list of offline nodes [164]. [165] * Reflect that coccia.debian.org has changed IP address. [166] * Ignore the Maximum transmission Unit (MTU) [167] on eth0 when checking for host health. [168] * Perform the "/usr merge [169]" variation in the *unstable*, *experimental* and *bullseye* distributions but not on *buster*. [170] * FreeBSD [171]: Upgrade the test VM to FreeBSD 12.1. [172] * Arch Linux [173]: * Don't fail build jobs if the call to diffoscope --version fails; be a bit more verbose in the job output instead. [174][175] * Attempt to be less error prone when ending schroot [176] sessions. [177] * OpenWrt [178]: * Additionally build the brcm47xx, kirkwood, lantiq, mediatek, omap, sunxi and tegra targets. [... [179][180] * Make build job outputs easier to read and thus understand. [181] * Include the build target and subtarget in summary paragraphs at the top of report pages. [182] * Add a reminder to fix the job URL later. [183] * Misc: * Attempt to fix the PureOS [184] package set. [185] * Shorten a "HOWTO" header a tiny bit. [186] * Drop hack to fix the clock. [187] * Improve a script header; patches are even more welcome than bugs! [188] * Disable the use of the OpenSSH [189] ControlMaster feature to prevent Jenkins [190] killing connections. [191] * Make a number of improvements to our boilerplate texts/scripts. [192][... [193][... [194] * Jelle van der Waa: Skip running the Arch Linux [195] tests for continuous builds and rebuilds. [196][... [197] * Mattia Rizzolo: * Set the maximum size for HTML pages generated by diffoscope [198] to 1MB (current default is 400 KB). [199][200] * Update and improve the backup routines for the email relay system managing reproducible-builds.org. [201][202] * Vagrant Cascadian: * Ensure OpenSSH [203] authorized_keys files are processed in the correct directory regardless of where they are run from. [204] * Reduce the level of parallelism on armhf architectures with a lot of cores to reduce swapping on highly parallel builds, additionally ensuring level of parallelism are odd and even numbers on the first and second builds respectfully. [205] [159] https://jenkins.io/ [160] https://tests.reproducible-builds.org [161] https://openwrt.org/ [162] https://salsa.debian.org/qa/jenkins.debian.net/commit/369038e3 [163] https://salsa.debian.org/qa/jenkins.debian.net/commit/aa2ace2f [164] https://salsa.debian.org/qa/jenkins.debian.net/blob/HEAD/jenkins-home/offli… [165] https://salsa.debian.org/qa/jenkins.debian.net/commit/8266e978 [166] https://salsa.debian.org/qa/jenkins.debian.net/commit/25964ef8 [167] https://en.wikipedia.org/wiki/Maximum_transmission_unit [168] https://salsa.debian.org/qa/jenkins.debian.net/commit/1a549068 [169] https://wiki.debian.org/UsrMerge [170] https://salsa.debian.org/qa/jenkins.debian.net/commit/a9919410 [171] https://www.freebsd.org/ [172] https://salsa.debian.org/qa/jenkins.debian.net/commit/75e87e5d [173] https://www.archlinux.org/ [174] https://salsa.debian.org/qa/jenkins.debian.net/commit/de6dce51 [175] https://salsa.debian.org/qa/jenkins.debian.net/commit/8ca7241c [176] https://wiki.debian.org/Schroot [177] https://salsa.debian.org/qa/jenkins.debian.net/commit/32b9fe26 [178] https://openwrt.org/ [179] https://salsa.debian.org/qa/jenkins.debian.net/commit/c40b5f5b [180] https://salsa.debian.org/qa/jenkins.debian.net/commit/c40b5f5b [181] https://salsa.debian.org/qa/jenkins.debian.net/commit/6baae828 [182] https://salsa.debian.org/qa/jenkins.debian.net/commit/3f67936e [183] https://salsa.debian.org/qa/jenkins.debian.net/commit/98c3e2c6 [184] https://pureos.net/ [185] https://salsa.debian.org/qa/jenkins.debian.net/commit/c9066d49 [186] https://salsa.debian.org/qa/jenkins.debian.net/commit/e6494947 [187] https://salsa.debian.org/qa/jenkins.debian.net/commit/3a7ef746 [188] https://salsa.debian.org/qa/jenkins.debian.net/commit/ebefdd51 [189] https://openssh.org/ [190] https://jenkins.io/ [191] https://salsa.debian.org/qa/jenkins.debian.net/commit/81273bf4 [192] https://salsa.debian.org/qa/jenkins.debian.net/commit/c3759e97 [193] https://salsa.debian.org/qa/jenkins.debian.net/commit/35c2b25f [194] https://salsa.debian.org/qa/jenkins.debian.net/commit/2326d159 [195] https://archlinux.org/ [196] https://salsa.debian.org/qa/jenkins.debian.net/commit/e16b3ee7 [197] https://salsa.debian.org/qa/jenkins.debian.net/commit/97145223 [198] https://diffoscope.org [199] https://salsa.debian.org/qa/jenkins.debian.net/commit/0a56df32 [200] https://salsa.debian.org/qa/jenkins.debian.net/commit/e2ce7d08 [201] https://salsa.debian.org/reproducible-builds/rb-mailx-ansible/commit/9563162 [202] https://salsa.debian.org/reproducible-builds/rb-mailx-ansible/commit/734de20 [203] https://openssh.org/ [204] https://salsa.debian.org/qa/jenkins.debian.net/commit/0316d461 [205] https://salsa.debian.org/qa/jenkins.debian.net/commit/620fa54d The usual node maintenance was performed by Holger Levsen. [206][207][208][209] [206] https://salsa.debian.org/qa/jenkins.debian.net/commit/ec74bdf4 [207] https://salsa.debian.org/qa/jenkins.debian.net/commit/37d071e5 [208] https://salsa.debian.org/qa/jenkins.debian.net/commit/06c70874 [209] https://salsa.debian.org/qa/jenkins.debian.net/commit/c3cd0018 Contributing ============ If you are interested in contributing the Reproducible Builds project, please visit the Contribute [210] page on our website. However, you can get in touch with us via: * IRC: #reproducible-builds on irc.oftc.net. * Twitter: https://twitter.com/ReproBuilds * Mailing list: rb-general(a)lists.reproducible-builds.org [211] [210] https://reproducible-builds.org/contribute/ [211] https://lists.reproducible-builds.org/listinfo/rb-general This month's report was written by Arnout Engelen, Chris Lamb, Holger Levsen, Jelle van der Waa, Bernhard M. Wiedemann and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list. -- Chris Lamb https://puri.sm

1 0

October 2019 in Reproducible Builds
by Chris Lamb 07 Nov '19

07 Nov '19

==================================================================== o ⬋ ⬊ October 2019 in Reproducible Builds o o ⬊ ⬋ https://reproducible-builds.org/reports/2019-10/ o ==================================================================== Welcome to the October 2019 report from the Reproducible Builds project. :) In our monthly reports we attempt outline the most important things that we have been up to recently. As a reminder on what our little project is all about, whilst anyone can inspect the source code of free software for malicious changes most software is distributed to end users or servers as precompiled binaries. Reproducible builds tries to ensure that no changes have been made during these compilation processes by promising identical results are always generated from a given source, allowing multiple third-parties to come to a consensus on whether a build was compromised. In this month's report, we will cover: * Media coverage & conferences — Reproducible builds in Belfast & science * Reproducible Builds Summit 2019 — Registration & attendees, etc. * Distribution work — The latest work in Debian, OpenWrt, openSUSE, and more... * Software development — More diffoscope development, etc. * Getting in touch — How to contribute & get in touch If you are interested in contributing to our venture, please visit our *Contribute* page on our website. Media coverage & conferences ============================ Jonathan McDowell [2] gave an introduction on Reproducible Builds in Debian [3] at the Belfast Linux User Group. Whilst not strictly related to reproducible *builds*, Sean Gallagher from Ars Technica wrote an article entitled *Researchers find bug in Python script may have affected hundreds of studies* [6]: > A programming error in a set of Python scripts commonly used for > computational analysis of chemistry data returned varying results based > on which operating system they were run on. [ 2] https://www.earth.li/~noodles/ [ 3] https://www.meetup.com/belfast-lug/events/264951460/ [ 6] https://arstechnica.com/information-technology/2019/10/chemists-discover-cr… Reproducible Builds Summit 2019 =============================== Registration for our fifth annual Reproducible Builds summit that will take place between the 1st and 8th December in Marrakesh, Morocco has opened and invitations have been sent out. Similar to previous incarnations of the event, the heart of the workshop will be three days of moderated sessions with surrounding "hacking" days and will include a huge diversity of participants from Arch Linux, coreboot, Debian, F-Droid, GNU Guix, Google, Huawei, in-toto, MirageOS, NYU, openSUSE, OpenWrt, Tails, Tor Project and many more. We are still seeking additional sponsorship for the event. Sponsoring enables us to enable the attendance of people who would not otherwise be able to attend. If you or your company would be able to sponsor the event, please contact <info(a)reproducible-builds.org>. If you would like to learn more about the event and how to register, please visit our dedicated event page: https://reproducible-builds.org/events/Marrakesh2019/ Distribution work ================= GNU Guix [10] announced that they had significantly reduced the size of their "bootstrap seed" [11] by replacing binutils [12], GCC [13] and glibc [14] with smaller alternatives resulting in the package manager "possessing a formal description of how to build all underlying software" in a reproducible way from a mere 120MB seed. OpenWrt [15] is a Linux-based operating system targeting wireless network routers and other embedded devices. This month Paul Spooren (*aparcar*) posted a patch to their mailing list adding KCFLAGS to the kernel build flags [16] to make it easier to rebuild the official binaries. Bernhard M. Wiedemann posted his monthly Reproducible Builds status update [17] for the openSUSE [18] distribution which describes how rpm was updated [19] to run most builds with the -flto=auto argument, saving mirror disk space/bandwidth. In addition, maven-javadoc-plugin received a toolchain patch [20] (originating from Debian [21]) in order to normalise a date. [10] http://guix.gnu.org/ [11] https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/ [12] https://en.wikipedia.org/wiki/GNU_Binutils [13] https://gcc.gnu.org/ [14] https://www.gnu.org/software/libc/ [15] https://openwrt.org/ [16] https://lists.infradead.org/pipermail/openwrt-devel/2019-October/019248.html [17] https://lists.opensuse.org/opensuse-factory/2019-10/msg00367.html [18] https://opensuse.org/ [19] https://build.opensuse.org/request/show/732635 [20] https://build.opensuse.org/request/show/735873 § ,''`. : :' : `. `'` `- In Debian this month Didier Raboud (*OdyX*) started a discussion on the debian-devel [22] mailing list regarding building Debian source packages in a reproducible manner (thread index at [23]). In addition, Lukas Pühringer prepared an upload of in-toto [24], a framework to protect supply chain integrity by the Secure Systems Lab [25] at New York University [26] which was uploaded by Holger Levsen. Holger Levsen started a new section on the Debian wiki [27] to centralise to document the progress made on various Debian-specific reproducibility issues [28] and noticed that the "essential" package set in the *bullseye* distribution [29] became unreproducible again, likely due to a a bug in Perl [30] itself. Holger also restarted a discussion [31] on Debian bug #774415 [32] which requests that the devscripts collection of utilities that "make the life of a Debian package maintainer easier" adds a script/wrapper to enable easier end-user testing of whether a package is reproducible. Johannes Schauer (*josch*) explained that their mmdebstrap [33] tool can create bit-for-bit identical [34] Debian chroots [35] of the *unstable* and *buster* distributions for both the essential and minbase bootstrap "variants" [36], and Bernhard M. Wiedemann contributed to a discussion [37] regarding adding a "global" build switch to enable/disable Profile-Guided Optimisation [38] (PGO) and Link-time optimisation [39] in the dpkg-buildflags tool, nothing that "overall it is still very hard to get reproducible builds with PGO enabled." 64 reviews of Debian packages were added, 10 were updated and 35 were removed this month adding to our knowledge about identified issues[40]. Three new types were added by Chris Lamb (*lamby*): nondeterministic_output_in_code_generated_by_ros_genpy [41], nondeterministic_ordering_in_include_graphs_generated_by_doxygen[42] & nondeterministic_defaults_in_documentation_generated_by_pyth- on_traitlets [43]. Lastly, there was a far-reaching discussion regarding the correctness and suitability of setting the TZ environment variable [44] to UTC when it was noted that the value UTC0 [45] was "technically" more correct. [21] https://salsa.debian.org/java-team/maven-javadoc-plugin/blob/master/debian/… [22] https://lists.debian.org/debian-devel/ [23] https://lists.debian.org/debian-devel/2019/10/threads.html#00301 [24] https://in-toto.io/ [25] https://ssl.engineering.nyu.edu/ [26] https://engineering.nyu.edu/ [27] https://wiki.debian.org/ [28] https://wiki.debian.org/ReproducibleBuilds#Solved_issues [29] https://tests.reproducible-builds.org/debian/buster/amd64/pkg_set_essential… [30] https://bugs.debian.org/791362 [31] https://bugs.debian.org/774415#270 [32] https://bugs.debian.org/774415 [33] https://tracker.debian.org/mmdebstrap [34] https://lists.debian.org/debian-devel/2019/10/msg00101.html [35] https://en.wikipedia.org/wiki/Chroot [36] https://sources.debian.org/src/debootstrap/1.0.116/debootstrap.8/#L78-L85 [37] https://bugs.debian.org/940571#26 [38] https://en.wikipedia.org/wiki/Profile-guided_optimization [39] https://en.wikipedia.org/wiki/Interprocedural_optimization [40] https://tests.reproducible-builds.org/debian/index_issues.html [41] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/d219… [42] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/2577… [43] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/3ecd… [44] https://en.wikipedia.org/wiki/Environment_variable#Unix_2 [45] https://lists.reproducible-builds.org/pipermail/rb-general/2019-October/001… Software development ==================== Upstream patches ---------------- The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Bernhard M. Wiedemann: * keeperrl [46] (merged, date) * sphinx-doc [47] (nondeterminism from parallelism via Sphinx [48]) * vlc [49] (sort tar) * A number of expiring SSL testing certificates have been extended to 2049 to fix future builds: * python-M2Crypto [50] * python-aiosmtplib [51] * python-distlib [52] * python-geventhttpclient [53] * python-moto [54] (has a remaining year 2038 bug) * python-oslo.service [55] * python-thriftpy2 [56] [46] https://github.com/miki151/keeperrl/pull/1489 [47] https://github.com/sphinx-doc/sphinx/issues/6714 [48] http://www.sphinx-doc.org/en/master/ [49] https://mailman.videolan.org/pipermail/vlc-devel/2019-October/128188.html [50] https://gitlab.com/m2crypto/m2crypto/merge_requests/235 [51] https://github.com/cole/aiosmtplib/pull/92 [52] https://bitbucket.org/pypa/distlib/pull-requests/44/extend-test-cert-validi… [53] https://github.com/gwik/geventhttpclient/pull/115 [54] https://github.com/spulec/moto/pull/2500 [55] https://review.opendev.org/687822 [56] https://github.com/Thriftpy/thriftpy2/pull/91 * Chris Lamb (*lamby*): * #934698 filed against libchamplain (merged upstream [59]). * #941714 filed against bst-external. * #941715 filed against checkinstall. * #941716 filed against gobject-introspection. * #942005 filed against elph. * #942006 filed against squeak-plugins-scratch. * #942009 filed against stgit. (forwarded upstream [72]). * #942342 filed against traitlets (forwarded upstream [75]). * #942479 filed against frobby. * #942767 filed against python-oslo.reports. * #942847 filed against cloudkitty. * #942848 filed against designate. * #943471 filed against khard (forwarded upstream [90]). * #943674 filed against flask (forwarded upstream [93]). * #943694 filed against ros-genpy (forwarded upstream [96]). * #943829 filed against pmemkv. * #943954 filed against tm-align * #943956 filed against snakemake (forwarded upstream [103]) * spirv-tools [104]. * #942867 & #942870: Filed against r-base (not respecting nocheck and nodoc Debian build profiles [87]). [59] https://gitlab.gnome.org/GNOME/libchamplain/merge_requests/9 [72] https://github.com/ctmarinas/stgit/pull/43 [75] https://github.com/ipython/traitlets/pull/535 [87] https://wiki.debian.org/BuildProfileSpec [90] https://github.com/scheibler/khard/pull/233 [93] https://github.com/pallets/flask/pull/3408 [96] https://github.com/ros/genpy/pull/110 [103] https://github.com/snakemake/snakemake/pull/80 [104] https://github.com/KhronosGroup/SPIRV-Tools/pull/2982 * Mattias Ellert: * #942671 filed against doxygen. Lastly, a request from Steven Engler [107] to sort fields in the PKG-INFO files generated by the setuptools [108] Python module build utilities was resolved [109] by Jason R. Coombs [110] and Vagrant Cascadian added SOURCE_DATE_EPOCH [111] support to LTSP [112]'s manual page generation. [106] https://tracker.debian.org/pkg/doxygen [107] https://github.com/stevenengler [108] https://pypi.org/project/setuptools/ [109] https://github.com/pypa/setuptools/pull/1305#issuecomment-538810632 [110] https://www.jaraco.com/ [111] https://reproducible-builds.org/docs/source-date-epoch/ [112] https://ltsp.github.io/ strip-nondeterminism & reprotest -------------------------------- strip-nondeterminism [113] is our tool to remove specific non- deterministic results from successful builds. This month, Chris Lamb made a number of changes including uploading version 1.6.1-1 was to Debian unstable [114]. This dropped a bug_803503.zip test fixture as it is no longer compatible with the latest version of Perl's Archive::Zip [115] module (#940973) [116]. reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, Iñaki Malerba updated our Salsa CI [117] scripts [118] as well as adding a --control-build parameter [119]. Holger Levsen uploaded the package as 0.7.10, bumping the Debian "standards version" [120] to 4.4.1 [121]. [113] https://tracker.debian.org/pkg/strip-nondeterminism [114] https://tracker.debian.org/news/1071922/accepted-strip-nondeterminism-161-1… [115] https://metacpan.org/pod/Archive::Zip [116] https://bugs.debian.org/940973 [117] https://salsa.debian.org [118] https://salsa.debian.org/reproducible-builds/reprotest/commit/a547967 [119] https://salsa.debian.org/reproducible-builds/reprotest/commit/52f6eeb [120] https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-standard… [121] https://salsa.debian.org/reproducible-builds/reprotest/commit/fa0e286 diffoscope ---------- diffoscope [123] is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure [124] and is essential for identifying fixes and causes of non-deterministic behaviour. [123] https://diffoscope.org [124] https://tests.reproducible-builds.org/debian/reproducible.html This month, Chris Lamb (*lamby*) made the following changes, including uploading versions 126, 127, 128 and 129 to the Debian *unstable* distribution: * Disassembling and reporting on files related to the R (programming language) [125]): * Expose an .rdb file's absolute paths in the semantic/human- readable output, not hidden deep in a hexdump. [126] * Rework and refactor the handling of .rdb files with respect to locating the parallel .rdx prior to inspecting the file to ensure that we do not add files to the user's filesystem in the case of directly comparing two .rdb files or — worse — overwriting a file in is place. [127] * Query the container for the full path of the parallel .rdx file to the .rdb file as well as looking in the same directory. This ensures that comparing two Debian packages shows any varying path. [128] * Correct the matching of .rds files by also detecting newer versions of this file format. [129] * Don't read the site and user environment when comparing .rdx, .rdb or .rds files by using Rscript's --vanilla option. [130][131] * Ensure all object names are displayed, including ones beginning with a fullstop (.) [132] and sort package fields when dumping data from .rdb files [133]. * Mask/hide standard error when processing .rdb files [134] and don't include useless/misleading NULL when dumping data from them. [135] * Format package contents as foo = bar rather than using ugly and misleading brackets, etc. [136] and include the object's type [137]. * Don't pass our long script to parse .rdb files via the command line; use standard input [138]) instead. [139] * Call the deparse function to ensure that we do not error out and revert to a binary diff when processing .rdb files with internal "vector" types; they do not automatically coerce to strings. [140] * Other misc/cosmetic changes. [141][142][143] * Output/logging: * When printing an error from a command, format the command for the user. [144] * Truncate very long command lines when displaying them as an external source of data. [145] * When formatting command lines ensure newlines and other metacharacters appear escaped as \n, etc. 146][147] * When displaying the standard error from commands, ensure we use the escaped version. [148] * Use "exit code" over "return code" terminology when referring to UNIX error codes in displayed differences. [149] * Internal API: * Add ability to pass bytestring [150] input to external commands. [151] * Split out command-line formatting into a separate utility function. [152] * Add support for easily masking the standard error of commands. [153][154] * To match the libarchive [155] container, raise a KeyError exception if we request an invalid member from a directory. [156] * Correct string representation output in the traceback when we cannot locate a specific item in a container. [157] * Misc: * Move build-dependency on python-argcomplete to its Python 3 equivalent to facilitate Python 2.x removal. (#942967 [158]) * Track and report on missing Python modules. (#72 [159]) * Move from deprecated $ADTTMP to $AUTOPKGTEST_TMP in the autopkgtests [160]. [161] * Truncate the tcpdump expected diff to 8KB (from ~600KB). [162] * Try and ensure that new test data files are generated dynamically, ie. at least no new ones are added without "good" reasons. [163] * Drop unused BASE_DIR global in the tests. [164] [125] https://en.wikipedia.org/wiki/R_(programming_language [126] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f1e80ca [127] https://salsa.debian.org/reproducible-builds/diffoscope/commit/ea4c94a [128] https://salsa.debian.org/reproducible-builds/diffoscope/commit/322a568 [129] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2c9fbc1 [130] https://salsa.debian.org/reproducible-builds/diffoscope/commit/b8236d4 [131] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f8e436d [132] https://salsa.debian.org/reproducible-builds/diffoscope/commit/1f89609 [133] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9f60724 [134] https://salsa.debian.org/reproducible-builds/diffoscope/commit/0092be0 [135] https://salsa.debian.org/reproducible-builds/diffoscope/commit/cb83076 [136] https://salsa.debian.org/reproducible-builds/diffoscope/commit/343d01d [137] https://salsa.debian.org/reproducible-builds/diffoscope/commit/895f398 [138] https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin [139] https://salsa.debian.org/reproducible-builds/diffoscope/commit/07a013f [140] https://salsa.debian.org/reproducible-builds/diffoscope/commit/91d7029 [141] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c23651e [142] https://salsa.debian.org/reproducible-builds/diffoscope/commit/face6fb [143] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f23f2b4 [144] https://salsa.debian.org/reproducible-builds/diffoscope/commit/138aac1 [145] https://salsa.debian.org/reproducible-builds/diffoscope/commit/ecccd71 [146] https://salsa.debian.org/reproducible-builds/diffoscope/commit/691ce88 [147] https://salsa.debian.org/reproducible-builds/diffoscope/commit/338dbdf [148] https://salsa.debian.org/reproducible-builds/diffoscope/commit/bbfdb57 [149] https://salsa.debian.org/reproducible-builds/diffoscope/commit/6a8251d [150] https://docs.python.org/3/library/stdtypes.html#bytes [151] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c525ba9 [152] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f784d2c [153] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9b5c5fd [154] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2e33ad6 [155] https://www.libarchive.org/ [156] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c98e40f [157] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2478e9c [158] https://bugs.debian.org/942967 [159] https://salsa.debian.org/diffoscope/reproducible-builds/diffoscope [160] https://ci.debian.net/ [161] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f06c44f [162] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c6517e6 [163] https://salsa.debian.org/reproducible-builds/diffoscope/commit/e83b360 [164] https://salsa.debian.org/reproducible-builds/diffoscope/commit/7b44c80 In addition, Mattia Rizzolo updated our tests to run against all supported Python versions [165] and to exit with a UNIX exit status [166] of 2 instead of 1 in case of running out of disk space [167]. Lastly Vagrant Cascadian updated diffoscope 126 [168] and 129 [169] in GNU Guix [170], and updated inputs for additional test suite coverage [171]. trydiffoscope [172] is the web-based version of diffoscope [173] and this month Chris Lamb migrated the tool to depend on the python3-docutils package over python-docutils to allow for Python 2.x removal (#943293 [174]) as well as updating the packaging to the latest Debian standards and conventions [175][176][177]. [165] https://salsa.debian.org/reproducible-builds/diffoscope/commit/23c6112 [166] https://en.wikipedia.org/wiki/Exit_status [167] https://salsa.debian.org/reproducible-builds/diffoscope/commit/59267e8 [168] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c3704ecaa537f96dfca2f… [169] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d332fd860f89ed426a2b0… [170] https://guix.gnu.org/ [171] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=36f5f23c1af640782aa47… [172] https://try.diffoscope.org [173] https://diffoscope.org [174] https://bugs.debian.org/943293 [175] https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/75e8b14 [176] https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/95d7faf [177] https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/01df0a4 Project website --------------- URL: https://reproducible-builds.org/ There was yet more effort put into our our website this month, including Chris Lamb improving the formatting of reports [179][180][181][182][183] and tidying the new "Testing framework" [184] links [185], etc. In addition, Holger Levsen add the Tor Project's Reproducible Builds Manager [186] to our "Who is Involved? [187]" page and Mattia Rizzolo dropped a literal HTML element [188]. [179] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/db… [180] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/45… [181] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f3… [182] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/46… [183] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ad… [184] https://tests.reproducible-builds.org/debian/reproducible.html [185] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/13… [186] https://rbm.torproject.org/ [187] https://reproducible-builds.org/who/ [188] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8b… Test framework -------------- We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, the following changes were made: * Holger Levsen: * Debian-specific changes: * Add a script to ease powercycling x86 and arm64 nodes. [192][193] * Don't create suite-based directories for buildinfos.debian.net [194]. [195] * Make all four suites being tested shown in a single row on the performance page. [196] * OpenWrt changes: * Only run jobs every third day. [198] * Create jobs to run the reproducible_openwrt_rebuild.py script today and in the future. [199] * Mattia Rizzolo: * Add some packages that were lost while updating to *buster*. [200] * Fix the auto-offline functionality by checking the content of the permalinks file instead of following the lastSuccessfulBuild that no longer being updated. [201] * Paul Spooren (OpenWrt [202]): * Add a reproducible_common utilities file. [203] * Update the openwrt-rebuild script to to use schroot. [204] * Use unbuffered [205] Python output [206] as well as fixing newlines [207][208] The usual node maintenance was performed by Holger Levsen [209][210], Mattia Rizzolo [211][212][213] and Vagrant Cascadian [214][215][216]. [192] https://salsa.debian.org/qa/jenkins.debian.net/commit/8a69efc8 [193] https://salsa.debian.org/qa/jenkins.debian.net/commit/64d87e9e [194] https://buildinfos.debian.net/ [195] https://salsa.debian.org/qa/jenkins.debian.net/commit/e4a15fc4 [196] https://salsa.debian.org/qa/jenkins.debian.net/commit/cd8f363f [198] https://salsa.debian.org/qa/jenkins.debian.net/commit/d75af2d4 [199] https://salsa.debian.org/qa/jenkins.debian.net/commit/fa9febb0 [200] https://salsa.debian.org/qa/jenkins.debian.net/commit/69c765d7 [201] https://salsa.debian.org/qa/jenkins.debian.net/commit/a395b84f [202] https://openwrt.org [203] https://salsa.debian.org/qa/jenkins.debian.net/commit/94dcfb4c [204] https://salsa.debian.org/qa/jenkins.debian.net/commit/f73cf72f [205] https://eklitzke.org/stdout-buffering [206] https://salsa.debian.org/qa/jenkins.debian.net/commit/e2630fb7 [207] https://salsa.debian.org/qa/jenkins.debian.net/commit/dcbacce5 [208] https://salsa.debian.org/qa/jenkins.debian.net/commit/0443a133 [209] https://salsa.debian.org/qa/jenkins.debian.net/commit/cfbc58fb [210] https://salsa.debian.org/qa/jenkins.debian.net/commit/5f9424da [211] https://salsa.debian.org/qa/jenkins.debian.net/commit/9d3df188 [212] https://salsa.debian.org/qa/jenkins.debian.net/commit/88db9f0a [213] https://salsa.debian.org/qa/jenkins.debian.net/commit/5cdad8fd [214] https://salsa.debian.org/qa/jenkins.debian.net/commit/974bca24 [215] https://salsa.debian.org/qa/jenkins.debian.net/commit/8d4b533c [216] https://salsa.debian.org/qa/jenkins.debian.net/commit/3da81a76 Getting in touch ================ If you are interested in contributing the Reproducible Builds project, please visit our *Contribute* [217] page on our website. However, you can get in touch with us via: * Mailing list: rb-general(a)lists.reproducible-builds.org [218] * IRC: #reproducible-builds on irc.oftc.net. * Twitter: @ReproBuilds / https://twitter.com/ReproBuilds [217] https://reproducible-builds.org/contribute/ [218] https://lists.reproducible-builds.org/listinfo/rb-general This month's report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levesen and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list. Best wishes, -- o ⬋ ⬊ Chris Lamb o o reproducible-builds.org ⬊ ⬋ o

1 0

Bits from PureOS | Sharks!
by Jeremiah C. Foster 25 Oct '19

25 Oct '19

Hi, Summer is here on the East Coast of North America and that means another edition of Bits from PureOS. And no, I wasn't eaten by a shark last week on Cape Cod, but thanks for asking. To reiterate, the long-running topics we're responsible for are; 1. Reproducible builds 2. Supporting Pureboot and Purism hardware 3. Continuous delivery of PureOS 4. PureOS store (and application discovery) To that list I think we ought to add; 5. PureOS Security 6. Additional packages for PureOS We have already a security mailing list here: https://lists.puri.sm/listinfo/security We should likely develop a plan for its use. Other security topics perhaps ought to be discussed privately and I intend to kick off the topic with stakeholders. On the reproducible builds front you can follow the thread here; https://lists.puri.sm/pipermail/pureos-project/2019-July/000177.html What is happening is we're trying to adjust various time stamps that various tools insert into a given build. Those time stamps vary between builds (of course, because any two builds occur at different times). Lamby has patched many of the tools and I'm trying to find the right context to inform the tools that they're to use the standardised time stamp. Discussion with various folks has led us to cease maintaining PureBrowser. The reasons are pretty simple; we are investing in the GNOME ecosystem and GNOME web runs as a flatpak bringing in a bit of isolation to the browser which adds an additional layer of security. I think that we should make an announcement about this going forward in various places since there is often much discussion on browsers in our forums. Upcoming discussions on how to seed a build of PureOS for the L5 should be interesting, I plan to send out an invitation for discussing this in the next day or so. More info to come. Lastly, getting apps into PureOS needs to be better documented. I'm doing some packaging but am struggling with getting pbuilder to play nicely with my requests to pull in older versions of software. If we had clear instructions on how to build a PureOS image for pbuilder that would help me immensely. I'll reach out to folks regarding this since I'm certain that the knowledge needed is available. Also, uploads to Laniakea of certain packages are failing occasionally, something Guido has pointed out. We ought to speak with him to determine what the issues are and to smooth out the upload process. As always, feedback most welcome.

6 20

Lollypop in PureOS
by Andreas Ronnquist 07 Oct '19

07 Oct '19

I have uploaded new upstream versions of Lollypop to Debian unstable (1.1.14.16-4) and also I have uploaded release candidates (1.1.9*) to Debian experimental - I believe a stable upstream release of 1.2 is not far away now. How can I get these into Pureos? I believe the version in Pureos is way back at 1.1.14.11-1 (Thats what I can find on repo.pureos.net) - development in Lollypop upstream have been really intense lately, and I have tried my best to keep Debian up to date, at least in experimental when it comes to RC releases. Also, I am not really sure on how the different releases of PureOS is handled - I believe you have a special repository for the Phone OS, and keep non-phone OS separate from that? And lastly - This kind of question - maybe you would prefer this in tracker.pureos.net? Many thanks for all your work on free software stuff of all sorts - really appreciated! best regards -- Andreas Rönnquist mailinglists(a)gusnan.se andreas(a)ronnquist.net gusnan(a)debian.org [Please don't CC me, if I mail to a mailinglist, I am subscribed to it.]

2 1

PureOS 10 (Byzantium) is open!
by Matthias Klumpp 01 Oct '19

01 Oct '19

[ I messed up all version numbers in the previous mail, so here it is again, corrected ] Hi! As of now, PureOS 9 (Amber) is frozen, all future uploads to that suite will go to the respective "*-updates" pocket. This means that PureOS 11 (Byzantium) is now open for business! All uploads to "landing" will migrate to that suite, and so do all uploads to the "byzantium" suite. So, if you want to upload to the next PureOS development release, just put "byzantium" as suite name into your changelog file. If you want to switch to using byzantium, replace "amber" with "byzantium" in your PureOS sources.list. Please let me know if you experience any issues! Happy hacking! Matthias

2 2