Zandronum Chat @ irc.zandronum.com
#zandronum
Get the latest version: 3.0
Source Code

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0003238Doomseeker[All Projects] Suggestionpublic2017-09-01 13:492017-10-25 00:36
ReporterWubTheCaptain 
Assigned To 
PrioritylowSeveritytweakReproducibilityN/A
StatusclosedResolutionwon't fix 
Platformx86_64 (really cross-platform)OSDebian GNU/LinuxOS Versionbuster/sid
Product Version1.1 
Target VersionFixed in Version 
Summary0003238: Split Doomseeker's build dependencies off source archive distribution, distribute seperately
DescriptionWhen packaging for Debian GNU/Linux, inclusion of source for bzip2, zlib and libwadseeker (included in Doomseeker's tar archive distribution) may be unwanted in the Debian source distribution of Doomseeker. Debian has its own Build-Depends management for building, which checks the presence of required build dependencies.

The before mentioned source dependencies may be marked to be excluded in a Debian source package distribution (a version number ending in +ds, or +dfsg if non-free files are also removed), but it could simplify the Debian package distribution if these were not included upstream.
Steps To ReproduceDirectories "bzip2" and "zlib" are both present in the "doomseeker-1.1_src.tar.bz2" distribution, downloadable fromhttp://doomseeker.drdteam.org/files/doomseeker-1.1_src.tar.bz2 [^] .

"tar xfvj doomseeker-1.1_src.tar.bz2" extracts the archive.
Additional Informationlibwadseeker may be ok to include, but I would personally prefer it not be included justifying reduced burden on Debian package maintainer to exclude building libwadseeker from Doomseeker sources.
Attached Files

- Relationships
related to 0003297resolvedZalewa Update bundled (and possibly vulnerable) zlib 1.2.7 dependency 
related to 0003310closed qt-json dependency is duplicated to different files, deduplicate it 
related to 0003299closed Wadseeker: Hook into CMake's FindLibLZMA module 
child of 0003246assignedWubTheCaptain Debian packaging. 

-  Notes
User avatar (0018209)
WubTheCaptain (developer)
2017-09-01 14:37
edited on: 2017-09-01 18:37

zlib is also included in tools/updateinstaller/external/zlib. (It also configures RPATH, which would usually trigger Lintian error binary-or-shlib-defines-rpath but in my build it didn't.)

bzip2 also in tools/updateinstaller/external/bzip2.

User avatar (0018220)
Blzut3 (administrator)
2017-09-01 22:31

I think this one is going to have to be a "no can do." These are in the source directory since they're needed to make compiling on Windows easy which, as you might imagine, is way more important to us than distro requirements.

These sources are not normally used in Linux builds.
User avatar (0018222)
WubTheCaptain (developer)
2017-09-02 01:33

For what little it's worth, as an example Qt provides single and submodule source downloads for Qt 5.9 if you'd like to consider it. Debian keeps track of the submodule directory and each submodule individually.
User avatar (0018319)
Zalewa (developer)
2017-09-18 09:39

I will also have to agree with Blzut3. Providing extensive documentation on how to obtain dependencies instead of providing a solution that simply builds out-of-the box is a major step backwards.

Quote from "WubTheCaptain"
For what little it's worth, as an example Qt provides single and submodule source downloads for Qt 5.9 if you'd like to consider it.


So, essentially, would it be a viable solution to provide on our website a download of an alternative repackaged source archive for "Linux" with bzip2 and zlib removed?

Even if so, doing this properly wouldn't be as simple as just deleting the directories from the archive, as the "Force internal bzip2/zlib" CMake options would have to be removed from CMakeLists.txt also. These options rely on these directories being present.
User avatar (0018320)
WubTheCaptain (developer)
2017-09-18 10:33
edited on: 2017-09-18 10:33

I've not tried it myself on Windows, but I'm curious to understand how difficult would it be?

My suggestion here was to host the specific bzip2 and zlib dependencies athttp://doomseeker.drdteam.org/files/dependencies/ [^] , which already seems to have Qt4, Qt5 and OpenSSL 1.0.2. If bzip2 and zlib were hosted there, I thought compile instructions for Windows could instruct to acquire them easily from there and extract to a path instructed in COMPILE.Windows.txt.

In addition, I suggested the source archive of Doomseeker to not bundle libwadseeker, which is already hosted seperately too.

Quote from Zalewa
So, essentially, would it be a viable solution to provide on our website a download of an alternative repackaged source archive for "Linux" with bzip2 and zlib removed?


I think one kind of source archive for Doomseeker is the primary way to go. As I said, directories and individual files can be excluded from a Debian distribution. The "Force internal bzip2/zlib" option is not causing any harm.

The Qt approach I gave as an example distributes everything seperately, which in this case I suggested offering a source archive "all-in-one" including Qt, OpenSSL, Doomseeker, libwadseeker, bzip2 and zlib if it solves the Windows problem. (Whether this is possible, I don't know.)

Alternatively: Since bzip2 and zlib also live in the root directory of the archive, would it be an improvement to move them to a specific "external source" directory in the archive? That way it's one less definition of directories to exclude if really needed. It'd still consume extra disk space and bandwidth for the source download, though.

This ticket is more of a suggestion anyway, after all.

User avatar (0018321)
WubTheCaptain (developer)
2017-09-18 10:39
edited on: 2017-09-18 10:39

For clarification: Maybe some of my concern from bundled libraries raises from someone building from source mistakenly using an internal or possibly vulnerable version of bzip2/zlib, if the operating system already provides zlib1g and libbz2.

User avatar (0018329)
WubTheCaptain (developer)
2017-09-18 20:10
edited on: 2017-09-18 20:18

I just found out there's a Debian policy about this, section 4.13 in v4.1.0.0 "Convenience copies of code":https://www.debian.org/doc/debian-policy/#convenience-copies-of-code [^]

Quote from Debian Policy v4.1.0.0
Debian packages should not make use of these convenience copies unless the included package is explicitly intended to be used in this way.


This kind of brought me to thinking there's no quick way to visually determine from the build logs/output if the internal (convenience copy) bzip2/zlib were used or not. I have experience from Charybdis or Atheme IRC software builds telling that.

User avatar (0018330)
Zalewa (developer)
2017-09-18 21:40

CMake should state during configuration when internal versions will be used.
https://bitbucket.org/Doomseeker/doomseeker/src/ffe0a4a4c3ba625d3724e0fbd02d8aa0fb18667f/CMakeLists.txt?at=default&fileviewer=file-view-default#CMakeLists.txt-25 [^] - Lines 25 and 39.
User avatar (0018331)
Blzut3 (administrator)
2017-09-18 23:49

A thing to remember is that people don't read, so the closer we can get to "cmake .. && make" on all platforms the more accessible the source code is. I'm not going to make a move that potentially makes the project less welcoming to new developers.

While making a second source distribution seems trivial, if it is only Debian that needs a particular arrangement than in my opinion they can take on the task of maintaining that source distribution. Thankfully it sounds like their policy permits the convenience code to exist as long as it's not used which is the case.

With that said, if it helps your effort, I'd be happy to discuss relocating the libraries to a place in the tree where it is clearer that they're optional dependencies. If the message that Zalewa linked to is not clear enough I'm also happy to entertain ideas on how to make it easier for you to confirm that the code is not being used.

I'm also open to looking into making sure that it is feasible to build wadseeker and doomseeker as independent steps (while wadseeker should be able to build standalone, I don't think we have any way to use a system copy of wadseeker when building doomseeker). This should mostly be a matter of making sure our CMake usage is idiomatic anyway.
User avatar (0018333)
Zalewa (developer)
2017-09-19 05:39

Quote from "Blzut3"
With that said, if it helps your effort, I'd be happy to discuss relocating the libraries to a place in the tree where it is clearer that they're optional dependencies.

We have the dependencies/ directory where MSVC2008 prebuilt binaries of zlib and bzip2 reside. I think that we can remove those binaries - building them from sources is better because it supports all the compilers and not just one version of one compiler.

After removal, we can move the sources of bzip2 and zlib there and amend NOTES.txt to remove the "in binary format" part from the first sentence and parentheses from "(on Windows)".

We can also add extra sentence to NOTES.txt stating that on Linux these sources are not used by default as the libraries residing in the OS should be used.

If we ever add anything there that is mandatory on all systems, NOTES.txt will have to be revised accordingly.
User avatar (0018334)
WubTheCaptain (developer)
2017-09-19 06:27
edited on: 2017-09-19 07:19

Sorry for the confusion in original post.

Quote from Blzut3
I'd be happy to discuss relocating the libraries to a place in the tree where it is clearer that they're optional dependencies.


It's not entirely necessary, just a bit tidier. There's a lot of moving parts.

I didn't notice the zlib/bzip2 messages and oddly never read CMakeLists.txt source, because of CMake Deprecation Warnings filling the screen during cmake in the build step. That I should file another ticket about. (0003266)

I gave an example of Atheme earlier. Its ./configure step (comparable to cmake step in Doomseeker) has final output for many of the moving parts, like such:

Configuration:
    Atheme version       : 7.2.9
    Installation prefix  : /home/wub/atheme
    Module root directory: /home/wub/atheme
    Config directory     : ${prefix}/etc
    Logfile directory    : ${prefix}/var
    Data directory       : ${prefix}/etc
    PID directory        : ${prefix}/var
    Reproducible builds  : no
    Large network support: no
    OpenSSL SASL support : yes
    Contrib modules      : no
    Mowgli installation  : internal
    PCRE support         : no
    Perl support         : no
    QR Code support      : no
    CFLAGS               : -g -O2
    Internationalization : yes

Type make to build Atheme, and make install to install it.


(Atheme 7.2.9:https://github.com/atheme/atheme [^])

I've liked this experience, as an example. It gives immediate feedback at the end in accessible place for verifying all steps of build configuration.

Quote from Blzut3
I'm also open to looking into making sure that it is feasible to build wadseeker and doomseeker as independent steps


Yes, please.

User avatar (0018407)
WubTheCaptain (developer)
2017-09-27 23:16

I think this is a "won't fix".

Both Zalewa and Blzut3 have expressed the convenience copies of bzip2 and zlib should stay. The discussion has since deviated to other suggested improvements to the build system, which I created issue 0003280 for.

Debian's Policy Manual (v4.1.0.0, section 4.13) also doesn't seem to shun convenience copies completely, so their distribution in the source archive is fine.

The build system is also working correctly by preferring system installed libraries over convenience copies by default. This was previously not so easy to find out because of 0003266, but has been "fixed" since.
User avatar (0018471)
Zalewa (developer)
2017-10-07 12:20
edited on: 2017-10-07 12:21

I moved bzip2 and zlib to the dependencies/ directory where NOTES.txt explains what they are for and when they should be used.
https://bitbucket.org/Doomseeker/doomseeker/commits/58cc465927c77baaaa41e5d5089767e57cf20c06 [^]

User avatar (0018479)
WubTheCaptain (developer)
2017-10-08 01:11
edited on: 2017-10-08 01:21

Zalewa: Thank you, that's an improvement.

I'll make a note for 0003246 those two dependencies are "private forks" – not pristine, unchanged upstream sources – optional on GNU/Linux, but they will be nuked in Debian. Not for legal reasons, but practical/policy reasons.

Debian would require packaging those private forks into seperate source packages, but there is no real sensible reason to do that over using the pristine upstream sources already packaged into Debian.

See also Gentoo GNU/Linux's matter on the subject:https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies [^]

I understand they're mainly for convenience on Windows operating system, and that's fine by me. The bad downstream consequences of development time spent on analysis have been seen already, but it wasn't too much of effort. Hope you understand.


Issue Community Support
This issue is already marked as resolved.
If you feel that is not the case, please reopen it and explain why.
Supporters: No one explicitly supports this issue yet.
Opponents: No one explicitly opposes this issue yet.

- Issue History
Date Modified Username Field Change
2017-09-01 13:49 WubTheCaptain New Issue
2017-09-01 14:37 WubTheCaptain Note Added: 0018209
2017-09-01 16:30 Zalewa Relationship added child of 0003246
2017-09-01 18:37 WubTheCaptain Note Edited: 0018209 View Revisions
2017-09-01 22:31 Blzut3 Note Added: 0018220
2017-09-02 01:33 WubTheCaptain Note Added: 0018222
2017-09-18 09:39 Zalewa Note Added: 0018319
2017-09-18 10:33 WubTheCaptain Note Added: 0018320
2017-09-18 10:33 WubTheCaptain Note Edited: 0018320 View Revisions
2017-09-18 10:39 WubTheCaptain Note Added: 0018321
2017-09-18 10:39 WubTheCaptain Note Edited: 0018321 View Revisions
2017-09-18 20:10 WubTheCaptain Note Added: 0018329
2017-09-18 20:14 WubTheCaptain Note Edited: 0018329 View Revisions
2017-09-18 20:14 WubTheCaptain Note Edited: 0018329 View Revisions
2017-09-18 20:18 WubTheCaptain Note Edited: 0018329 View Revisions
2017-09-18 21:40 Zalewa Note Added: 0018330
2017-09-18 23:49 Blzut3 Note Added: 0018331
2017-09-19 05:39 Zalewa Note Added: 0018333
2017-09-19 06:27 WubTheCaptain Note Added: 0018334
2017-09-19 07:03 WubTheCaptain Note Edited: 0018334 View Revisions
2017-09-19 07:19 WubTheCaptain Note Edited: 0018334 View Revisions
2017-09-27 23:16 WubTheCaptain Note Added: 0018407
2017-09-27 23:16 WubTheCaptain Status new => closed
2017-09-27 23:16 WubTheCaptain Resolution open => won't fix
2017-10-07 12:20 Zalewa Note Added: 0018471
2017-10-07 12:21 Zalewa Note Edited: 0018471 View Revisions
2017-10-08 01:11 WubTheCaptain Note Added: 0018479
2017-10-08 01:20 WubTheCaptain Note Edited: 0018479 View Revisions
2017-10-08 01:21 WubTheCaptain Note Edited: 0018479 View Revisions
2017-10-08 02:20 WubTheCaptain Relationship added related to 0003297
2017-10-25 00:32 WubTheCaptain Relationship added related to 0003310
2017-10-25 00:36 WubTheCaptain Relationship added related to 0003299






Questions or other issues? Contact Us.

Links


Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker