0003238: Split Doomseeker's build dependencies off source archive distribution, distribute seperately

Notes
(0018209) WubTheCaptain (reporter) 2017-09-01 14:37 edited on: 2017-09-01 18:37	zlib is also included in tools/updateinstaller/external/zlib. (It also configures RPATH, which would usually trigger Lintian error binary-or-shlib-defines-rpath but in my build it didn't.) bzip2 also in tools/updateinstaller/external/bzip2.

(0018220) Blzut3 (administrator) 2017-09-01 22:31	I think this one is going to have to be a "no can do." These are in the source directory since they're needed to make compiling on Windows easy which, as you might imagine, is way more important to us than distro requirements. These sources are not normally used in Linux builds.

(0018222) WubTheCaptain (reporter) 2017-09-02 01:33	For what little it's worth, as an example Qt provides single and submodule source downloads for Qt 5.9 if you'd like to consider it. Debian keeps track of the submodule directory and each submodule individually.

(0018319) Zalewa (developer) 2017-09-18 09:39	I will also have to agree with Blzut3. Providing extensive documentation on how to obtain dependencies instead of providing a solution that simply builds out-of-the box is a major step backwards. Quote from "WubTheCaptain" For what little it's worth, as an example Qt provides single and submodule source downloads for Qt 5.9 if you'd like to consider it. So, essentially, would it be a viable solution to provide on our website a download of an alternative repackaged source archive for "Linux" with bzip2 and zlib removed? Even if so, doing this properly wouldn't be as simple as just deleting the directories from the archive, as the "Force internal bzip2/zlib" CMake options would have to be removed from CMakeLists.txt also. These options rely on these directories being present.

(0018320) WubTheCaptain (reporter) 2017-09-18 10:33 edited on: 2017-09-18 10:33	I've not tried it myself on Windows, but I'm curious to understand how difficult would it be? My suggestion here was to host the specific bzip2 and zlib dependencies at'http://doomseeker.drdteam.org/files/dependencies/ [^]' , which already seems to have Qt4, Qt5 and OpenSSL 1.0.2. If bzip2 and zlib were hosted there, I thought compile instructions for Windows could instruct to acquire them easily from there and extract to a path instructed in COMPILE.Windows.txt. In addition, I suggested the source archive of Doomseeker to not bundle libwadseeker, which is already hosted seperately too. Quote from Zalewa So, essentially, would it be a viable solution to provide on our website a download of an alternative repackaged source archive for "Linux" with bzip2 and zlib removed? I think one kind of source archive for Doomseeker is the primary way to go. As I said, directories and individual files can be excluded from a Debian distribution. The "Force internal bzip2/zlib" option is not causing any harm. The Qt approach I gave as an example distributes everything seperately, which in this case I suggested offering a source archive "all-in-one" including Qt, OpenSSL, Doomseeker, libwadseeker, bzip2 and zlib if it solves the Windows problem. (Whether this is possible, I don't know.) Alternatively: Since bzip2 and zlib also live in the root directory of the archive, would it be an improvement to move them to a specific "external source" directory in the archive? That way it's one less definition of directories to exclude if really needed. It'd still consume extra disk space and bandwidth for the source download, though. This ticket is more of a suggestion anyway, after all.

(0018321) WubTheCaptain (reporter) 2017-09-18 10:39 edited on: 2017-09-18 10:39	For clarification: Maybe some of my concern from bundled libraries raises from someone building from source mistakenly using an internal or possibly vulnerable version of bzip2/zlib, if the operating system already provides zlib1g and libbz2.

(0018329) WubTheCaptain (reporter) 2017-09-18 20:10 edited on: 2017-09-18 20:18	I just found out there's a Debian policy about this, section 4.13 in v4.1.0.0 "Convenience copies of code":'https://www.debian.org/doc/debian-policy/#convenience-copies-of-code [^]' Quote from Debian Policy v4.1.0.0 Debian packages should not make use of these convenience copies unless the included package is explicitly intended to be used in this way. This kind of brought me to thinking there's no quick way to visually determine from the build logs/output if the internal (convenience copy) bzip2/zlib were used or not. I have experience from Charybdis or Atheme IRC software builds telling that.

(0018330) Zalewa (developer) 2017-09-18 21:40	CMake should state during configuration when internal versions will be used. 'https://bitbucket.org/Doomseeker/doomseeker/src/ffe0a4a4c3ba625d3724e0fbd02d8aa0fb18667f/CMakeLists.txt?at=default&fileviewer=file-view-default#CMakeLists.txt-25 [^]' - Lines 25 and 39.

(0018331) Blzut3 (administrator) 2017-09-18 23:49	A thing to remember is that people don't read, so the closer we can get to "cmake .. && make" on all platforms the more accessible the source code is. I'm not going to make a move that potentially makes the project less welcoming to new developers. While making a second source distribution seems trivial, if it is only Debian that needs a particular arrangement than in my opinion they can take on the task of maintaining that source distribution. Thankfully it sounds like their policy permits the convenience code to exist as long as it's not used which is the case. With that said, if it helps your effort, I'd be happy to discuss relocating the libraries to a place in the tree where it is clearer that they're optional dependencies. If the message that Zalewa linked to is not clear enough I'm also happy to entertain ideas on how to make it easier for you to confirm that the code is not being used. I'm also open to looking into making sure that it is feasible to build wadseeker and doomseeker as independent steps (while wadseeker should be able to build standalone, I don't think we have any way to use a system copy of wadseeker when building doomseeker). This should mostly be a matter of making sure our CMake usage is idiomatic anyway.

(0018333) Zalewa (developer) 2017-09-19 05:39	Quote from "Blzut3" With that said, if it helps your effort, I'd be happy to discuss relocating the libraries to a place in the tree where it is clearer that they're optional dependencies. We have the dependencies/ directory where MSVC2008 prebuilt binaries of zlib and bzip2 reside. I think that we can remove those binaries - building them from sources is better because it supports all the compilers and not just one version of one compiler. After removal, we can move the sources of bzip2 and zlib there and amend NOTES.txt to remove the "in binary format" part from the first sentence and parentheses from "(on Windows)". We can also add extra sentence to NOTES.txt stating that on Linux these sources are not used by default as the libraries residing in the OS should be used. If we ever add anything there that is mandatory on all systems, NOTES.txt will have to be revised accordingly.

(0018334) WubTheCaptain (reporter) 2017-09-19 06:27 edited on: 2017-09-19 07:19	Sorry for the confusion in original post. Quote from Blzut3 I'd be happy to discuss relocating the libraries to a place in the tree where it is clearer that they're optional dependencies. It's not entirely necessary, just a bit tidier. There's a lot of moving parts. I didn't notice the zlib/bzip2 messages and oddly never read CMakeLists.txt source, because of CMake Deprecation Warnings filling the screen during cmake in the build step. That I should file another ticket about. (0003266) I gave an example of Atheme earlier. Its ./configure step (comparable to cmake step in Doomseeker) has final output for many of the moving parts, like such: Configuration: Atheme version : 7.2.9 Installation prefix : /home/wub/atheme Module root directory: /home/wub/atheme Config directory : ${prefix}/etc Logfile directory : ${prefix}/var Data directory : ${prefix}/etc PID directory : ${prefix}/var Reproducible builds : no Large network support: no OpenSSL SASL support : yes Contrib modules : no Mowgli installation : internal PCRE support : no Perl support : no QR Code support : no CFLAGS : -g -O2 Internationalization : yes Type make to build Atheme, and make install to install it. (Atheme 7.2.9:'https://github.com/atheme/atheme [^]') I've liked this experience, as an example. It gives immediate feedback at the end in accessible place for verifying all steps of build configuration. Quote from Blzut3 I'm also open to looking into making sure that it is feasible to build wadseeker and doomseeker as independent steps Yes, please.

(0018407) WubTheCaptain (reporter) 2017-09-27 23:16	I think this is a "won't fix". Both Zalewa and Blzut3 have expressed the convenience copies of bzip2 and zlib should stay. The discussion has since deviated to other suggested improvements to the build system, which I created issue 0003280 for. Debian's Policy Manual (v4.1.0.0, section 4.13) also doesn't seem to shun convenience copies completely, so their distribution in the source archive is fine. The build system is also working correctly by preferring system installed libraries over convenience copies by default. This was previously not so easy to find out because of 0003266, but has been "fixed" since.

(0018471) Zalewa (developer) 2017-10-07 12:20 edited on: 2017-10-07 12:21	I moved bzip2 and zlib to the dependencies/ directory where NOTES.txt explains what they are for and when they should be used. 'https://bitbucket.org/Doomseeker/doomseeker/commits/58cc465927c77baaaa41e5d5089767e57cf20c06 [^]'

(0018479) WubTheCaptain (reporter) 2017-10-08 01:11 edited on: 2017-10-08 01:21	Zalewa: Thank you, that's an improvement. I'll make a note for 0003246 those two dependencies are "private forks" – not pristine, unchanged upstream sources – optional on GNU/Linux, but they will be nuked in Debian. Not for legal reasons, but practical/policy reasons. Debian would require packaging those private forks into seperate source packages, but there is no real sensible reason to do that over using the pristine upstream sources already packaged into Debian. See also Gentoo GNU/Linux's matter on the subject:'https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies [^]' I understand they're mainly for convenience on Windows operating system, and that's fine by me. The bad downstream consequences of development time spent on analysis have been seen already, but it wasn't too much of effort. Hope you understand.

This issue is already marked as resolved. If you feel that is not the case, please reopen it and explain why.
Supporters:	No one explicitly supports this issue yet.
Opponents:	No one explicitly opposes this issue yet.

Issue History
Date Modified	Username	Field	Change
2017-09-01 13:49	WubTheCaptain	New Issue
2017-09-01 14:37	WubTheCaptain	Note Added: 0018209
2017-09-01 16:30	Zalewa	Relationship added	child of 0003246
2017-09-01 18:37	WubTheCaptain	Note Edited: 0018209	View Revisions
2017-09-01 22:31	Blzut3	Note Added: 0018220
2017-09-02 01:33	WubTheCaptain	Note Added: 0018222
2017-09-18 09:39	Zalewa	Note Added: 0018319
2017-09-18 10:33	WubTheCaptain	Note Added: 0018320
2017-09-18 10:33	WubTheCaptain	Note Edited: 0018320	View Revisions
2017-09-18 10:39	WubTheCaptain	Note Added: 0018321
2017-09-18 10:39	WubTheCaptain	Note Edited: 0018321	View Revisions
2017-09-18 20:10	WubTheCaptain	Note Added: 0018329
2017-09-18 20:14	WubTheCaptain	Note Edited: 0018329	View Revisions
2017-09-18 20:14	WubTheCaptain	Note Edited: 0018329	View Revisions
2017-09-18 20:18	WubTheCaptain	Note Edited: 0018329	View Revisions
2017-09-18 21:40	Zalewa	Note Added: 0018330
2017-09-18 23:49	Blzut3	Note Added: 0018331
2017-09-19 05:39	Zalewa	Note Added: 0018333
2017-09-19 06:27	WubTheCaptain	Note Added: 0018334
2017-09-19 07:03	WubTheCaptain	Note Edited: 0018334	View Revisions
2017-09-19 07:19	WubTheCaptain	Note Edited: 0018334	View Revisions
2017-09-27 23:16	WubTheCaptain	Note Added: 0018407
2017-09-27 23:16	WubTheCaptain	Status	new => closed
2017-09-27 23:16	WubTheCaptain	Resolution	open => won't fix
2017-10-07 12:20	Zalewa	Note Added: 0018471
2017-10-07 12:21	Zalewa	Note Edited: 0018471	View Revisions
2017-10-08 01:11	WubTheCaptain	Note Added: 0018479
2017-10-08 01:20	WubTheCaptain	Note Edited: 0018479	View Revisions
2017-10-08 01:21	WubTheCaptain	Note Edited: 0018479	View Revisions
2017-10-08 02:20	WubTheCaptain	Relationship added	related to 0003297
2017-10-25 00:32	WubTheCaptain	Relationship added	related to 0003310
2017-10-25 00:36	WubTheCaptain	Relationship added	related to 0003299