MantisBT - Doomseeker
View Issue Details
0003841Doomseeker[All Projects] Cleanuppublic2020-07-03 07:012020-07-11 11:20
WubTheCaptain 
 
lowtweakalways
newopen 
0.6 Beta 
 
0003841: Source code tarball downloads from DRDTeam often use non-portable tar extensions (GNU)
Doomseeker and libwadseeker (numerous versions, since libwadseeker 0.2 or 0.5 and Doomseeker 0.6b) are distributed with numerous tar formats, for varying portability. Practically this could be an issue in the way that a mix of releases may only be extracted portably only with GNU tar (and libarchive bsdtar).
The expectation was to have all source code tar files available in one common tar format (before considering compression format), preferably a portable one.
For steps to reproduce on Doomseeker / Wadseeker, I knew beforehand there will be no short-hand compressed tarball filename extensions such as .txz or .tbz2.
file 1:5.38-5 on Debian bullseye/sid was used to recognize the type of data in a file, in these steps to reproduce.

To reproduce on GNU/Linux distributions, prepare around 233 MB of free space on disk to download tarballs from the Internet (DRDTeam's website). Once unextracted, around 456 MB of disk space will be required (as of Doomseeker 1.3.1 / Wadseeker 2.0).
Download the tarballs first:
$ wget -nv -r -A "*.tar*" -R "*.sig"https://doomseeker.drdteam.org/files/ [^]

# ...
$ cd doomseeker.drdteam.org/files/
$ ls -l
total 96228
drwxr-xr-x 2 wub wub    4096 Jul  3 06:20 dependencies
-rw-r--r-- 1 wub wub 7115012 May  8  2013 doomseeker-0.10b_src.tar.bz2
-rw-r--r-- 1 wub wub 7114701 Nov 30  2013 doomseeker-0.11.1b_src.tar.bz2
-rw-r--r-- 1 wub wub 7465478 Nov 14  2013 doomseeker-0.11b_src.tar.bz2
-rw-r--r-- 1 wub wub 7506532 Jun 24  2014 doomseeker-0.12.1b_src.tar.bz2
-rw-r--r-- 1 wub wub 7606408 Jan 10  2015 doomseeker-0.12.2b_src.tar.bz2
-rw-r--r-- 1 wub wub 7593942 May 16  2014 doomseeker-0.12b_src.tar.bz2
-rw-r--r-- 1 wub wub   36871 May 21  2009 doomseeker-0.1a_src.tar.bz2
-rw-r--r-- 1 wub wub 1478709 May 21  2009 doomseeker-0.2a_src.tar.bz2
-rw-r--r-- 1 wub wub 1499944 Aug 20  2009 doomseeker-0.3a_src.tar.bz2
-rw-r--r-- 1 wub wub 1032157 Oct 25  2009 doomseeker-0.4.1b_src.tar.bz2
-rw-r--r-- 1 wub wub 1028265 Oct 17  2009 doomseeker-0.4b_src.tar.bz2
-rw-r--r-- 1 wub wub 2413283 Mar 16  2010 doomseeker-0.5b_src.tar.bz2
-rw-r--r-- 1 wub wub 1900183 Aug 22  2010 doomseeker-0.6b_src.tar.bz2
-rw-r--r-- 1 wub wub 2051354 Nov 17  2010 doomseeker-0.7b_src.tar.bz2
-rw-r--r-- 1 wub wub 2085088 Jan 29  2012 doomseeker-0.8.1b_src.tar.bz2
-rw-r--r-- 1 wub wub 2118701 Jan  3  2012 doomseeker-0.8b_src.tar.bz2
-rw-r--r-- 1 wub wub 7509735 Aug 22  2012 doomseeker-0.9b_src.tar.bz2
-rw-r--r-- 1 wub wub 8415394 Mar 30  2015 doomseeker-1.0_src.tar.bz2
-rw-r--r-- 1 wub wub 8647902 Jun 11  2017 doomseeker-1.1_src.tar.bz2
-rw-r--r-- 1 wub wub 4020152 Oct 27  2018 doomseeker-1.2.tar.xz
-rw-r--r-- 1 wub wub 4084212 Jan 30 01:25 doomseeker-1.3.1.tar.xz
-rw-r--r-- 1 wub wub 4084952 Jul 29  2019 doomseeker-1.3.tar.xz
-rw-r--r-- 1 wub wub   23596 May 21  2009 libwadseeker-0.1_src.tar.bz2
-rw-r--r-- 1 wub wub   28448 Aug 20  2009 libwadseeker-0.2_src.tar.bz2
-rw-r--r-- 1 wub wub   19884 Oct 17  2009 libwadseeker-0.3_src.tar.bz2
-rw-r--r-- 1 wub wub   92364 Mar 16  2010 libwadseeker-0.4_src.tar.bz2
-rw-r--r-- 1 wub wub   75883 Aug 22  2010 libwadseeker-0.5_src.tar.bz2
-rw-r--r-- 1 wub wub   75992 Nov 17  2010 libwadseeker-0.6_src.tar.bz2
-rw-r--r-- 1 wub wub   86873 Jan 29  2012 libwadseeker-0.7.1_src.tar.bz2
-rw-r--r-- 1 wub wub   91692 Mar  4  2012 libwadseeker-0.7.2_src.tar.bz2
-rw-r--r-- 1 wub wub   91918 Apr 15  2012 libwadseeker-0.7.3_src.tar.bz2
-rw-r--r-- 1 wub wub   92255 Nov 14  2013 libwadseeker-0.7.4_src.tar.bz2
-rw-r--r-- 1 wub wub   86919 Jan  3  2012 libwadseeker-0.7_src.tar.bz2
-rw-r--r-- 1 wub wub   96710 Jun 24  2014 libwadseeker-0.8.1_src.tar.bz2
-rw-r--r-- 1 wub wub   96827 May 16  2014 libwadseeker-0.8_src.tar.bz2
-rw-r--r-- 1 wub wub  159040 Jun 11  2017 libwadseeker-1.1_src.tar.bz2
-rw-r--r-- 1 wub wub  157988 Oct 27  2018 libwadseeker-1.2.tar.xz
-rw-r--r-- 1 wub wub  364380 Jul 29  2019 libwadseeker-2.0.tar.xz
drwxr-xr-x 2 wub wub    4096 Jul  3 06:20 translations

Extract them (required packages in Debian are bzip2 and xz-utils packages):
$ bunzip2 ./*.bz2; unxz ./*.xz
$ ls -l
total 324768
drwxr-xr-x 2 wub wub     4096 Jul  3 06:20 dependencies
-rw-r--r-- 1 wub wub 24998912 May  8  2013 doomseeker-0.10b_src.tar
-rw-r--r-- 1 wub wub 25212928 Nov 30  2013 doomseeker-0.11.1b_src.tar
-rw-r--r-- 1 wub wub 25060864 Nov 14  2013 doomseeker-0.11b_src.tar
-rw-r--r-- 1 wub wub 25681920 Jun 24  2014 doomseeker-0.12.1b_src.tar
-rw-r--r-- 1 wub wub 25705984 Jan 10  2015 doomseeker-0.12.2b_src.tar
-rw-r--r-- 1 wub wub 25620992 May 16  2014 doomseeker-0.12b_src.tar
-rw-r--r-- 1 wub wub   206848 May 21  2009 doomseeker-0.1a_src.tar
-rw-r--r-- 1 wub wub  2146816 May 21  2009 doomseeker-0.2a_src.tar
-rw-r--r-- 1 wub wub  2326528 Aug 20  2009 doomseeker-0.3a_src.tar
-rw-r--r-- 1 wub wub  2051072 Oct 25  2009 doomseeker-0.4.1b_src.tar
-rw-r--r-- 1 wub wub  2030080 Oct 17  2009 doomseeker-0.4b_src.tar
-rw-r--r-- 1 wub wub  6078976 Mar 16  2010 doomseeker-0.5b_src.tar
-rw-r--r-- 1 wub wub  3860480 Aug 22  2010 doomseeker-0.6b_src.tar
-rw-r--r-- 1 wub wub  4392960 Nov 17  2010 doomseeker-0.7b_src.tar
-rw-r--r-- 1 wub wub  4810240 Jan 29  2012 doomseeker-0.8.1b_src.tar
-rw-r--r-- 1 wub wub  4782080 Jan  3  2012 doomseeker-0.8b_src.tar
-rw-r--r-- 1 wub wub 34775040 Aug 22  2012 doomseeker-0.9b_src.tar
-rw-r--r-- 1 wub wub 26958336 Mar 30  2015 doomseeker-1.0_src.tar
-rw-r--r-- 1 wub wub 28037120 Jun 11  2017 doomseeker-1.1_src.tar
-rw-r--r-- 1 wub wub 15237120 Oct 27  2018 doomseeker-1.2.tar
-rw-r--r-- 1 wub wub 15953920 Jan 30 01:25 doomseeker-1.3.1.tar
-rw-r--r-- 1 wub wub 15953920 Jul 29  2019 doomseeker-1.3.tar
-rw-r--r-- 1 wub wub   133120 May 21  2009 libwadseeker-0.1_src.tar
-rw-r--r-- 1 wub wub   157696 Aug 20  2009 libwadseeker-0.2_src.tar
-rw-r--r-- 1 wub wub   119808 Oct 17  2009 libwadseeker-0.3_src.tar
-rw-r--r-- 1 wub wub   972800 Mar 16  2010 libwadseeker-0.4_src.tar
-rw-r--r-- 1 wub wub   471040 Aug 22  2010 libwadseeker-0.5_src.tar
-rw-r--r-- 1 wub wub   471040 Nov 17  2010 libwadseeker-0.6_src.tar
-rw-r--r-- 1 wub wub   557568 Jan 29  2012 libwadseeker-0.7.1_src.tar
-rw-r--r-- 1 wub wub   594944 Mar  4  2012 libwadseeker-0.7.2_src.tar
-rw-r--r-- 1 wub wub   604160 Apr 15  2012 libwadseeker-0.7.3_src.tar
-rw-r--r-- 1 wub wub   595456 Nov 14  2013 libwadseeker-0.7.4_src.tar
-rw-r--r-- 1 wub wub   563200 Jan  3  2012 libwadseeker-0.7_src.tar
-rw-r--r-- 1 wub wub   655360 Jun 24  2014 libwadseeker-0.8.1_src.tar
-rw-r--r-- 1 wub wub   646144 May 16  2014 libwadseeker-0.8_src.tar
-rw-r--r-- 1 wub wub  1054720 Jun 11  2017 libwadseeker-1.1_src.tar
-rw-r--r-- 1 wub wub  1054720 Oct 27  2018 libwadseeker-1.2.tar
-rw-r--r-- 1 wub wub  2027520 Jul 29  2019 libwadseeker-2.0.tar
drwxr-xr-x 2 wub wub     4096 Jul  3 06:20 translations

Check and replicate the findings in OP:
$ file ./*.tar
./doomseeker-0.10b_src.tar:   tar archive
./doomseeker-0.11.1b_src.tar: tar archive
./doomseeker-0.11b_src.tar:   tar archive
./doomseeker-0.12.1b_src.tar: POSIX tar archive (GNU)
./doomseeker-0.12.2b_src.tar: tar archive
./doomseeker-0.12b_src.tar:   tar archive
./doomseeker-0.1a_src.tar:    tar archive
./doomseeker-0.2a_src.tar:    tar archive
./doomseeker-0.3a_src.tar:    tar archive
./doomseeker-0.4.1b_src.tar:  tar archive
./doomseeker-0.4b_src.tar:    tar archive
./doomseeker-0.5b_src.tar:    tar archive
./doomseeker-0.6b_src.tar:    POSIX tar archive (GNU)
./doomseeker-0.7b_src.tar:    POSIX tar archive (GNU)
./doomseeker-0.8.1b_src.tar:  POSIX tar archive
./doomseeker-0.8b_src.tar:    POSIX tar archive (GNU)
./doomseeker-0.9b_src.tar:    POSIX tar archive (GNU)
./doomseeker-1.0_src.tar:     tar archive
./doomseeker-1.1_src.tar:     POSIX tar archive (GNU)
./doomseeker-1.2.tar:         POSIX tar archive (GNU)
./doomseeker-1.3.1.tar:       POSIX tar archive (GNU)
./doomseeker-1.3.tar:         POSIX tar archive (GNU)
./libwadseeker-0.1_src.tar:   POSIX tar archive
./libwadseeker-0.2_src.tar:   tar archive
./libwadseeker-0.3_src.tar:   tar archive
./libwadseeker-0.4_src.tar:   tar archive
./libwadseeker-0.5_src.tar:   POSIX tar archive (GNU)
./libwadseeker-0.6_src.tar:   POSIX tar archive (GNU)
./libwadseeker-0.7.1_src.tar: tar archive
./libwadseeker-0.7.2_src.tar: tar archive
./libwadseeker-0.7.3_src.tar: POSIX tar archive (GNU)
./libwadseeker-0.7.4_src.tar: tar archive
./libwadseeker-0.7_src.tar:   POSIX tar archive (GNU)
./libwadseeker-0.8.1_src.tar: POSIX tar archive (GNU)
./libwadseeker-0.8_src.tar:   tar archive
./libwadseeker-1.1_src.tar:   POSIX tar archive (GNU)
./libwadseeker-1.2.tar:       POSIX tar archive (GNU)
./libwadseeker-2.0.tar:       POSIX tar archive (GNU)
https://dev.gentoo.org/~mgorny/articles/portability-of-tar-features.html [^]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html [^]

file 1:5.38-5 formats explained:
$ file doomseeker-{v7,ustar,pax,gnutar}.tar
doomseeker-v7.tar:     tar archive
doomseeker-ustar.tar:  POSIX tar archive
doomseeker-pax.tar:    POSIX tar archive
doomseeker-gnu.tar:    POSIX tar archive (GNU)

These tarballs were created like so (/home/wub/.local/src/doomseeker/doomseeker being the Git repository's master branch HEAD, with no build/ directory):
$ pwd
/home/wub/.local/src/doomseeker
$ tar --version
tar (GNU tar) 1.30
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
$ tar --format=v7 -cf doomseeker-v7.tar doomseeker/
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1chrono_1_1duration-members.html: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1condition__variable-members.html: 
file name
is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1condition__variable.html: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1thread_1_1id-members.html: 
file name is too
long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1lock__guard-members.html: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1fast__mutex-members.html: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1chrono_1_1duration.html: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/TinyThread/doc/html/classtthread_1_1recursive__mutex-members.html: 
file name is
too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial2/Tutorial2_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial1/Tutorial1_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial3/Tutorial3_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial8/Tutorial8_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial6/Tutorial6_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial4/Tutorial4_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial7/Tutorial7_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial9/Tutorial9_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: doomseeker/tools/updateinstaller/external/win32cpp/tutorials/Tutorial5/Tutorial5_2010.vcxproj.filters: 
file name is too long (max 99); not dumped
tar: Exiting with failure status due to previous errors
$ tar --format=ustar -cf doomseeker-ustar.tar doomseeker/
$ tar --format=pax -cf doomseeker-pax.tar doomseeker/
$ tar --format=gnu -cf doomseeker-gnu.tar doomseeker/

Less common tar formats such as star and sun formats were untested.
No tags attached.
Issue History
2020-07-03 07:01WubTheCaptainNew Issue
2020-07-03 07:05WubTheCaptainNote Added: 0021485
2020-07-03 07:07WubTheCaptainSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=13239#r13239
2020-07-03 07:07WubTheCaptainAdditional Information Updatedbug_revision_view_page.php?rev_id=13241#r13241
2020-07-03 07:08WubTheCaptainNote Added: 0021486
2020-07-03 07:14WubTheCaptainCategoryWebsite => Cleanup
2020-07-03 07:16WubTheCaptainReproducibilitysometimes => always
2020-07-03 07:16WubTheCaptainSummarySource code tarball downloads from DRDTeam use non-portable tar extensions (GNU) => Source code tarball downloads from DRDTeam often use non-portable tar extensions (GNU)
2020-07-11 11:20doomersaregaysNote Added: 0021493
2020-07-12 04:57Blzut3Note Deleted: 0021493

Notes
(0021485)
WubTheCaptain   
2020-07-03 07:05   
Changing existing tarballs on DRDTeam with the same name is probably not friendly for distributors, who might be using checksums in build/port files to verify the downloaded tarball is correct. And doing that might mess with timestamps.
From now on, the best we could perhaps do is ensure the tarball build script uses a portable, common tar format. Not v7 format obviously, because of too many restrictions. pax or ustar format, while in practice the old POSIX ustar format seems to be the most supported format in archivers (and the newer POSIX pax format not so much).
(0021486)
WubTheCaptain   
2020-07-03 07:08   
And let me just say that I am opposed to the GNU tar format, for not being POSIX.