MantisBT - Doomseeker
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0003952DoomseekerIP2Cpublic2021-12-20 15:412024-11-03 19:11
ReporterZalewa 
Assigned ToZalewa 
PrioritynormalSeverityminorReproducibilityN/A
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target Version1.4.0Fixed in Version1.4.0 
Summary0003952: Embed the IP2C copyright blurb into the IP2C database
DescriptionDoomseeker displays the copyright blurb for the current IP2C database in the "About" dialog. This is appropriate, but the way it's currently being done permits some scenarios where it will be inaccurate. This blurb is hardcoded and will always be displayed, even if:

Scenario 1: there is no IP2C database loaded.
Scenario 2: we change the source of the IP2C database (ticket 0003950) and will no longer use the MaxMind's DB.

Therefore, the copyright blurb should be embedded into the IpToCountry.dat file. If the database is in use, Doomseeker will extract that blurb from that file and display it in the "About" dialog instead of the hardcoded blurb.
Steps To Reproduce
Additional InformationThe current copyright blurb is:
Quote from blurb
This program uses GeoLite2 data for IP-to-Country (IP2C) purposes, available from'https://www.maxmind.com [^]'
Database and Contents Copyright (c) 2018 MaxMind, Inc.

This blurb is translatable to other languages.
TagsNo tags attached.
Relationships
related to 0003950closed Zalewa MaxMind no longer a viable source of IP2C data 
Attached Filespng About_230106-0013.png (33,809) 2023-01-06 15:30
https://zandronum.com/tracker/file_download.php?file_id=2822&type=bug
png
Issue History
Date ModifiedUsernameFieldChange
2021-12-20 15:41ZalewaNew Issue
2021-12-20 15:45ZalewaNote Added: 0021959
2021-12-20 15:46ZalewaRelationship addedrelated to 0003950
2021-12-20 15:47ZalewaNote Edited: 0021959bug_revision_view_page.php?bugnote_id=21959#r13488
2021-12-20 17:34Pol MStatusnew => confirmed
2021-12-20 22:09Pol MAssigned To => Pol M
2021-12-20 22:09Pol MStatusconfirmed => assigned
2022-01-19 02:45Pol MNote Added: 0022114
2022-01-19 02:45Pol MStatusassigned => needs review
2022-01-19 04:25WubTheCaptainNote Added: 0022115
2022-05-23 20:13ZalewaAssigned ToPol M => Zalewa
2022-05-23 20:13ZalewaStatusneeds review => assigned
2022-08-24 23:07ZalewaNote Added: 0022311
2022-08-24 23:07ZalewaStatusassigned => needs review
2022-08-25 10:06WubTheCaptainNote Added: 0022316
2022-08-25 10:07WubTheCaptainNote Edited: 0022316bug_revision_view_page.php?bugnote_id=22316#r13674
2022-08-25 11:08ZalewaNote Added: 0022317
2022-08-25 12:04WubTheCaptainNote Added: 0022320
2022-08-25 12:08WubTheCaptainNote Edited: 0022320bug_revision_view_page.php?bugnote_id=22320#r13676
2022-08-25 15:01ZalewaNote Added: 0022322
2022-08-25 15:01ZalewaStatusneeds review => assigned
2022-09-06 07:02ZalewaNote Added: 0022351
2022-09-06 07:02ZalewaStatusassigned => needs review
2022-09-07 05:28ZalewaTarget Version => 1.4.0
2022-09-07 11:21WubTheCaptainNote Added: 0022362
2022-09-07 11:34WubTheCaptainNote Added: 0022363
2022-09-07 11:36WubTheCaptainNote Edited: 0022363bug_revision_view_page.php?bugnote_id=22363#r13688
2022-09-07 11:36WubTheCaptainNote Edited: 0022363bug_revision_view_page.php?bugnote_id=22363#r13689
2022-09-07 11:36WubTheCaptainNote Edited: 0022363bug_revision_view_page.php?bugnote_id=22363#r13690
2022-09-07 11:39WubTheCaptainNote Added: 0022364
2022-09-07 11:42WubTheCaptainNote Edited: 0022364bug_revision_view_page.php?bugnote_id=22364#r13692
2022-09-07 11:55WubTheCaptainNote Added: 0022365
2022-09-07 11:57WubTheCaptainNote Edited: 0022365bug_revision_view_page.php?bugnote_id=22365#r13694
2022-09-07 12:04WubTheCaptainNote Added: 0022366
2022-09-07 12:04WubTheCaptainNote Edited: 0022366bug_revision_view_page.php?bugnote_id=22366#r13696
2022-09-07 12:07WubTheCaptainNote Added: 0022367
2022-09-07 12:25WubTheCaptainNote Edited: 0022367bug_revision_view_page.php?bugnote_id=22367#r13698
2023-01-05 12:06ZalewaNote Added: 0022636
2023-01-05 12:06ZalewaStatusneeds review => needs testing
2023-01-06 15:30KaminskyFile Added: About_230106-0013.png
2023-01-06 15:31KaminskyNote Added: 0022679
2023-01-06 20:07ZalewaStatusneeds testing => resolved
2023-01-06 20:07ZalewaFixed in Version => 1.4.0
2023-01-06 20:07ZalewaResolutionopen => fixed
2024-11-03 19:11ZalewaStatusresolved => closed

Notes
(0021959)
Zalewa   
2021-12-20 15:45   
(edited on: 2021-12-20 15:47)
The current specs are:'https://bitbucket.org/Doomseeker/doomseeker-blobs/src/da879d1f87668ef2bb14a7cba8b56b3af027feb2/IpToCountry.dat.SPECS.txt [^]'

Implementation notes (just an outline):

1. Bump the file structure version to 3. Older versions of Doomseeker will not read this database. Doomseeker needs to be adapted.
2. Improve the structure of the file to enable having the blurbs. Possibly add a header flag field to indicate what info is available in the database - imitate how the Zandronum servers send their challenge reply.
3. Add a structure for UTF-8 encoded blurbs for multiple languages.
4. The updates to the database will need to be deployed at a different URL to cover for the different version.
(0022114)
Pol M   
2022-01-19 02:45   
'https://bitbucket.org/Doomseeker/doomseeker-blobs/pull-requests/1/implement-new-mailfud-database [^]' python script

'https://bitbucket.org/Doomseeker/doomseeker/pull-requests/9/implement-the-new-file-format-and-display [^]' Doomseeker changes

The things left are:
- Update documentation with the new format
- Write the actual licenses, too tired to look at legal stuff :)
(0022115)
WubTheCaptain   
2022-01-19 04:25   
Quote from Pol M
https://bitbucket.org/Doomseeker/doomseeker/pull-requests/9/implement-the-new-file-format-and-display Doomseeker changes

Some unrelated formatting changes may not be needed or desired. Newlines, extra tabs, ... (IP2CParser::doReadDatabase, AboutDialog::copyrightVerboseNotice())
(0022311)
Zalewa   
2022-08-24 23:07   
The next Doomseeker version will migrate to version 3 of the IpToCountry.dat blob file. I've already deployed these files to the website, where they can be downloaded from. V3 sits alongside V2 on the website and should remain so to support the current and older Doomseeker versions (1.3.3 and older) which will still rely on V2. The generator can create both the V2 and V3 formats, which is intentional and should also remain so. Because the source of the DB is changed too, Doomseekers 1.3.3 and older will now display invalid URL and copyright note for the IP2C database. The V3 DB has sections for the copyright note and the origin URL, and the new V2 DBs will be built from the same source as V3.

Involved commits:

- Doomseeker:
1. Pol's much appreciated work to implement the V3 parser.
2. Sanitize the URL from the IP2C DB before showing it in the About dialog, just in case.
3. Refactor IP2CLoader and IP2CParser so that the parser does the parsing and doesn't do thread management.
4. Hardcode the about dialog's origin URL for version 2 of the IP2C database. I don't believe Doomseekers newer than 1.3.3 will ever use this, as they will use V3 where the origin URL is in the database, but it's there for completion.
5. Refactor: extract subparsers for V2 and V3 as classes separate from IP2CParser.
6. Fix: the IP2C V3 doesn't have the country names, so don't parse them - this is a bug in Pol's original commit (1).
7. Check the magic number of IP2C .dat and be more conservative about reading the file's header - just a minor improvement, not relevant to this ticket directly, but affects IP2C.
8. The IP2C updater will download V3 from now on.

- Doomseeker-blobs:
1. Pol's V3 generator, plus migration to Mailfud provider (0003950)
2. My continuation on the generator, plus the description of the format's specs.
3. A relevant change in the generator's filename. (0003950 again)
4. Minor cleanups and fixes.
5. V2 generator got bugged in the process, this fixes it.

- Website:
1. The IP2C files hosted on the website are renamed so that we can keep the V2 and V3 next to each other and also to detach the filenames from MaxMind's geolite2.
(0022316)
WubTheCaptain   
2022-08-25 10:06   
(edited on: 2022-08-25 10:07)
0003952:0022311:
Quote from Zalewa
Hardcode the about dialog's origin URL for version 2 of the IP2C database. I don't believe Doomseekers newer than 1.3.3 will ever use this, as they will use V3 where the origin URL is in the database, but it's there for completion.

This looks wrong to me (hardcoding to Mailfud), because the IpToCountry.dat blob can be loaded externally. Legacy V2 databases had free (pre-EULA) MaxMind databases distributed before Doomseeker 1.3.3, not via Mailfud (EULA'd).
(0022317)
Zalewa   
2022-08-25 11:08   
Quote from WubTheCaptain
because the IpToCountry.dat blob can be loaded externally

Just as it always was, so this was always wrong. Nothing changed in that regard. That's why we have V3 to fix that problem properly. And because Doomseeker shall use V3 from now on, this hardcoded case for V2 shouldn't trigger in normal operation unless the user goes about and mucks around with their files. Perhaps hardcoding this to MaxMind would be a bit more appropriate, but only a bit. The V2 database on the website is already replaced with the one sourced from Mailfud.
(0022320)
WubTheCaptain   
2022-08-25 12:04   
(edited on: 2022-08-25 12:08)
Quote from Zalewa
Perhaps hardcoding this to MaxMind would be a bit more appropriate, but only a bit.

Perhaps hardcoding it to a Doomseeker website hosted page would be the most appropriate for explanatory notes, in the event the support for (alternative) V2 databases stretches for much longer than foreseen.
(0022322)
Zalewa   
2022-08-25 15:01   
Good idea.
(0022351)
Zalewa   
2022-09-06 07:02   
The blobs generator will now also generate the .php file with licensing info that should be uploaded next to the .dat.gz and .md5 files to the ip2c directory on the website.

Blobs generator:'https://bitbucket.org/Doomseeker/doomseeker-blobs/commits/1401483906a6a6efede29f55eb7a0542165fdf46 [^]'

Doomseeker has been changed to point to this php for the licensing info if the loaded DB version is V2.

Doomseeker:'https://bitbucket.org/Doomseeker/doomseeker/commits/2950d7cd0ff98832a57a98dea4d004ce82a0c77c [^]'
(0022362)
WubTheCaptain   
2022-09-07 11:21   
My ideal scenario would also include information about the past free MaxMind databases (before Mailfud) on that page.
(0022363)
WubTheCaptain   
2022-09-07 11:34   
(edited on: 2022-09-07 11:36)
I also raise a concern neither Mailfud and Doomseeker satisfy MaxMind's attribution requirement by the license, as a derivative work.
Quote
This product includes GeoLite2 data created by MaxMind, available from
'https://www.maxmind.com. [^]'

I tried interpreting the meaning of CC BY-SA 4.0 3(a)(1)(A)(iii), but could not conclude if the text of CC BY-SA 4.0 or a hyperlink to it must also be included on the webpage.
(0022364)
WubTheCaptain   
2022-09-07 11:39   
(edited on: 2022-09-07 11:42)
Quote from WubTheCaptain
I tried interpreting the meaning of CC BY-SA 4.0 3(a)(1)(A)(iii), but could not conclude if the text of CC BY-SA 4.0 or a hyperlink to it must also be included on the webpage.

Oops, CC BY-SA 4.0 3(a)(1)(C) says yes. And that's not done on the webpage.
(0022365)
WubTheCaptain   
2022-09-07 11:55   
(edited on: 2022-09-07 11:57)
To summarize my two previous notes, I see the following license conditions are not met by IpToCountry.py and the associated generated webpage:
Quote from Creative Commons Attribution-ShareAlike 4.0 International
identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);

MaxMind is the "other" party designated to receive attribution. Specifically, their requested attribution is the "This product includes GeoLite2 data created by MaxMind ..." blurb (aforementioned).
Quote from Creative Commons Attribution-ShareAlike 4.0 International
indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.

Currently, only the license name is indicated, but no text or URI or hyperlink to it.
(0022366)
WubTheCaptain   
2022-09-07 12:04   
I did not check what loading the V3 database does in-program, but I assume from reading IpToCountry.py's license text array the same issues are pertinent there with the IP2C blob that is currently being distributed by Doomseeker and generated by IpToCountry.py. I read the V3 spec and don't see a need for a spec change, can modify the existing "Text" string in the "License" section.
(0022367)
WubTheCaptain   
2022-09-07 12:07   
(edited on: 2022-09-07 12:25)
What I did not understand from reading the V3 spec is if there can be multiple sections with the same Type BYTE, e.g. multiple Type 0 (license) sections.
0004027
(0022636)
Zalewa   
2023-01-05 12:06   
Beta package for Windows available at the beta auto-update channel and at:'https://devbuilds.drdteam.org/doomseeker/doomseeker-1.3.3~beta-230105-1140_windows.zip [^]'

Please test if the copyright blurb for Mailfud appears in the about dialog and that there's an URL pointing to the mailfud.org website.
(0022679)
Kaminsky   
2023-01-06 15:31   
I tested using 230106-0013 on Windows, and the copyright blurb for Malifud ("Mailfud GeoIP legacy databases are provided with Creative Commons Attribution-ShareAlike 4.0 International License.") appears in the dialog box. At the bottom, there's also a URL to the IP2C database:'https://mailfud.org/geoip-legacy/GeoIP-legacy.csv.gz [^]' which lets me download the database file.