Zandronum Chat @ irc.zandronum.com
#zandronum
Get the latest version: 3.0
Source Code

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0003235Doomseeker[All Projects] Suggestionpublic2017-09-01 12:092018-08-28 03:37
ReporterWubTheCaptain 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusclosedResolutionsuspended 
Platformx86_64 (really cross-platform)OSDebian GNU/LinuxOS Versionbuster/sid
Product Version1.1 
Target VersionFixed in Version 
Summary0003235: Integrate IP2C with geoip-database package in Debian GNU/Linux
DescriptionDoomseeker uses an IP2C (IP to country) database to give match listed servers with their respective country flag.

Autodownloading a binary IP2C database (GeoLite2 Country) fromhttp://doomseeker.drdteam.org/ip2c/geolite2.gz [^] is not very Debian-ish (doesn't fit into the Debian GNU/Linux ecosystem), even though a tool to convert the CSV database from Maxmind is present in source distribution (tools/geolite2_conv.py).

Worse, it may lead to reusers not being able to (easily) reproduce a derivative screenshot of Doomseeker at all if doomseeker.drdteam.org goes down and no instructions are provided how to reproduce the binary database or to acquire the CSV database, with immediate effect of non-GPLv2+ license compliance. This is obviously an issue with redistribution, and a possible blocker of inclusion in Debian GNU/Linux?

If present, /usr/share/GeoIP/GeoIP.dat and /usr/share/GeoIP/GeoIPv6.dat should instead be read (and optionally converted to a format Doomseeker understands). These are provided by the geoip-database package in Debian.
Steps To ReproduceThe IP2C database is auto-fetched and compared against checksums (local & remote) on every start of Doomseeker (unless specifically disabled in configuration?).
Additional InformationThere are legal considerations about license compatibility with GNU GPLv2+ (Doomseeker) and CC BY-SA 4.0/OpenDataLicense (GeoLite2/GeoLiteCountry).

geoip-database-contrib also exists in the contrib repository of Debian, consisting of Maxmind's pre-packaged binary databases (where no source code is available to reproduce those binary databases).
Attached Files

- Relationships
child of 0003246confirmedWubTheCaptain Debian packaging 

-  Notes
User avatar (0019409)
Zalewa (developer)
2018-08-26 11:55
edited on: 2018-08-26 11:56

Just by looking at the file format specification located athttp://maxmind.github.io/MaxMind-DB/ [^] I'd say that adding support for this database to Doomseeker will need a substantial amount of new code that will have to work in parallel with the current code. This specification document is a piece of garbage (just compare it to the WAVE file format spec). Fortunately, there's a reader available on GitHub:https://github.com/maxmind/libmaxminddb [^] , if you'd like to welcome another addition to the "dependencies/" directory. However, the implementation difficulty is just the top of the iceberg.

It's unknown for me what will be the performance of reading this database from within Doomseeker. I won't allow increase of startup time or consuming over 9000% of CPU during server refresh. Technically, we could convert the database just once at first use to a more Doomseeker-friendly format and then just do MD5 checksums to see if the cache needs to be recreated, but you know what they say. Moving on, there are more problems.

When it comes to OS availability, it at least seems that GeoIP is available under the same location also in Arch, so it could be extrapolated that adding the functionality of reading this database would actually cover this issue on all Linux distros. Mac and Windows Doomseeker releases still need to fend for themselves. Just as programmers can be creative in hiding switch/case/default scenarios and subsequently trying to get away with the missing 'default' block, this is a monstrously well hidden #ifdef one.

To fix the problem of maintaining parallel parsers (#ifdefs), we could, hypothetically, drop our custom format and switch to MaxMinds, however I am not going to do that. Yes, the current code operates on our own database, but that's actually a big boon. What if MaxMind decides to change its format, decides to hide all of their databases behind a paywall, dies or becomes unreliable like the other service? In current approach we can just use a different provider and just mash their database into our own. No re-release of Doomseeker is needed to cover that.

The snapshot of the already converted database format is available in Doomseeker's repository and it is incorporated into the executable through Qt's resource system. This serves as a vaguely accurate fallback in case if someone really doesn't want the program to "phone home" and download the database maintained by us. Therefore Doomseeker is not dependant on any download from its home page and there's no issue with "easily reproducing derivative snapshots of Doomseeker".

One thing that I can agree on is the lack of specification for our own database format. The small piece of Python code that prepares it is insufficient. I'll create a small document explaining the format.

In regards to the implementation of reading of /usr/share/GeoIP/GeoIP.dat, I'm in strong opposition to that and will not spend any of my time on it.

Quote from "WubTheCaptain"
The IP2C database is auto-fetched and compared against checksums (local & remote) on every start of Doomseeker (unless specifically disabled in configuration?).


This statement is inaccurate. The remote checksum is fetched first. If remote & local checksums match, no further communication is done. This currently amounts to 34 bytes + TCP and HTTP boilerplate of traffic.

User avatar (0019419)
WubTheCaptain (developer)
2018-08-27 02:30

Looks like GeoLiteCountry (GeoLite Legacy) will also go away starting January 2, 2019, replaced with GeoLite2 Country (free, CC BY-SA 4.0) or the proprietary GeoIP2 Country databases. Doomseeker ships with GeoLite2 Country / custom DFSG-free writer in Python 3.

I'm not sure what Debian is going to do with this, because the geoip-database (and the related libgeoip1 package) uses the legacy database. There's a Debian bug #885442 - Please ship GeoLite2 databases for this. For clarification, /usr/share/GeoIP/GeoIP.dat and /usr/share/GeoIP/GeoIPv6.dat currently use the legacy GeoLiteCountry database.

I think this will be tricky to implement for other reasons too, as the small incompatibility between CC BY-SA 4.0 and LGPLv2.1+ licenses would mean binaries could only be distributed under GPLv3-only (one-way compatibility, to my awareness) – if LGPLv2.1+ allows redistributing under GPLv3-only.

The idea of closing this ticket hurts me, but there might not be an appropriate solution available at this time unless someone steps forward willing to rewrite large parts of Doomseeker code and solve any issues that may come with it.
User avatar (0019420)
WubTheCaptain (developer)
2018-08-27 02:30
edited on: 2018-08-27 02:32

Now, onto Zalewa's comments.

libmaxminddb exists in Debian's free repositories (Apache 2.0 licensed), which would be not pose a much larger problem for a packager to depend on – "dependencies/" would be "fine" (but not preferrable), as long as the system installed library is preferred like with other dependencies. But a CSV format is also available for GeoLite2 (CSV is what Doomseeker uses). It's questionable which format (CSV or MMDB) is the preferred form of modification.

I believe writing the legacy GeoCountryLite (?) database from CSV to be very slow, but I've never tried reading from it, especially hundreds of times at once. So, I don't have a better clue of the performance either. "Compacted format is optimized for faster loading and for searching country by IP" says tools/geolite2_conv.py, but there are no actual benchmarking results available.

Quote from Zalewa
What if MaxMind decides to change its format, decides to hide all of their databases behind a paywall, dies or becomes unreliable like the other service?


I don't think I can I follow or I've not understood. And which "other service" are we talking about?

If GNU/Linux distros eventually decide to switch their /usr/share/GeoIP/GeoIP.dat to GeoLite2 (MMDB format), and Doomseeker dropped its custom database format, then I don't think readability to be too hard if libmaxminddb was used.

Only legacy GeoLite database updates seem to be going behind a "paywall" at the moment, GeoLite2 will remain free. Even so, the databases are freely redistributable under CC BY-SA 4.0 – but this has another concern with license compatibility (see previous note). Theoretically we can ship a copy of the database (MMDB or CSV source, preferred form of modification) for Windows and macOS, and convert it at Doomseeker format at compile time – but this is expensive, and has been iterated in the past this won't likely be done due to cost. But wait, maybe there's something else to do:

There is a geoipupdate tool in Debian's contrib repositories to download GeoLite2 updates. Imagine a launcher/shell script to do this at launch, possibly removing a need to do this in Doomseeker code with IP2C autoupdate. Support on Windows & macOS... dunno. It's C and under GPLv2+ (mostly, with bits under LGPLv2.1+ and free RSA-MD5 license).

User avatar (0019436)
Zalewa (developer)
2018-08-27 10:04

None of this is less contrived than what we currently have.

Quote from "WubTheCaptain"

which "other service" are we talking about?

http://software77.net/geo-ip/ [^] - this is the service that we've been using in the past. It has grown inaccurate over time, pointing IPs to wrong countries.

Issue Community Support
This issue is already marked as resolved.
If you feel that is not the case, please reopen it and explain why.
Supporters: No one explicitly supports this issue yet.
Opponents: Zalewa

- Issue History
Date Modified Username Field Change
2017-09-01 12:09 WubTheCaptain New Issue
2017-09-01 16:30 Zalewa Relationship added child of 0003246
2017-10-04 19:40 WubTheCaptain Reproducibility always => N/A
2017-10-04 19:40 WubTheCaptain Summary Integrate IP2C with geopip-database package in Debian GNU/Linux => Integrate IP2C with geoip-database package in Debian GNU/Linux
2018-08-26 11:55 Zalewa Note Added: 0019409
2018-08-26 11:56 Zalewa Note Edited: 0019409 View Revisions
2018-08-27 01:15 WubTheCaptain Status new => feedback
2018-08-27 02:30 WubTheCaptain Note Added: 0019419
2018-08-27 02:30 WubTheCaptain Status feedback => new
2018-08-27 02:30 WubTheCaptain Note Added: 0019420
2018-08-27 02:31 WubTheCaptain Status new => feedback
2018-08-27 02:32 WubTheCaptain Note Edited: 0019420 View Revisions
2018-08-27 10:04 Zalewa Note Added: 0019436
2018-08-28 03:37 WubTheCaptain Status feedback => closed
2018-08-28 03:37 WubTheCaptain Resolution open => suspended






Questions or other issues? Contact Us.

Links


Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker