0003235: Integrate IP2C with geoip-database package in Debian GNU/Linux

Notes
(0019409) Zalewa (developer) 2018-08-26 11:55 edited on: 2018-08-26 11:56	Just by looking at the file format specification located at'http://maxmind.github.io/MaxMind-DB/ [^]' I'd say that adding support for this database to Doomseeker will need a substantial amount of new code that will have to work in parallel with the current code. This specification document is a piece of garbage (just compare it to the WAVE file format spec). Fortunately, there's a reader available on GitHub:'https://github.com/maxmind/libmaxminddb [^]' , if you'd like to welcome another addition to the "dependencies/" directory. However, the implementation difficulty is just the top of the iceberg. It's unknown for me what will be the performance of reading this database from within Doomseeker. I won't allow increase of startup time or consuming over 9000% of CPU during server refresh. Technically, we could convert the database just once at first use to a more Doomseeker-friendly format and then just do MD5 checksums to see if the cache needs to be recreated, but you know what they say. Moving on, there are more problems. When it comes to OS availability, it at least seems that GeoIP is available under the same location also in Arch, so it could be extrapolated that adding the functionality of reading this database would actually cover this issue on all Linux distros. Mac and Windows Doomseeker releases still need to fend for themselves. Just as programmers can be creative in hiding switch/case/default scenarios and subsequently trying to get away with the missing 'default' block, this is a monstrously well hidden #ifdef one. To fix the problem of maintaining parallel parsers (#ifdefs), we could, hypothetically, drop our custom format and switch to MaxMinds, however I am not going to do that. Yes, the current code operates on our own database, but that's actually a big boon. What if MaxMind decides to change its format, decides to hide all of their databases behind a paywall, dies or becomes unreliable like the other service? In current approach we can just use a different provider and just mash their database into our own. No re-release of Doomseeker is needed to cover that. The snapshot of the already converted database format is available in Doomseeker's repository and it is incorporated into the executable through Qt's resource system. This serves as a vaguely accurate fallback in case if someone really doesn't want the program to "phone home" and download the database maintained by us. Therefore Doomseeker is not dependant on any download from its home page and there's no issue with "easily reproducing derivative snapshots of Doomseeker". One thing that I can agree on is the lack of specification for our own database format. The small piece of Python code that prepares it is insufficient. I'll create a small document explaining the format. In regards to the implementation of reading of /usr/share/GeoIP/GeoIP.dat, I'm in strong opposition to that and will not spend any of my time on it. Quote from "WubTheCaptain" The IP2C database is auto-fetched and compared against checksums (local & remote) on every start of Doomseeker (unless specifically disabled in configuration?). This statement is inaccurate. The remote checksum is fetched first. If remote & local checksums match, no further communication is done. This currently amounts to 34 bytes + TCP and HTTP boilerplate of traffic.

(0019419) WubTheCaptain (reporter) 2018-08-27 02:30	Looks like GeoLiteCountry (GeoLite Legacy) will also go away starting January 2, 2019, replaced with GeoLite2 Country (free, CC BY-SA 4.0) or the proprietary GeoIP2 Country databases. Doomseeker ships with GeoLite2 Country / custom DFSG-free writer in Python 3. I'm not sure what Debian is going to do with this, because the geoip-database (and the related libgeoip1 package) uses the legacy database. There's a Debian bug #885442 - Please ship GeoLite2 databases for this. For clarification, /usr/share/GeoIP/GeoIP.dat and /usr/share/GeoIP/GeoIPv6.dat currently use the legacy GeoLiteCountry database. I think this will be tricky to implement for other reasons too, as the small incompatibility between CC BY-SA 4.0 and LGPLv2.1+ licenses would mean binaries could only be distributed under GPLv3-only (one-way compatibility, to my awareness) – if LGPLv2.1+ allows redistributing under GPLv3-only. The idea of closing this ticket hurts me, but there might not be an appropriate solution available at this time unless someone steps forward willing to rewrite large parts of Doomseeker code and solve any issues that may come with it.

(0019420) WubTheCaptain (reporter) 2018-08-27 02:30 edited on: 2018-08-27 02:32	Now, onto Zalewa's comments. libmaxminddb exists in Debian's free repositories (Apache 2.0 licensed), which would be not pose a much larger problem for a packager to depend on – "dependencies/" would be "fine" (but not preferrable), as long as the system installed library is preferred like with other dependencies. But a CSV format is also available for GeoLite2 (CSV is what Doomseeker uses). It's questionable which format (CSV or MMDB) is the preferred form of modification. I believe writing the legacy GeoCountryLite (?) database from CSV to be very slow, but I've never tried reading from it, especially hundreds of times at once. So, I don't have a better clue of the performance either. "Compacted format is optimized for faster loading and for searching country by IP" says tools/geolite2_conv.py, but there are no actual benchmarking results available. Quote from Zalewa What if MaxMind decides to change its format, decides to hide all of their databases behind a paywall, dies or becomes unreliable like the other service? I don't think I can I follow or I've not understood. And which "other service" are we talking about? If GNU/Linux distros eventually decide to switch their /usr/share/GeoIP/GeoIP.dat to GeoLite2 (MMDB format), and Doomseeker dropped its custom database format, then I don't think readability to be too hard if libmaxminddb was used. Only legacy GeoLite database updates seem to be going behind a "paywall" at the moment, GeoLite2 will remain free. Even so, the databases are freely redistributable under CC BY-SA 4.0 – but this has another concern with license compatibility (see previous note). Theoretically we can ship a copy of the database (MMDB or CSV source, preferred form of modification) for Windows and macOS, and convert it at Doomseeker format at compile time – but this is expensive, and has been iterated in the past this won't likely be done due to cost. But wait, maybe there's something else to do: There is a geoipupdate tool in Debian's contrib repositories to download GeoLite2 updates. Imagine a launcher/shell script to do this at launch, possibly removing a need to do this in Doomseeker code with IP2C autoupdate. Support on Windows & macOS... dunno. It's C and under GPLv2+ (mostly, with bits under LGPLv2.1+ and free RSA-MD5 license).

(0019436) Zalewa (developer) 2018-08-27 10:04	None of this is less contrived than what we currently have. Quote from "WubTheCaptain" which "other service" are we talking about? 'http://software77.net/geo-ip/ [^]' - this is the service that we've been using in the past. It has grown inaccurate over time, pointing IPs to wrong countries.

This issue is already marked as resolved. If you feel that is not the case, please reopen it and explain why.
Supporters:	No one explicitly supports this issue yet.
Opponents:	Zalewa

Issue History
Date Modified	Username	Field	Change
2017-09-01 12:09	WubTheCaptain	New Issue
2017-09-01 16:30	Zalewa	Relationship added	child of 0003246
2017-10-04 19:40	WubTheCaptain	Reproducibility	always => N/A
2017-10-04 19:40	WubTheCaptain	Summary	Integrate IP2C with geopip-database package in Debian GNU/Linux => Integrate IP2C with geoip-database package in Debian GNU/Linux
2018-08-26 11:55	Zalewa	Note Added: 0019409
2018-08-26 11:56	Zalewa	Note Edited: 0019409	View Revisions
2018-08-27 01:15	WubTheCaptain	Status	new => feedback
2018-08-27 02:30	WubTheCaptain	Note Added: 0019419
2018-08-27 02:30	WubTheCaptain	Status	feedback => new
2018-08-27 02:30	WubTheCaptain	Note Added: 0019420
2018-08-27 02:31	WubTheCaptain	Status	new => feedback
2018-08-27 02:32	WubTheCaptain	Note Edited: 0019420	View Revisions
2018-08-27 10:04	Zalewa	Note Added: 0019436
2018-08-28 03:37	WubTheCaptain	Status	feedback => closed
2018-08-28 03:37	WubTheCaptain	Resolution	open => suspended
2021-12-20 01:27	WubTheCaptain	Relationship added	related to 0003950