0003950: MaxMind no longer a viable source of IP2C data

Notes
(0021930) Zalewa (developer) 2021-12-19 14:04	1.https://mailfud.org/geoip-legacy/ [^] The "GeoIP-legacy.csv.gz" seems to contain the data in the format we want. The "latest update" also looks promising: 15 December 2021. The DBs are on the CC 4.0 license. 2.https://github.com/wp-statistics/GeoLite2-Country [^] The DB is in a binary format, for which we don't have the parser currently. The last update also was 3 months ago, which may (may!) mean that this repo is dead. 3.https://ip2c.org [^] They appear "free" enough, but there's no DB download, no licensing info and no contact info. Contacting webmaster@ip2c.org may work, maybe?

(0021935) WubTheCaptain (reporter) 2021-12-19 23:57	Re: GeoLite Legacy: 0003235 / 0003235:0019419. On 2018-08-27, I wrote: Quote from WubTheCaptain Looks like GeoLiteCountry (GeoLite Legacy) will also go away starting January 2, 2019, replaced with GeoLite2 Country (free, CC BY-SA 4.0) or the proprietary GeoIP2 Country databases. Quote from WubTheCaptain I think this will be tricky to implement for other reasons too, as the small incompatibility between CC BY-SA 4.0 and LGPLv2.1+ licenses would mean binaries could only be distributed under GPLv3-only (one-way compatibility, to my awareness) – if LGPLv2.1+ allows redistributing under GPLv3-only. Debian did the following changes to a contrib package: Quote from geoipupdate_4.6.0-1_changelog geoipupdate (4.6.0-1) unstable; urgency=medium [ Faidon Liambotis ] * New upstream version. * Reflect changes with regards to GeoLite2's availability. (Closes: #948469) - Adjust the package description to reflect that an account is now needed. - Adjust the default GeoIP.conf to remove the now defunct default account. - Add a README.Debian to help guide new users on how to use this package. - Add a condition to the shipped cron file to avoid running geoipupdate when no accounts have been configured. * Stop referring to deprecated GeoIP Legacy products, or Suggest geoip-bin. * [...] -- Faidon Liambotis <paravoid@debian.org> Sat, 09 Jan 2021 18:02:37 +0200 Re: wp-statistics/GeoLite2-Country: It's unsuitable for us. Re: ip2c.org: Quote from ip2c.org This service is provided FREE under the terms of the GNU LGPLv3, 29 June 2007 Quote from ip2c.org This service uses IP to Country Database (csv) maintained by Webnet77.com http://webnet77.com/contact.html [^] http://software77.net/geo-ip [^] There's a different mail contact address on that page for ip2c.org. However their data source (Webnet77) also poof'd earlier in this year (April 2021, from what I could find). If I had the radical freedom to choose, I would throw out all the IP2C code and rely on servers implementing and broadcasting country information themselves (I'm aware this is only possible with Zandronum 3.1 right now). The move by MaxMind is an attempt of a power grab, this disgusts me.

(0021936) WubTheCaptain (reporter) 2021-12-20 00:04	Quote from GeoLite2 End User License Agreement 6. ADDITIONAL RESTRICTIONS. [...] (c) Destructions of GeoLite2 Database and GeoLite2 Data. From time to time, MaxMind will release an updated version of the GeoLite2 Databases, and you agree to promptly use the updated version of the GeoLite2 Databases. You shall cease use of and destroy (i) any old versions of the Services within thirty (30) days following the release of the updated GeoLite2 Databases; and (ii) all Services immediately upon termination of the license under this Agreement. Upon request, you shall provide MaxMind with written confirmation of such destruction. https://www.maxmind.com/en/geolite2/eula [^] Seems like a power grab to me, yeah.

(0021937) WubTheCaptain (reporter) 2021-12-20 00:16	My choice would be to keep the user-local database path ($HOME/.doomseeker/IpToCountry.dat?) in Doomseeker, but to no longer ship a binary with a copy of the database blob with Doomseeker as the current terms have become unacceptable for a software distribution like Doomseeker on any platform. Perhaps strip out IP2C auto-update code, or modify that options dialogue to allow entering MaxMind account details (if enabled at build-time, because of contrib) or document how to use geoipupdate with Doomseeker. I'd rather not touch this myself with a ten-foot pole, but it's a compromise. Anything less makes things more murky from an already murky licensing perspective (e.g. DRD Team distributing Doomseeker without source packages, current binaries needing to be shipped under GPLv3-only instead of the desired LGPLv2.1+, ...).

(0021938) WubTheCaptain (reporter) 2021-12-20 00:40 edited on: 2021-12-20 00:42	I'd also regrettably supporthttps://mailfud.org/geoip-legacy/ [^] for the Windows operating system, although it won't move us forward much to resolve the CC BY-SA 4.0 → GPLv3-only forward compatibility. The reason why I don't express my direct support to use it for GNU/Linux is that the geoip-database package (latest update 2019-12-24) is not the same (now unmaintained), although there is no interface from Doomseeker to use that data if installed. No idea if geoip-database's OpenDataLicense is compatible with the GPL, although the package and database is free from Debian's perspective. To use it on GNU/Linux, the geolite2legacy software to do the conversion from GeoLite2 to the old legacy format is free (MIT/Expat). This is fine. To effectively use Mailfud's geoip-legacy database on GNU/Linux, it'd need to be packaged for Debian (can be distributed via DRD Team repo). This would be plausible with source, except the data is available in a binary format. The CSV is created with yet another GeoLite2xtables software (unmaintained and "not production software"), from the very same GeoLite2 data from MaxMind. But I suppose we don't have a problem using a CSV generated by someone else. In comparison, the current status-quo with doomseeker-blobs is that we have the original CSV databases too. Freetard-ism truly is a mental illness.

(0021939) WubTheCaptain (reporter) 2021-12-20 01:09	Sorry for many notes, but I guess if I had to make a point, I don't suggest Zalewa or other developer to create an account with MaxMind for the maintenance of Doomseeker. If the user wants to, there's the geoipupdate (contrib) tool. If the goal is to create the legacy CSV data (free packaging) for auditing purposes, we need the MMDB sources (account, ugh), then it's complicated with existing tools (MMDB → DAT → CSV) and MaxMind's EULA is quite unacceptable or outright impossible for auditing / reproducing while attempting to maintain an up-to-date copy.

(0021940) WubTheCaptain (reporter) 2021-12-20 01:09	Debian Bug report logs #985457 - geoip-database update

(0021941) WubTheCaptain (reporter) 2021-12-20 01:13	Wait, the modified geolite2legacy source used by Mailfud is not available as free software? It only links to what seems to be the original geolite2legacy source.

(0021943) WubTheCaptain (reporter) 2021-12-20 01:45	Never guessed the GDPR (General Data Protection Regulation) or CCPA (California Consumer Privacy Act) was MaxMind's reason behind this.

(0021944) WubTheCaptain (reporter) 2021-12-20 01:47	Quote from geoip-database_20191224-3_changelog geoip-database (20191224-1) unstable; urgency=medium * The last free new upstream release :'( Happy christmas. https://bit.ly/34XLOxa [^] -- Patrick Matthäi <pmatthaei@debian.org> Tue, 24 Dec 2019 09:46:45 +0100

(0021945) WubTheCaptain (reporter) 2021-12-20 01:50 edited on: 2021-12-20 01:50	To clarify, Mailfud's adapted data is under MaxMind's EULA and so I don't think Doomseeker can use/distribute them in a legally sensible way.

(0021946) WubTheCaptain (reporter) 2021-12-20 02:00	Oh, I really want this to be the last note for now... I propose the next (free!) Doomseeker release to use 2019-12-24 data (available from Debian sources) and maybe put up an EOL/deprecation notice on the next Doomseeker release. I'd say to not support newer versions, although it will be theoretically possible for the user to do so under an EULA. Doomseeker 1.3, 1.3.1 and 1.3.2 releases were based on GeoLite2 2019-07-23 data.

(0021947) WubTheCaptain (reporter) 2021-12-20 02:02	But if you want to distribute an extra non-free package "update" for more up-to-date IP2C data under the EULA, that's up to you Zalewa.

(0021948) WubTheCaptain (reporter) 2021-12-20 03:07	Quote from WubTheCaptain To clarify, Mailfud's adapted data is under MaxMind's EULA and so I don't think Doomseeker can use/distribute them in a legally sensible way. I meant to say there is no way every user of Doomseeker could possibly accept to the four MaxMind agreements to use Doomseeker with the latest IP2C data: Quote from GeoLite2 End User License Agreement 1. ADDITIONAL POLICIES, TERMS AND CONDITIONS. The following policies are incorporated into this Agreement by reference and provide additional terms and conditions incorporated herein by this reference and/or related to the use of the Website: Creative Commons Corporation Attribution-ShareAlike 4.0 International License (the "Creative Commons License") MaxMind Data Processing Addendum ("DPA") MaxMind Privacy Policy ("PP") MaxMind Website Terms of Use ("WT") This Agreement controls in the event of any conflict with the above-referenced documents, except as otherwise provided in Section 7 (Personal Data). Thereafter, for any conflicts among the above 4 documents, the priority and precedence of interpretation is DPA, PP, WT and Creative Commons License. This is a sneaky license change for anything newer than 2019-12-24. The CC BY-SA 4.0 is the least important license agreement there. Sorry for making a note.

(0021957) Zalewa (developer) 2021-12-20 09:39	We're not removing this feature and we need a more up-to-date DB. If there's no better alternative, I'll start using mailfud. Quote from Wub the small incompatibility between CC BY-SA 4.0 and LGPLv2.1+ licenses would mean binaries could only be distributed under GPLv3-only (one-way compatibility, to my awareness) – if LGPLv2.1+ allows redistributing under GPLv3-only. What small incompatibility? What's the source on that claim? The current MM GeoLite2 DB is on CC-BY-SA 4.0 as well.

(0021958) WubTheCaptain (reporter) 2021-12-20 10:53	Quote from Zalewa If there's no better alternative, I'll start using mailfud. I will be greatly disappointed, if this is the direction Doomseeker is heading to. Fortunately, this is free software where anyone who disagrees can maintain a desired version of the software. Quote from Zalewa What small incompatibility? What's the source on that claim? Compatible Licenses page on Creative Commons website. (CC BY-SA 4.0: Section 1 c; Section 3 b 1.) I'm aware that the previous changes to IP2C helped here (0003537).

(0022310) WubTheCaptain (reporter) 2022-08-20 11:45	Although I've also seen Zandronum do their own server protocol thing of Zandronum servers (optionally) publishing a server configured country code, there's a proposed standardized way of getting authorative data: RFC 8805 - A Format for Self-Published IP Geolocation Feeds (format of geofeed.csv) RFC 9092 - Finding and Using Geofeed Data This authorative data comes from the network operators themselves, for ASNs / INETNUM / INET6NUM objects. If the data is incorrect, no need to contact 10+ geolocation database operators (including MaxMind) to fix the data, just contact the LIR (ISP) responsible for the network. RIPE NCC disallows defining "geofeed:" attributes on /48 ASSIGNED or ASSIGNED PI, because their lawyers want to protect individuals (personal data under the GDPR). But aggregations are okay. https://www.ripe.net/participate/mail/forum/db-wg/PFlkTHQ0VU9Rakl5b01IVGZAYW50b24uc29ib3Jub3N0Lm5ldD4= [^] And generally, RDAP will replace WHOIS someday (HTTPS/JSON vs free-text with no standard format globally). RDAP could be easier (and less bandwidth-expensive) to parse for the "geofeed:" attributes. By the way, for comparison: RIPE NCC objects already have "country:" data, but it has never been specified what it represents (the server centre or the home of the end-user), and is thus unreliable. Plausible these geolocation database operators (such as MaxMind) have gathered this unreliable data previously.

(0022312) Zalewa (developer) 2022-08-24 23:16 edited on: 2022-08-24 23:21	The source code is purged of references to MaxMind and the generator script will now use Mailfud (https://mailfud.org/geoip-legacy/) as the source of the "GeoIP Legacy database". I've already generated the IpToCountry.dat from this source today and it appears to accurately place the Ouroboros.net (EB) servers in Germany. The generated IpToCountry.dat V3 will have an embedded origin URL and an also embedded copyright note pointing to Mailfud. Relevant commits: - Doomseeker: 1. Pol's work to implement the V3 .dat format permits to include the origin URL and the copyright note in the .dat file itself, which means it will stay relevant to the currently used database, rather than being hardcoded in Doomseeker. 2. Hardcode the origin URL to Mailfud for V2 in case if Doomseeker >1.3.3 would still use V2 for some reason. 3. Remove the note about IpToCountry.dat from SOURCES.txt as it's not in the repository. - Doomseeker-blobs: 1. Pol's work on the generator to implement V3 also changes the source to Mailfud. 2. Further replacements of MaxMind with Mailfud. 3. Remove the MaxMind database from the repo. 4. Rename the geolite2_conv.py generator to IpToCountry.py. - Website: 1. Get rid of geolite2.gz from the filenames.

(0022314) WubTheCaptain (reporter) 2022-08-25 09:53 edited on: 2022-08-25 09:54	As I said earlier, the Mailfud sources are derivative works of MaxMind's work under the GeoLite2 End User License Agreement. The CC BY-SA 4.0 component is only one part of the agreement. Quote By downloading or using our GeoLite2 Database, you are accepting and agreeing to the terms and conditions set forth in this GeoLite2 End User License Agreement (this "Agreement"). Quote This Agreement controls in the event of any conflict with the above-referenced documents, except as otherwise provided in Section 7 (Personal Data). Thereafter, for any conflicts among the above 4 documents, the priority and precedence of interpretation is DPA, PP, WT and Creative Commons License. It is my opinion the use and distribution of this data from Mailfud is not in the spirit of the copyright licenses of GeoLite2 and Doomseeker, and may be illegal. -1 for these changes.

(0022315) WubTheCaptain (reporter) 2022-08-25 09:55 edited on: 2022-08-25 09:56	Additionally, the doomseeker-blobs repository's README makes no mention about the licensing under GeoLite2 End User License Agreement currently.

This issue is already marked as resolved. If you feel that is not the case, please reopen it and explain why.
Supporters:	No one explicitly supports this issue yet.
Opponents:	No one explicitly opposes this issue yet.

Issue History
Date Modified	Username	Field	Change
2021-12-19 13:57	Zalewa	New Issue
2021-12-19 13:57	Zalewa	File Added: geolite2_not_for_dumbledore.jpg
2021-12-19 13:58	Zalewa	Description Updated	View Revisions
2021-12-19 14:04	Zalewa	Note Added: 0021930
2021-12-19 23:57	WubTheCaptain	Note Added: 0021935
2021-12-19 23:57	WubTheCaptain	Status	new => acknowledged
2021-12-20 00:04	WubTheCaptain	Note Added: 0021936
2021-12-20 00:16	WubTheCaptain	Note Added: 0021937
2021-12-20 00:40	WubTheCaptain	Note Added: 0021938
2021-12-20 00:42	WubTheCaptain	Note Edited: 0021938	View Revisions
2021-12-20 01:09	WubTheCaptain	Note Added: 0021939
2021-12-20 01:09	WubTheCaptain	Note Added: 0021940
2021-12-20 01:13	WubTheCaptain	Note Added: 0021941
2021-12-20 01:27	WubTheCaptain	Relationship added	related to 0003235
2021-12-20 01:42	WubTheCaptain	Additional Information Updated	View Revisions
2021-12-20 01:45	WubTheCaptain	Note Added: 0021943
2021-12-20 01:47	WubTheCaptain	Note Added: 0021944
2021-12-20 01:50	WubTheCaptain	Note Added: 0021945
2021-12-20 01:50	WubTheCaptain	Note Edited: 0021945	View Revisions
2021-12-20 02:00	WubTheCaptain	Note Added: 0021946
2021-12-20 02:02	WubTheCaptain	Note Added: 0021947
2021-12-20 03:07	WubTheCaptain	Note Added: 0021948
2021-12-20 09:39	Zalewa	Note Added: 0021957
2021-12-20 10:53	WubTheCaptain	Note Added: 0021958
2021-12-20 15:46	Zalewa	Relationship added	related to 0003952
2022-08-20 11:45	WubTheCaptain	Note Added: 0022310
2022-08-24 12:41	Zalewa	Assigned To	=> Zalewa
2022-08-24 12:41	Zalewa	Status	acknowledged => assigned
2022-08-24 23:16	Zalewa	Note Added: 0022312
2022-08-24 23:16	Zalewa	Status	assigned => needs review
2022-08-24 23:20	Zalewa	Note Edited: 0022312	View Revisions
2022-08-24 23:21	Zalewa	Note Edited: 0022312	View Revisions
2022-08-25 09:53	WubTheCaptain	Note Added: 0022314
2022-08-25 09:54	WubTheCaptain	Note Edited: 0022314	View Revisions
2022-08-25 09:55	WubTheCaptain	Note Added: 0022315
2022-08-25 09:56	WubTheCaptain	Note Edited: 0022315	View Revisions
2022-09-10 14:55	Zalewa	Relationship added	related to 0003699
2023-02-19 14:21	Zalewa	Target Version	=> 1.4.0
2023-02-19 14:23	Zalewa	Status	needs review => resolved
2023-02-19 14:23	Zalewa	Fixed in Version	=> 1.4.0
2023-02-19 14:23	Zalewa	Resolution	open => fixed