Zandronum Chat @ irc.zandronum.com
#zandronum
Get the latest version: 3.0
Source Code

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0003369Doomseeker[All Projects] Suggestionpublic2018-02-10 03:282018-12-13 17:43
ReporterMarcaek 
Assigned ToPol M 
PrioritynormalSeverityfeatureReproducibilityalways
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version 
Target Version1.3Fixed in Version 
Summary0003369: Wadseeker desperately needs a way to compare downloaded wads to wads being requested by server
DescriptionOn multiple occasions Wadseeker has downloaded incorrect files which are modified but have the same name across multiple websites. It makes little sense that the game should launch with incorrect files.
Attached Files

- Relationships
related to 0003552needs reviewWubTheCaptain DogSoft, included in Wadseeker's WAD sites, often gives outdated WADs (and/or is very slow) 

-  Notes
User avatar (0019027)
Korshun (reporter)
2018-02-10 12:59
edited on: 2018-02-10 13:05

How would it do that if Zandronum servers don't publish wad hashes until you've tried to connect?

How about making Zandronum servers publish a list of wad hashes along with wad names? According tohttps://wiki.zandronum.com/Launcher_protocol [^] , introducing SQF_PWADHASHES won't break backwards compatibility.

EDIT: the comma breaks the URL.

EDIT2: publishing wad hashes would also allow directly querying for wads by hash, instead of downloading by name and verifying using the hash.http://wad-archive.com/ [^] has that functionality and can provide many download links for any wad hash, but I haven't seen any other places that allow querying by wad hash.

User avatar (0019028)
Blzut3 (administrator)
2018-02-10 21:53

For some reason I thought we already added hashes to the protocol, but don't see it in the source (probably was thinking of SQF_DATA_MD5SUM which is no longer used). In any case though Odamex already exposes thing information so it could be made part of our API today.
User avatar (0019029)
Zalewa (developer)
2018-02-11 09:30

If we could recognize hosted WADs by hash and download them by this hash, there still would remain a problem of having duplicate WADs on disk.

In "ye olde" times, especially when it was about WADs designed for multiplayer, the creators were usually aware of the duplicated WADs problem and incorporated WAD's version number in to the WAD's file name. It was simple: you changed something in the WAD, then you also changed its name and, thanks to this, there were no incompatible "duplicates" being hosted over different servers. I can understand keeping the same filename when the WAD is designed for single player, but if the author claims "multiplayer: designed for" then the author should be aware how big of a problem it is to release an incompatible version with the same name.

However, let's assume that the server browser can differentiate WADs by hashes. It also detects that the user, in their "WAD downloads" directory, already has the WAD with the same name. What should it do?

a) Tell the user that there's a duplicate and let them choose to overwrite or cancel the download, optionally allowing to ignore the problem? In the last case the issue or pressing "Ignore" all the time will become annoying.

b) Incorporate the hash into the duplicated WAD's filename when it's downloaded? If we see "mod.wad" but hash mismatches we can always look for "mod_abcdef1234567890hashishash.wad". If that is not found we can always ask to download & rename.
User avatar (0019030)
Blzut3 (administrator)
2018-02-11 09:55

At the very least supporting checking the hash if the plugin supports it and telling the user that there's a mismatch before launching would be useful.

I don't think HTTP provides any standardized way of asking for a content match by hash, so as far as fetching goes outside of something like wad-archive I think the only option is to get a full list of URLs and try them one by one until a match is found. Off hand the only thing we could possibly do to improve that is implement a custom header that hosts could choose to do something with, but I wouldn't expect any sizable number of servers to comply. (Web browser caching relies on the "Etag" header which appears to be an arbitrary server side value so not useful to us.)

I like your idea on auto-renaming files with the hash if there's a duplicate. My initial thought was to just blindly overwrite (assuming the user accepts the usual "would you like to download" dialog), but that sounds like it would be fairly easy to implement and solve the problem nicely.

From the impression I got when Marcaek when he told me of the issue I would assume the bigger problem is Wadseeker picking the wrong version for people without any version of the wad since an older version happens to be hosted on one of our default locations. With that in mind having Wadseeker hash check and reject would be useful even if the duplicate problem isn't solved.
User avatar (0019032)
WubTheCaptain (developer)
2018-02-12 22:14

Is the future of Wadseeker to be a WAD downloader or a WAD (file) manager? In Doomseeker (a server browser)?
User avatar (0019033)
Blzut3 (administrator)
2018-02-13 04:01

Not sure where your question comes from. The goal is to get people into servers with the right wads loaded in the case of Doomseeker. For Wadseeker it to find and download the correct wad.
User avatar (0019055)
WubTheCaptain (developer)
2018-02-15 18:15

Quote from Blzut3
The goal is to get people into servers with the right wads loaded in the case of Doomseeker. For Wadseeker it to find and download the correct wad.


Oh sorry, I was thinking a bit naively. I'm inclined to support this, then.
User avatar (0019399)
AOSP (reporter)
2018-08-25 22:42

This should do for getting the hashes from the server, right?https://bitbucket.org/csnxs/zandronum-stable2/commits/0041e93150f4c2734ebf4574acf49fb6f768ef2f?at=sb-query-md5s [^]

Just checking before I send a PR upstream...
User avatar (0019400)
Blzut3 (administrator)
2018-08-25 22:48

Check if there's anything that needs to be done for DEH patches and optional wads, but otherwise that would indeed be what we need.
User avatar (0019401)
AOSP (reporter)
2018-08-25 22:53

Optional WADs work fine, but I don't think DEH patches have their hashes stored anywhere (they're not even checked when connecting), and they're so infrequently used that I don't think it's worth adding them. I'll submit the PR if nobody has anything else to say.
User avatar (0019415)
AOSP (reporter)
2018-08-26 19:46
edited on: 2018-08-26 19:49

Torr noted that we were running out of space for fields, so a new field for more information was created and it was moved there.

Change in zandronum-stable
Also documented on the wiki (diff)

User avatar (0020130)
Pol M (developer)
2018-10-17 17:17

This looks like an interesting ticket. I'm gonna do some work on getting the hashes from the server ftm. (I'll extend PWad with a variable, and a constructor)
User avatar (0020131)
Pol M (developer)
2018-10-17 18:35

PR (gets the hashes)
User avatar (0020177)
Pol M (developer)
2018-11-23 17:25
edited on: 2018-11-23 17:26

I'm gonna do some extra (small) stuff. Right now I think the next logical step is to get Doomseeker to:
- Make it able to hash WADs
- List in the WADs column the ones that are incompatible (when you hover the mouse over it)
- Make it acknowledge the incompatible WADs and not simply execute the client when you double-click a server (and do something also with "Find missing WADs" on a right-click).

So, the first part should be relatively easy. As far as I can tell WadPathFinder is used to deal with wads, so it should be easy to extend it to incorporate a hashing function and add whether is necessary to WadFindResult to accommodate the changes.

The second part should also be simple. ServerToolTip->createPwadToolTipInfo handles this and with the previous changes, there should be np.

With the last part, it seems trickier. What I've thought for the moment is to modify MissingWadsDialog constructor to get a new argument for the corrupted/non-equal WADs. I'll look at it more the next days, but it seems the only calls it gets are from JoinCommandBuilder (double-click) and a signal in MainMenu ("Find missing WADs" on a right-click) (If my guesses are correct). I'll get something working :)

User avatar (0020178)
Zalewa (developer)
2018-11-23 21:33

1 & 2 sound good, although Wadseeker will also need to be able to calculate the same hash so decide carefully where you put the code.

Point 3. can ask the user if they want their WADs redownloaded and overwritten and then just replace the existing files, although Wadseeker would also need to compare the hash before overwriting. Or Doomseeker could take the approach with incorporating the hash into the WAD name as I mentioned in my comment 0003369:0019029, though that option will be incompatible with other launchers.

Wadseeker could EXPORT a Checksum object with "enum Algorithm algorithm" and "QByteArray hash" types. Algorithm enum will initially only have MD5 value, but we will be able to expand it with other types in the future. It must also be possible to feed the Checksum objects together with requested WADs for the search operation and possibly consider that a single WAD may have more than one Checksum object if we'd ever like to query WAD archives by passing the checksums and those WAD archives would only support one of the algorithms.
User avatar (0020179)
Pol M (developer)
2018-11-24 22:23

Quote from Zalewa

1 & 2 sound good, although Wadseeker will also need to be able to calculate the same hash so decide carefully where you put the code.

Welp, this means that it will have to be located in Wadseeker if we don't want to duplicate code

Quote from Zalewa

can ask the user if they want their WADs redownloaded and overwritten and then just replace the existing files

I'll take a similar approach to optional WADs, with the difference that it will prompt regardless if there are required WADs to download or not.

Quote from Zalewa

although Wadseeker would also need to compare the hash before overwriting

I guess you mean once the WAD is downloaded to check if the WAD is the one used by the server. I agree that we should try to prevent overwriting files when it is not necessary.

Quote from Zalewa

Or Doomseeker could take the approach with incorporating the hash into the WAD name as I mentioned in my comment 0003369:0019029

Huh, this would be interesting. I'd agree on trying to conserve files, but maybe stuff would be more organized in directories? creating a directory with the name (Ex: "coolpk3mod/") and then inside the file with the hash (Ex: "coolpk3mod/6e38e7975656200ff6dccc11919a2e23.pk3"). Also, since I see we're extending this to other hash algorithms, maybe we could also append it's name (Ex: "coolpk3mod/md5_6e38e7975656200ff6dccc11919a2e23.pk3"). All that said, for the moment I'd be inclined on dealing with this in the future and for the moment focus on having only the right WADs for the connection. (My approach has the inconvenience of dealing with permissions)

Quote from Zalewa

though that option will be incompatible with other launchers.

Another reason to deal with this later on :P

Quote from Zalewa

Wadseeker could EXPORT a Checksum object with "enum Algorithm algorithm" and "QByteArray hash" types. Algorithm enum will initially only have MD5 value, but we will be able to expand it with other types in the future. It must also be possible to feed the Checksum objects together with requested WADs for the search operation and possibly consider that a single WAD may have more than one Checksum object if we'd ever like to query WAD archives by passing the checksums and those WAD archives would only support one of the algorithms.

Let's see if we are on the same page: The objective is to feed Wadseeker with names and Checksums(object with the hash and the algorithm) and get all the WADs that match a checksum (and also match one of the names) and also get the WADs that match the names (since we may not know any hash).
This has the problem that if we don't find any WAD that satisfies the checksum we won't know which WAD we're missing.

We could do something similar to the Pwad class: Wadseeker would EXPORT a class with a QString with the name of the WAD and the QList<Checksum> with all the checksums we could ever need. Then Wadseeker would look for WADs with the name and then check if that WAD validates with one of the checksums. If it does, or no checksums exist, we'd proceed to download, and if not, we'd keep looking. Also, in case we don't know the name (aka: name.IsEmpty() returns true) we would look for WADs that satisfy all the hashes we do know. If we don't find anything, We'll know the name of what we're missing (well, except if it is the case we're searching using only checksums)

I'm gonna take the wild guess, though, that not many pages give the hashes of their wads, so we'd probably have to download the wad to validate the checksums ourselves. Also, the search "without the name" will be limited.
User avatar (0020180)
Zalewa (developer)
2018-11-25 08:38

Quote from Pol M
Also, the search "without the name" will be limited.

The filename is always expected to be provided by the game and the program needs to know what it looks for and the hash by itself is not good enough. I was thinking about utilizing APIs such as WAD-Archive's (https://www.wad-archive.com/api [^] ) where you can use the hash directly in the query.

Quote from Pol M
I'd agree on trying to conserve files, but maybe stuff would be more organized in directories? creating a directory with the name (Ex: "coolpk3mod/") and then inside the file with the hash (Ex: "coolpk3mod/6e38e7975656200ff6dccc11919a2e23.pk3"). Also, since I see we're extending this to other hash algorithms, maybe we could also append it's name (Ex: "coolpk3mod/md5_6e38e7975656200ff6dccc11919a2e23.pk3").

The problem with trying to manage the WADs in a more complicated way than the current one is that the storage directory will become much more difficult to browse from a file explorer. Right now as there's a 1-to-1 matching of WADs to their filenames it's easy to find files that were downloaded lately or to just find them by name. Moreover, once we start to modify the filename, the user won't be able to host the server with this WAD without incurring further name confusion or without changing the name back manually.

Simply displaying the information about invalid checksums seems to be the least invasive and the best approach for now, but it won't solve problem with having A/B versions of the same WAD hosted in the wild. It will only be helpful in cases where the WAD got an update and everyone is hosting the new version while the user still has the previous one.

Quote from Pol M
I'll take a similar approach to optional WADs, with the difference that it will prompt regardless if there are required WADs to download or not.

Yes, that sounds exactly good.
User avatar (0020225)
Pol M (developer)
2018-12-06 22:29

Ok, I've made quite a few changes lately. I'll push something to the same PR today or tomorrow (I finally got a free day😄). More notes in bitbucket once I'm finished.

Issue Community Support
Only registered users can voice their support. Click here to register, or here to log in.
Supporters: Korshun Zalewa WubTheCaptain Michaelis DevilHunter Marcaek Laggy Blazko AOSP Leonard Vicious Pariah Pol M
Opponents: No one explicitly opposes this issue yet.

- Issue History
Date Modified Username Field Change
2018-02-10 03:28 Marcaek New Issue
2018-02-10 12:59 Korshun Note Added: 0019027
2018-02-10 13:00 Korshun Note Edited: 0019027 View Revisions
2018-02-10 13:05 Korshun Note Edited: 0019027 View Revisions
2018-02-10 21:53 Blzut3 Note Added: 0019028
2018-02-11 05:08 Blzut3 Project Zandronum => Doomseeker
2018-02-11 09:30 Zalewa Note Added: 0019029
2018-02-11 09:55 Blzut3 Note Added: 0019030
2018-02-12 22:12 WubTheCaptain Severity major => feature
2018-02-12 22:12 WubTheCaptain Status new => acknowledged
2018-02-12 22:12 WubTheCaptain Product Version 3.0 =>
2018-02-12 22:14 WubTheCaptain Note Added: 0019032
2018-02-13 04:01 Blzut3 Note Added: 0019033
2018-02-15 18:15 WubTheCaptain Note Added: 0019055
2018-08-25 22:42 AOSP Note Added: 0019399
2018-08-25 22:48 Blzut3 Note Added: 0019400
2018-08-25 22:53 AOSP Note Added: 0019401
2018-08-26 19:46 AOSP Note Added: 0019415
2018-08-26 19:48 AOSP Note Edited: 0019415 View Revisions
2018-08-26 19:49 AOSP Note Edited: 0019415 View Revisions
2018-08-27 02:43 WubTheCaptain Status acknowledged => confirmed
2018-08-27 07:05 Zalewa Status confirmed => acknowledged
2018-08-27 07:05 Zalewa Category Bug => Suggestion
2018-09-13 20:08 Zalewa Relationship added related to 0003483
2018-09-29 04:51 Blzut3 Target Version => 1.3
2018-09-29 04:51 Blzut3 Relationship deleted related to 0003483
2018-10-11 21:42 WubTheCaptain Priority high => low
2018-10-17 17:17 Pol M Note Added: 0020130
2018-10-17 18:35 Pol M Note Added: 0020131
2018-10-18 00:56 WubTheCaptain Assigned To => Pol M
2018-10-18 00:56 WubTheCaptain Status acknowledged => assigned
2018-10-20 03:12 Blzut3 Relationship added related to 0003552
2018-11-23 17:25 Pol M Note Added: 0020177
2018-11-23 17:26 Pol M Note Edited: 0020177 View Revisions
2018-11-23 21:33 Zalewa Note Added: 0020178
2018-11-24 22:23 Pol M Note Added: 0020179
2018-11-25 08:38 Zalewa Note Added: 0020180
2018-12-06 22:29 Pol M Note Added: 0020225
2018-12-07 11:20 WubTheCaptain Priority low => normal






Questions or other issues? Contact Us.

Links


Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker