MantisBT - Doomseeker
View Issue Details
0003369Doomseeker[All Projects] Suggestionpublic2018-02-10 03:282019-02-21 06:24
Marcaek 
Pol M 
normalfeaturealways
needs testingopen 
 
1.3 
0003369: Wadseeker desperately needs a way to compare downloaded wads to wads being requested by server
On multiple occasions Wadseeker has downloaded incorrect files which are modified but have the same name across multiple websites. It makes little sense that the game should launch with incorrect files.
No tags attached.
related to 0003552resolved WubTheCaptain DogSoft, included in Wadseeker's WAD sites, often gives outdated WADs (and/or is very slow) 
Issue History
2018-02-10 03:28MarcaekNew Issue
2018-02-10 12:59KorshunNote Added: 0019027
2018-02-10 13:00KorshunNote Edited: 0019027bug_revision_view_page.php?bugnote_id=19027#r11388
2018-02-10 13:05KorshunNote Edited: 0019027bug_revision_view_page.php?bugnote_id=19027#r11389
2018-02-10 21:53Blzut3Note Added: 0019028
2018-02-11 05:08Blzut3ProjectZandronum => Doomseeker
2018-02-11 09:30ZalewaNote Added: 0019029
2018-02-11 09:55Blzut3Note Added: 0019030
2018-02-12 22:12WubTheCaptainSeveritymajor => feature
2018-02-12 22:12WubTheCaptainStatusnew => acknowledged
2018-02-12 22:12WubTheCaptainProduct Version3.0 =>
2018-02-12 22:14WubTheCaptainNote Added: 0019032
2018-02-13 04:01Blzut3Note Added: 0019033
2018-02-15 18:15WubTheCaptainNote Added: 0019055
2018-08-25 22:42AOSPNote Added: 0019399
2018-08-25 22:48Blzut3Note Added: 0019400
2018-08-25 22:53AOSPNote Added: 0019401
2018-08-26 19:46AOSPNote Added: 0019415
2018-08-26 19:48AOSPNote Edited: 0019415bug_revision_view_page.php?bugnote_id=19415#r11699
2018-08-26 19:49AOSPNote Edited: 0019415bug_revision_view_page.php?bugnote_id=19415#r11700
2018-08-27 02:43WubTheCaptainStatusacknowledged => confirmed
2018-08-27 07:05ZalewaStatusconfirmed => acknowledged
2018-08-27 07:05ZalewaCategoryBug => Suggestion
2018-09-13 20:08ZalewaRelationship addedrelated to 0003483
2018-09-29 04:51Blzut3Target Version => 1.3
2018-09-29 04:51Blzut3Relationship deletedrelated to 0003483
2018-10-11 21:42WubTheCaptainPriorityhigh => low
2018-10-17 17:17Pol MNote Added: 0020130
2018-10-17 18:35Pol MNote Added: 0020131
2018-10-18 00:56WubTheCaptainAssigned To => Pol M
2018-10-18 00:56WubTheCaptainStatusacknowledged => assigned
2018-10-20 03:12Blzut3Relationship addedrelated to 0003552
2018-11-23 17:25Pol MNote Added: 0020177
2018-11-23 17:26Pol MNote Edited: 0020177bug_revision_view_page.php?bugnote_id=20177#r12271
2018-11-23 21:33ZalewaNote Added: 0020178
2018-11-24 22:23Pol MNote Added: 0020179
2018-11-25 08:38ZalewaNote Added: 0020180
2018-12-06 22:29Pol MNote Added: 0020225
2018-12-07 11:20WubTheCaptainPrioritylow => normal
2018-12-13 17:36Pol MNote Added: 0020244
2018-12-13 17:37Pol MNote Edited: 0020244bug_revision_view_page.php?bugnote_id=20244#r12300
2018-12-13 17:40Pol MNote Edited: 0020244bug_revision_view_page.php?bugnote_id=20244#r12301
2018-12-13 17:42Pol MNote Edited: 0020244bug_revision_view_page.php?bugnote_id=20244#r12302
2018-12-13 17:43Pol MNote Edited: 0020244bug_revision_view_page.php?bugnote_id=20244#r12303
2018-12-14 16:58ZalewaNote Added: 0020245
2018-12-14 17:12Pol MNote View State: 0020244: public
2018-12-14 17:12Pol MNote Added: 0020246
2018-12-17 03:32WubTheCaptainNote Deleted: 0020246
2018-12-17 03:32WubTheCaptainNote Deleted: 0020245
2018-12-20 16:32Pol MNote Added: 0020273
2018-12-20 16:36Pol MNote Edited: 0020273bug_revision_view_page.php?bugnote_id=20273#r12325
2018-12-20 17:10ZalewaNote Added: 0020274
2018-12-20 17:23Pol MNote Added: 0020275
2018-12-21 14:58Pol MNote Added: 0020277
2018-12-21 14:59Pol MNote Edited: 0020277bug_revision_view_page.php?bugnote_id=20277#r12327
2018-12-22 21:10ZalewaNote Added: 0020279
2018-12-23 22:51Pol MNote Added: 0020280
2018-12-31 01:16Pol MNote Added: 0020288
2018-12-31 01:28ZalewaNote Added: 0020289
2018-12-31 01:30Pol MNote Edited: 0020288bug_revision_view_page.php?bugnote_id=20288#r12337
2018-12-31 01:31Pol MNote Added: 0020290
2018-12-31 01:38Pol MNote Added: 0020291
2018-12-31 01:42Blzut3Note Added: 0020292
2018-12-31 01:47ZalewaNote Added: 0020293
2018-12-31 01:47ZalewaNote Edited: 0020293bug_revision_view_page.php?bugnote_id=20293#r12339
2018-12-31 01:47ZalewaNote Edited: 0020293bug_revision_view_page.php?bugnote_id=20293#r12340
2018-12-31 01:49Blzut3Note Added: 0020294
2018-12-31 18:18Pol MNote Added: 0020295
2018-12-31 18:32Pol MNote Edited: 0020295bug_revision_view_page.php?bugnote_id=20295#r12342
2018-12-31 22:50underscoreNote Added: 0020296
2019-01-01 10:58Pol MNote Added: 0020298
2019-01-01 11:19Pol MNote Edited: 0020298bug_revision_view_page.php?bugnote_id=20298#r12344
2019-01-01 16:01ZalewaNote Added: 0020299
2019-01-02 20:59Pol MNote Added: 0020300
2019-01-08 20:15Pol MNote Added: 0020308
2019-01-15 22:12Pol MNote Added: 0020310
2019-01-15 22:46Pol MNote Edited: 0020310bug_revision_view_page.php?bugnote_id=20310#r12358
2019-01-15 23:53Pol MStatusassigned => needs review
2019-01-21 23:31ZalewaNote Added: 0020317
2019-01-21 23:31ZalewaStatusneeds review => needs testing
2019-02-21 01:28MarcaekNote Added: 0020421
2019-02-21 06:24ZalewaNote Added: 0020423

Notes
(0019027)
Korshun   
2018-02-10 12:59   
(edited on: 2018-02-10 13:05)
How would it do that if Zandronum servers don't publish wad hashes until you've tried to connect?

How about making Zandronum servers publish a list of wad hashes along with wad names? According tohttps://wiki.zandronum.com/Launcher_protocol [^] , introducing SQF_PWADHASHES won't break backwards compatibility.

EDIT: the comma breaks the URL.

EDIT2: publishing wad hashes would also allow directly querying for wads by hash, instead of downloading by name and verifying using the hash.http://wad-archive.com/ [^] has that functionality and can provide many download links for any wad hash, but I haven't seen any other places that allow querying by wad hash.

(0019028)
Blzut3   
2018-02-10 21:53   
For some reason I thought we already added hashes to the protocol, but don't see it in the source (probably was thinking of SQF_DATA_MD5SUM which is no longer used). In any case though Odamex already exposes thing information so it could be made part of our API today.
(0019029)
Zalewa   
2018-02-11 09:30   
If we could recognize hosted WADs by hash and download them by this hash, there still would remain a problem of having duplicate WADs on disk.

In "ye olde" times, especially when it was about WADs designed for multiplayer, the creators were usually aware of the duplicated WADs problem and incorporated WAD's version number in to the WAD's file name. It was simple: you changed something in the WAD, then you also changed its name and, thanks to this, there were no incompatible "duplicates" being hosted over different servers. I can understand keeping the same filename when the WAD is designed for single player, but if the author claims "multiplayer: designed for" then the author should be aware how big of a problem it is to release an incompatible version with the same name.

However, let's assume that the server browser can differentiate WADs by hashes. It also detects that the user, in their "WAD downloads" directory, already has the WAD with the same name. What should it do?

a) Tell the user that there's a duplicate and let them choose to overwrite or cancel the download, optionally allowing to ignore the problem? In the last case the issue or pressing "Ignore" all the time will become annoying.

b) Incorporate the hash into the duplicated WAD's filename when it's downloaded? If we see "mod.wad" but hash mismatches we can always look for "mod_abcdef1234567890hashishash.wad". If that is not found we can always ask to download & rename.
(0019030)
Blzut3   
2018-02-11 09:55   
At the very least supporting checking the hash if the plugin supports it and telling the user that there's a mismatch before launching would be useful.

I don't think HTTP provides any standardized way of asking for a content match by hash, so as far as fetching goes outside of something like wad-archive I think the only option is to get a full list of URLs and try them one by one until a match is found. Off hand the only thing we could possibly do to improve that is implement a custom header that hosts could choose to do something with, but I wouldn't expect any sizable number of servers to comply. (Web browser caching relies on the "Etag" header which appears to be an arbitrary server side value so not useful to us.)

I like your idea on auto-renaming files with the hash if there's a duplicate. My initial thought was to just blindly overwrite (assuming the user accepts the usual "would you like to download" dialog), but that sounds like it would be fairly easy to implement and solve the problem nicely.

From the impression I got when Marcaek when he told me of the issue I would assume the bigger problem is Wadseeker picking the wrong version for people without any version of the wad since an older version happens to be hosted on one of our default locations. With that in mind having Wadseeker hash check and reject would be useful even if the duplicate problem isn't solved.
(0019032)
WubTheCaptain   
2018-02-12 22:14   
Is the future of Wadseeker to be a WAD downloader or a WAD (file) manager? In Doomseeker (a server browser)?
(0019033)
Blzut3   
2018-02-13 04:01   
Not sure where your question comes from. The goal is to get people into servers with the right wads loaded in the case of Doomseeker. For Wadseeker it to find and download the correct wad.
(0019055)
WubTheCaptain   
2018-02-15 18:15   
Quote from Blzut3
The goal is to get people into servers with the right wads loaded in the case of Doomseeker. For Wadseeker it to find and download the correct wad.


Oh sorry, I was thinking a bit naively. I'm inclined to support this, then.
(0019399)
AOSP   
2018-08-25 22:42   
This should do for getting the hashes from the server, right?https://bitbucket.org/csnxs/zandronum-stable2/commits/0041e93150f4c2734ebf4574acf49fb6f768ef2f?at=sb-query-md5s [^]

Just checking before I send a PR upstream...
(0019400)
Blzut3   
2018-08-25 22:48   
Check if there's anything that needs to be done for DEH patches and optional wads, but otherwise that would indeed be what we need.
(0019401)
AOSP   
2018-08-25 22:53   
Optional WADs work fine, but I don't think DEH patches have their hashes stored anywhere (they're not even checked when connecting), and they're so infrequently used that I don't think it's worth adding them. I'll submit the PR if nobody has anything else to say.
(0019415)
AOSP   
2018-08-26 19:46   
(edited on: 2018-08-26 19:49)
Torr noted that we were running out of space for fields, so a new field for more information was created and it was moved there.

Change in zandronum-stable
Also documented on the wiki (diff)

(0020130)
Pol M   
2018-10-17 17:17   
This looks like an interesting ticket. I'm gonna do some work on getting the hashes from the server ftm. (I'll extend PWad with a variable, and a constructor)
(0020131)
Pol M   
2018-10-17 18:35   
PR (gets the hashes)
(0020177)
Pol M   
2018-11-23 17:25   
(edited on: 2018-11-23 17:26)
I'm gonna do some extra (small) stuff. Right now I think the next logical step is to get Doomseeker to:
- Make it able to hash WADs
- List in the WADs column the ones that are incompatible (when you hover the mouse over it)
- Make it acknowledge the incompatible WADs and not simply execute the client when you double-click a server (and do something also with "Find missing WADs" on a right-click).

So, the first part should be relatively easy. As far as I can tell WadPathFinder is used to deal with wads, so it should be easy to extend it to incorporate a hashing function and add whether is necessary to WadFindResult to accommodate the changes.

The second part should also be simple. ServerToolTip->createPwadToolTipInfo handles this and with the previous changes, there should be np.

With the last part, it seems trickier. What I've thought for the moment is to modify MissingWadsDialog constructor to get a new argument for the corrupted/non-equal WADs. I'll look at it more the next days, but it seems the only calls it gets are from JoinCommandBuilder (double-click) and a signal in MainMenu ("Find missing WADs" on a right-click) (If my guesses are correct). I'll get something working :)

(0020178)
Zalewa   
2018-11-23 21:33   
1 & 2 sound good, although Wadseeker will also need to be able to calculate the same hash so decide carefully where you put the code.

Point 3. can ask the user if they want their WADs redownloaded and overwritten and then just replace the existing files, although Wadseeker would also need to compare the hash before overwriting. Or Doomseeker could take the approach with incorporating the hash into the WAD name as I mentioned in my comment 0003369:0019029, though that option will be incompatible with other launchers.

Wadseeker could EXPORT a Checksum object with "enum Algorithm algorithm" and "QByteArray hash" types. Algorithm enum will initially only have MD5 value, but we will be able to expand it with other types in the future. It must also be possible to feed the Checksum objects together with requested WADs for the search operation and possibly consider that a single WAD may have more than one Checksum object if we'd ever like to query WAD archives by passing the checksums and those WAD archives would only support one of the algorithms.
(0020179)
Pol M   
2018-11-24 22:23   
Quote from Zalewa

1 & 2 sound good, although Wadseeker will also need to be able to calculate the same hash so decide carefully where you put the code.

Welp, this means that it will have to be located in Wadseeker if we don't want to duplicate code

Quote from Zalewa

can ask the user if they want their WADs redownloaded and overwritten and then just replace the existing files

I'll take a similar approach to optional WADs, with the difference that it will prompt regardless if there are required WADs to download or not.

Quote from Zalewa

although Wadseeker would also need to compare the hash before overwriting

I guess you mean once the WAD is downloaded to check if the WAD is the one used by the server. I agree that we should try to prevent overwriting files when it is not necessary.

Quote from Zalewa

Or Doomseeker could take the approach with incorporating the hash into the WAD name as I mentioned in my comment 0003369:0019029

Huh, this would be interesting. I'd agree on trying to conserve files, but maybe stuff would be more organized in directories? creating a directory with the name (Ex: "coolpk3mod/") and then inside the file with the hash (Ex: "coolpk3mod/6e38e7975656200ff6dccc11919a2e23.pk3"). Also, since I see we're extending this to other hash algorithms, maybe we could also append it's name (Ex: "coolpk3mod/md5_6e38e7975656200ff6dccc11919a2e23.pk3"). All that said, for the moment I'd be inclined on dealing with this in the future and for the moment focus on having only the right WADs for the connection. (My approach has the inconvenience of dealing with permissions)

Quote from Zalewa

though that option will be incompatible with other launchers.

Another reason to deal with this later on :P

Quote from Zalewa

Wadseeker could EXPORT a Checksum object with "enum Algorithm algorithm" and "QByteArray hash" types. Algorithm enum will initially only have MD5 value, but we will be able to expand it with other types in the future. It must also be possible to feed the Checksum objects together with requested WADs for the search operation and possibly consider that a single WAD may have more than one Checksum object if we'd ever like to query WAD archives by passing the checksums and those WAD archives would only support one of the algorithms.

Let's see if we are on the same page: The objective is to feed Wadseeker with names and Checksums(object with the hash and the algorithm) and get all the WADs that match a checksum (and also match one of the names) and also get the WADs that match the names (since we may not know any hash).
This has the problem that if we don't find any WAD that satisfies the checksum we won't know which WAD we're missing.

We could do something similar to the Pwad class: Wadseeker would EXPORT a class with a QString with the name of the WAD and the QList<Checksum> with all the checksums we could ever need. Then Wadseeker would look for WADs with the name and then check if that WAD validates with one of the checksums. If it does, or no checksums exist, we'd proceed to download, and if not, we'd keep looking. Also, in case we don't know the name (aka: name.IsEmpty() returns true) we would look for WADs that satisfy all the hashes we do know. If we don't find anything, We'll know the name of what we're missing (well, except if it is the case we're searching using only checksums)

I'm gonna take the wild guess, though, that not many pages give the hashes of their wads, so we'd probably have to download the wad to validate the checksums ourselves. Also, the search "without the name" will be limited.
(0020180)
Zalewa   
2018-11-25 08:38   
Quote from Pol M
Also, the search "without the name" will be limited.

The filename is always expected to be provided by the game and the program needs to know what it looks for and the hash by itself is not good enough. I was thinking about utilizing APIs such as WAD-Archive's (https://www.wad-archive.com/api [^] ) where you can use the hash directly in the query.

Quote from Pol M
I'd agree on trying to conserve files, but maybe stuff would be more organized in directories? creating a directory with the name (Ex: "coolpk3mod/") and then inside the file with the hash (Ex: "coolpk3mod/6e38e7975656200ff6dccc11919a2e23.pk3"). Also, since I see we're extending this to other hash algorithms, maybe we could also append it's name (Ex: "coolpk3mod/md5_6e38e7975656200ff6dccc11919a2e23.pk3").

The problem with trying to manage the WADs in a more complicated way than the current one is that the storage directory will become much more difficult to browse from a file explorer. Right now as there's a 1-to-1 matching of WADs to their filenames it's easy to find files that were downloaded lately or to just find them by name. Moreover, once we start to modify the filename, the user won't be able to host the server with this WAD without incurring further name confusion or without changing the name back manually.

Simply displaying the information about invalid checksums seems to be the least invasive and the best approach for now, but it won't solve problem with having A/B versions of the same WAD hosted in the wild. It will only be helpful in cases where the WAD got an update and everyone is hosting the new version while the user still has the previous one.

Quote from Pol M
I'll take a similar approach to optional WADs, with the difference that it will prompt regardless if there are required WADs to download or not.

Yes, that sounds exactly good.
(0020225)
Pol M   
2018-12-06 22:29   
Ok, I've made quite a few changes lately. I'll push something to the same PR today or tomorrow (I finally got a free day😄). More notes in bitbucket once I'm finished.
(0020244)
Pol M   
2018-12-13 17:36   
(edited on: 2018-12-13 17:43)
Time for an update on what is going on!
So, the summary of the features of the pull request would be:
- Allow plugins to pass any amount of checksums with the pwads
- When trying to join a server or when clicking the "Find Missing WADs" button (right click on a server), doomseeker will check for invalid hashes
- The MissingWadsDialog lists incompatible wads, which you can de/select (selected by default) (very similar as optional WADs)
- Incompatible wads are listed in the "WADs" column of the main window

Now, this last feature sparked some questions from Zalewa:
Quote from Zalewa

It is good to have the valid/invalid state displayed in the tooltip, but what performance impact does it have?

To answer this I proceeded to make some benchmarking:
I added a few lines to measure the amount of time it took to complete the list of wads. I gathered a few massive wads and I hosted a Zandronum 3.0 and a Zandronum 3.1 servers. The list of wads is as follows:
Quote from ls command

bd21rc2b.pk3 bulletpuff_fix.wad cc4-tex.wad complexbrightmaps.pk3 complex-doom.v26a2.pk3 complexlcazan3soundpatch.wad Damnation.wad dbzone_alternative_6.0.pk3 hpbar-v16b.pk3 “MIDI Music.wad” newtextcolours_260.pk3 q2tex.wad skulltag_actors.pk3 skulltag_data.pk3

Total size: 491,6 MB
(MIDI Music.wad came with Damnation.wad)

The server wads and the client wads were in separate directories (just in case. Also, I forgot to mention I also corrupted the wads to know that the wads were being checked (more just in case))

Between neasurements I cleaned the cache of my HDD with "sync; blockdev --flushbufs /dev/sda; echo 3 > /proc/sys/vm/drop_caches" (thx internet) to achieve consistent measurements.
I’ve measured both setups 10 times, getting:

With checksums:
4710
5053
4689
4771
4710
4783
4722
4550
4706
4721
Average: 4741.5

Without checksums:
187
189
174
193
248
176
189
204
204
172
Average: 193.6

Quote from Zalewa

This means there will be a 5 seconds long program-wide hang up not only when displaying the tooltip but also when joining a server. This needs to be taken under consideration. We don’t want the join game operation to feel like it’s trying to read doom.exe off a floppy disk.

Ideas?

  1. Make checksum comparison an explicit action from the server’s context menu and don’t do it anywhere else. If the game fails to join a server, user will be able to return to Doomseeker and compare checksums in a separate dialog box with a responsive UI.

  2. Allow to configure the file size limit of the automatic checksum comparison. This is probably a poor idea considering that it will rule out most of the mods that may change frequently.

  3. ???




Aaaand that's pretty much it, right now ideas for what should be done are beeing discussed in order to solve this timing problem.😊

(0020273)
Pol M   
2018-12-20 16:32   
(edited on: 2018-12-20 16:36)
Okay, I've thought about a possible solution. First of all, I think that a failed connection is what Doomseeker should try to prevent by all means, so that not tech-savvy users can play doom. While adding an option to the left click seemed like a cool idea at first, it would mean that the user would have to see first an error screen and then figure out where the "Check files" button is (for instance, as I'm told Macs rarely use this functionality). If the user gives up or googles how to solve it, then this feature is not really that useful. It came to my mind (for a second) adding a button for that in the MissingWads dialog, which is also terrible since it should only appear if wads are missing (duhhh :)). With all of this, I believe that showing a dialog before joining a server detailing that files are being checked (with a progress bar, the file that is being checked on a QLabel and a Cancel/Skip button) would be the most noob friendly way to guarantee a successful connection. Of course, a way to disable this feature has to be added to the config menu so that users can decide if they want to check files. (I'd make it opt-out)

EDIT: Not to mention that validations should not be made in any other circumstance.

(0020274)
Zalewa   
2018-12-20 17:10   
So, to sum it up we'd go with:

Option enabled (by default):

1. User tries to join a server and this results in a progress bar pop-up being displayed.
2a. If all files are ok, hide the pop-up and pop-in to the game.
2b. If files are not ok, hide the progress pop-up and pop up the missing WADs dialog.
3. Continue with Wadseeker as in the current pull request.

Option disabled:

Behave as in the current release.

It may also be a good idea to give the checksum check operation a bit of grace period (like, about 500 milliseconds). The progress pop-up should only appear if the grace period expires and the check is still not done.
(0020275)
Pol M   
2018-12-20 17:23   
Quote from Zalewa

It may also be a good idea to give the checksum check operation a bit of grace period (like, about 500 milliseconds). The progress pop-up should only appear if the grace period expires and the check is still not done.

That sounds splendid! :D
If we reach consensus, then I'll get into it. For the moment, though, I'd focus on getting Wadseeker to use the Checksums.
In a glance, it looks like it will be a little bit tricky, but hey, nothing is impossible. ^-^
(0020277)
Pol M   
2018-12-21 14:58   
(edited on: 2018-12-21 14:59)
There's a small issue with Wadseeker. It won't be as easy as passing the PWads to Wadseeker to do it's job since it cannot access the PWad class. Also, Putting the PWad class in Wadseeker is not possible because plugins have to access it.

So, my first idea was to create a struct (with the names and the checksum list, the optional bool is not relevant) and simply extend PWad with a conversion function (the good old "operator PWadFromWadseeker()") and a conversion constructor to interact with Wadseeker pretty much seamlessly, but this is maybe not ideal. (It may lead to some code duplication)

Any thoughts on the problem? Maybe there is some magic cpp feature that I'm missing.

(0020279)
Zalewa   
2018-12-22 21:10   
There are ModFile and ModSet classes in Wadseeker that are used for Freedoom installation. Unfortunately these don't have a proper documentation, even though they are exported through WADSEEKER_API. You will have to trace their history and usage to see what's what.

If you look at ModFile it already has traces of checksum comparison as this functionality is actually being used for Freedoom. Feel free to modify this class to suit the new needs and don't worry about the minimal duplications that will occur.

ModFile has some functionality that will not be usable in the context of generic WAD searching. Usually this should be a warning flag stating that this should not be the class for the job, but I actually think that many of those fields can be understood as being optional.
(0020280)
Pol M   
2018-12-23 22:51   
Okay today I've been reading about it, and pretty much everything is in place to pass PWads thanks to the ModFile and ModSet classes (and also use them more internally in Wadseeker rather than simply transform them to QStringLists :P)

I'll probably commit tomorrow once I ensure everything works as usual, since I've done quite a few changes
(0020288)
Pol M   
2018-12-31 01:16   
(edited on: 2018-12-31 01:30)
I've been working in the feature of the Wad-Archive filter (it sounded interesting :P) using the API:https://www.wad-archive.com/api/latest/[md5-here] [^]
I'm receiving this: {"error":"No Bots"}
I'm not sure why, and I don't understand why would a public API not allow bots :/
Maybe someone has some idea of what is going on, I'm a little bit clueless.

I've committed my first approach. Please note that it may not work as intended, but it shows what I mean.

I used the server "[TSPG] Painkiller: ZDoomWars 3.15c Precious Days Mappack LMS" and the pk3 zdwarspdmp1.1.pk3

(0020289)
Zalewa   
2018-12-31 01:28   
Maybe there's an user-agent filter? Are you setting user-agent to Wadseeker?
(0020290)
Pol M   
2018-12-31 01:31   
Quote from Zalewa

Maybe there's an user-agent filter? Are you setting user-agent to Wadseeker?

Yes
(0020291)
Pol M   
2018-12-31 01:38   
I can confirm that replacing user-agent with anything else does the job.
(0020292)
Blzut3   
2018-12-31 01:42   
Curious. I can't imagine that Mega-Dog is hashing files on the fly with that API so not sure why he would lock it out to bots? You should probably send him an email and ask about it as this would be a good solution on our end for solving the issue of wad archive serving the wrong files.
(0020293)
Zalewa   
2018-12-31 01:47   
Blzut, I believe Wad-Archive is run by Hashim, who's not Mega-Dog.

Regardless, I dug out some old emails from 2014 where we cooperated with Hashim to extend the API on his site. The relevant ticket is 0001512. When you go tohttps://www.wad-archive.com/api/ [^] you'll actually notice that there's an explicit Wadseeker API which was made for the purpose of 0001512. Still, I don't know why Hashim blocked Wadseeker's user-agent at the other endpoint.

(0020294)
Blzut3   
2018-12-31 01:49   
Derp! Not sure why I confused dogsoft and wad archive.
(0020295)
Pol M   
2018-12-31 18:18   
(edited on: 2018-12-31 18:32)
Okay, I've put a different header ftm. Also, now if a downloaded wad is incompatible, it will be redownloaded from the next URL. Commit pushed

Just to recapitulate, what is left to do is:
 - Add loading bar for the checksum process when logging into servers and when selecting "Find Missing WADs" in the left-click menu
 - Add an option in the config to disable said function. (maybe we should consider the option to also disable checksum checks in Wadseeker, but I'd be against it since wads will surely be cached, so the performance loss will certainly not be noticeable)
 - ?

(0020296)
underscore   
2018-12-31 22:50   
I have relaxed the user agent restriction on my other APIhttps://www.wad-archive.com/api/latest/333ca955620fd1678619e62976fb96bb [^] . The error message is also updated to provide a better message.

The reason for the restriction is to force users to provide a user agent that gives some idea about what it is being used for rather than generic curl or wget agents. For some reason WadSeeker was on that list. I wrote that API somewhere around 5 years ago so I can't remember the reasoning behind blocking WadSeeker.

"Blzut, I believe Wad-Archive is run by Hashim, who's not Mega-Dog."

I am neither :)
(0020298)
Pol M   
2019-01-01 10:58   
(edited on: 2019-01-01 11:19)
Updated the header! :)
Thank you so much underscore for the quick response!
(Happy new year!)

(0020299)
Zalewa   
2019-01-01 16:01   
Quote from Pol M

- Add loading bar for the checksum process when logging into servers and when selecting "Find Missing WADs" in the left-click menu
 - Add an option in the config to disable said function. (maybe we should consider the option to also disable checksum checks in Wadseeker, but I'd be against it since wads will surely be cached, so the performance loss will certainly not be noticeable)

Sounds good. Wadseeker can do the checksum checks regardless of the setting as it is expected that the installed file will be checked for correctness if the checksum is available.

Quote from underscore

"Blzut, I believe Wad-Archive is run by Hashim, who's not Mega-Dog."

Double-checks the emails. Sees "Hamish" this time. Bah, sorry.
(0020300)
Pol M   
2019-01-02 20:59   
Wad-Archive also supports sha1 (thank you Hamish). I've wired it up :)
(0020308)
Pol M   
2019-01-08 20:15   
Config added (commit pushed)! I'm gonna be quite occupied the following days, so if anyone that already knows how to wire up progress bar dialogs wants to finish the ticket, he's more than welcome. (I've done some quick approximations that ended terribly :/)
(0020310)
Pol M   
2019-01-15 22:12   
(edited on: 2019-01-15 22:46)
I'm working on the loading bar, expect a commit today/tomorrow. This will conclude this ticket! (after the review)

EDIT: commit pushed

(0020317)
Zalewa   
2019-01-21 23:31   
I merged the pull request. This new feature can be tested now by building Doomseeker from the official tip. I'll also produce beta build for Windows some time later this week if no immediate problems arise.
(0020421)
Marcaek   
2019-02-21 01:28   
Looking forward to testing said beta build.
(0020423)
Zalewa   
2019-02-21 06:24   
Windows build is already available on the Beta auto update channel. You can download it through Doomseeker. In the Configuration box go to the "Auto Updates" section and select "Beta" from the "Update channel" combo box, save the configuration and close the config box. Then from the "Help" menu, select "Check for Updates". The program will guide you through the next steps.