Fixing Gallery-dl's 'KeyError: Metadata' On Fandom

Hey guys, so I've been running into a pesky issue lately while scraping Fandom wikis using gallery-dl. It throws a KeyError: 'metadata' error, and it's a real head-scratcher. Let's dive into what's causing this, how to fix it, and what it all means for you. We'll break down the problem, the fix, and some considerations. I'll also add some external links and some important keywords to help you understand.

The Problem: 'KeyError: metadata'

This KeyError: 'metadata' pops up when gallery-dl tries to grab info about images, but the metadata field is missing. This happens because the files don't actually exist or are unavailable. It's like trying to find a book in a library that's been removed. Specifically, the error occurs in the wikimedia.py file within gallery-dl, where the code tries to access image metadata that isn't there. It's a common problem that appears when dealing with image-revisions greater than 1.

The error message in the console looks something like this:

[fandom][error] An unexpected error occurred: KeyError - 'metadata'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[fandom][debug]
Traceback (most recent call last):
 File "/home/[redacted]/programming/gallery-dl/gallery_dl/job.py", line 153, in run
 for msg in extractor:
 ^^^^^^^^^^^
 File "/home/[redacted]/programming/gallery-dl/gallery_dl/extractor/wikimedia.py", line 107, in items
 self.prepare_image(image)
 ~~~~~~~~~~~~~~~~~~^^^^^^^
 File "/home/[redacted]/programming/gallery-dl/gallery_dl/extractor/wikimedia.py", line 85, in prepare_image
 for m in image["metadata"] or ()
 ~~~~~^^^^^^^^^^^^^
KeyError: 'metadata'

This traceback points directly to the prepare_image function, where the code expects metadata for each image. The core issue stems from how the MediaWiki API handles non-existent files.

MediaWiki's API and File Handling

Understanding how the MediaWiki API works is crucial. When you request image information, the API uses prop=imageinfo. However, if a file is missing, the API intentionally omits the metadata field. This change was introduced in MediaWiki version 1.34, and it's the root cause of our KeyError. The API response does not include the metadata field, leading to the error when gallery-dl tries to access it.

The Fix: Skipping Missing Files

The good news is that the fix is pretty straightforward. By checking for the filemissing field, we can tell if an image is missing and skip processing it. This prevents the code from trying to access a non-existent metadata field. Here's how the fix works:

diff --git a/gallery_dl/extractor/wikimedia.py b/gallery_dl/extractor/wikimedia.py
index 2e8136f1..a103a06b 100644
--- a/gallery_dl/extractor/wikimedia.py
+++ b/gallery_dl/extractor/wikimedia.py
@@ -104,6 +104,12 @@ class WikimediaExtractor(BaseExtractor):
  yield Message.Directory, info

  for info["num"], image in enumerate(images, 1):
+ # https://www.mediawiki.org/wiki/Release_notes/1.34
+ if "filemissing" in image:
+ self.log.warning(
+ "File %s (or its revision) is missing",
+ image["canonicaltitle"].partition(":")[2])
+ continue
  self.prepare_image(image)
  image.update(info)
  yield Message.Url, image["url"], image

This fix adds a check for the filemissing key in the image data. If filemissing exists, it means the file is missing, and the code skips processing it. This avoids the KeyError altogether. This simple patch effectively handles cases where images are missing from the MediaWiki API response, preventing the KeyError and allowing gallery-dl to continue working smoothly. By incorporating this check, the script gracefully handles situations where files are missing, ensuring that the program doesn’t crash and that it can continue processing other valid image entries.

Considerations and Implications

While the fix is effective, it has a couple of implications. First, it breaks the continuity of the sequence number if invalid entries appear in the middle of imageinfo, but this is generally acceptable. The key thing is that gallery-dl will run without crashing. Also, the warning message can be improved to include more context or information, like the name of the missing file. This allows for better error handling and gives users more information about what happened.

Impact on Sequence Numbers

One thing to keep in mind is that skipping missing files can affect the sequence numbers of your downloaded images. If you're relying on sequential numbering, there might be gaps. This is because the script will skip the missing file and move on to the next one, leaving a number missing in the sequence. This could be a problem if you're using gallery-dl for tasks that require sequential image numbering.

Improving the Warning Message

The current warning message is pretty basic. To make it more useful, you could add more context, like the name of the missing file. This would help users understand which files are causing issues and why. Better error messages help users debug and understand the problem better. This way, users have enough information to understand what happened. This additional context can make debugging easier.

Conclusion

Dealing with the KeyError: 'metadata' issue in gallery-dl when scraping Fandom/Wikimedia wikis is a common problem. The fix involves adding a check to identify and skip missing files, preventing the error and ensuring that gallery-dl continues to run smoothly. Remember to consider the implications for sequence numbers and the value of improved warning messages. This fix is effective for most use cases. By implementing this fix, you ensure that gallery-dl handles missing image files gracefully, providing a more robust and user-friendly experience. I hope this guide helped you and was easy to understand.

For more information on gallery-dl and its features, visit the gallery-dl GitHub repository. This is where you can find the latest updates, report bugs, and contribute to the project. The repository is the central hub for information, updates, and community support. Also, check the MediaWiki Release Notes for more information. These release notes provide valuable insights into API changes and updates. This can help you stay informed about potential issues.