References

Functions download_images and verify_images functions are based on the below sources. Please visit attached links for further understanding.

• fastai library by Jeremy Howard and contributors: Building your own dataset documentation
• Blog post by Mark Needham: Parallel download files using requests

Download

download_image[source]

download_image(dest, url, timeout=5)

Downloads an image from url to destination file path (pathlib.Path object). Tries to detect and preserve original suffix and defaults to '.jpg' if none Assigns a randomly generated filename to avoid name clashes.

Additional links on user agents to handle HTTP 406:

• Python Requests HTTP Response 406
• Detect your user agent by whatismybrowser.com

download_images[source]

download_images(url_file_path, dest, n_threads=4, timeout=5)

Downloads images from list of urls. Makes use of multiprocessing library to support concurrent downloads if supported by CPU. Visualizes progress using tqdm.

Verify

verify_image_tf[source]

verify_image_tf(img_file)

Verifies if an image can be opened by tf.image module tf.image currently supports only BMP, JPEG, GIF and PNG. Function attempts to convert non-supported formats to JPEG. If the image cannot be opened or converted, it is deleted.

verify_images[source]

verify_images(file_path, n_threads=4, recurse=False)

Verifies images from file path. Makes use of multiprocessing library to support concurrent verification if supported by CPU. Visualizes progress using tqdm.