One problem with the code as given is that the entire file will be read into memory, and then written out to the file; briefly, but for a moment your program could get very memory hungry. urlretrieve will write the file out in chunks. shutil.copyfileobj will also copy between file objects in a chunked manner. wget is rather blunt, and will download all files it finds in a directory, though as we noted you can specify a specific file extension. If you want to be more granular about which files you download, you can use Python to parse through the data file links it finds and have it download only the files you really want. This is especially useful
Continue reading