Linux :: How to prevent wget from creating duplicates

wget is used to download file from internet. For instance:

1	wget http://url/test.csv

So far so good but two things may happen. First, you may interrupt the download. Second, you may load the file again. Sometimes, files are huge and you do not want to download the same file again.

In the first case, this is eve worse: imagine you have downloaded half of the file and you interrupt the process. Then, you call wget again, you wait, it is over and your are happy. However, because there was already a file called “test.csv” locally, wget downloaded the new file into test.csv.1 ! Moreover, it started the download from srcratch.

So, the solution is to used the two options -c and -N .

1	wget -c -N http://url/test.csv

The first one tells to continue an interrupted download where it was stopped. And, the -N option checks the timestamps to prevent the download of the same file.