While I was doing my database project, I encountered thousands of duplicate data, to eliminate these data firstly I used a software called CSVed. It easily eliminates the duplicate data, but it distruped my CSV file schema so that I cannot dump my data into the MySQL.
Then I found very very easy and fast solution, using NotePad++. To eliminate the duplicates you should follow these instructions:
1) Click TextFX –> TextFX Tools –> +Sort outputs only UNIQUE (at coulumn) lines
2) Select all your data
3) Click TextFX –> TextFX Tools –> “Sort lines … “