notepad++

While I was doing my database project, I encountered thousands of duplicate data, to eliminate these data firstly I used a software called CSVed. It easily eliminates the duplicate data, but it distruped my CSV file schema so that I cannot dump my data into the MySQL.

Then I found very very easy and fast solution, using NotePad++. To eliminate the duplicates you should follow these instructions:

1) Click TextFX –> TextFX Tools –> +Sort outputs only UNIQUE (at coulumn) lines

2) Select all your data

3) Click TextFX –> TextFX Tools –> “Sort lines …  “