Managing big data [solved]

Linux Support
1

Happy New Year everyone…

I thought I’d start 2020 with a nice new thread.

I have a 1TB external drive and three external backups of it. Although I’m fairly careful about taking backups, I allowed the large amount of space to cause me to get sloppy about organising data. Consequently, there are a lot of duplicate files, plus it’s not always obvious where to find things. I need to tidy up.

That’s going to involve a lot of deleting and moving files, which is fine, but I need to duplicate the effort on all four drives. Not so easy.

My first thought was to do it on one drive, then just empty the others, one by one, and copy the whole lot across. The problem with that is that if any of the files on the first drive are corrupt, the three backups will also end up corrupt.

My second thought was to set up a raid 1 array which, as I understand it, would mirror any changes I make on the first drive without copying anything from one drive to another. There seem to be problems with that idea though, in that the four drives would cease to be separate entities which I could return to using as incremental backups.

I’m becoming resigned to doing this manually (and over a long period of time) unless anyone can suggest a better idea, (which must include data security as the paramount consideration).

This is your mission, should you choose to accept it…

Thanks in advance.

M

2

I use GRSync which I find serves my purpose. You could sort out the one drive and then copy things to the other drives using GRSync. That`s just my observation. Others might suggest something that may be better for you.
Regards
Tony

And a Happy New Year to all.

3

[quote author=wishbone link=topic=13745.msg111027#msg111027 date=1577959618]
I use GRSync which I find serves my purpose. You could sort out the one drive and then copy things to the other drives using GRSync. That`s just my observation. Others might suggest something that may be better for you.
Regards
Tony

And a Happy New Year to all

4

Thanks Tony,

But that still leaves me with the problem in my own first solution…

do it on one drive, then just empty the others, one by one, and copy the whole lot across. The problem with that is that if any of the files on the first drive are corrupt, the three backups will also end up corrupt.

I think I’m going to end up doing this manually. I suppose there’s no real hurry, as long as all the data is backed up at all times.

Regards,

M