Welcome guest. Before posting on our computer help forum, you must register. Click here it's easy and free.

Author Topic: Looking for a way to maintain path in this backup method  (Read 1568 times)

0 Members and 1 Guest are viewing this topic.

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Looking for a way to maintain path in this backup method
« on: January 15, 2018, 07:36:31 PM »
I have 3 batches below...

The 1st batch compares between current data at c:\test3\ and last backup data c:\test1\ and writes the output of what would be xcopied to a list, which I appended the output to Files2Copy.txt

Here is test data from the output of /L

Quote
C:\test3\Perl\man\man3\Win32.Event.3
C:\test3\Perl\man\man3\Win32.IPC.3
C:\test3\Perl\man\man3\Win32.Mutex.3
3 File(s)

The 2nd batch conditions the Files2Copy.txt file removing the last line ( such as  3 File(s) ) which removes a crash condition from the 3rd batch below in which all data except for last line is written to xFiles2Copyx.txt

So last line is removed as seen quoted here

Quote
C:\test3\Perl\man\man3\Win32.Event.3
C:\test3\Perl\man\man3\Win32.IPC.3
C:\test3\Perl\man\man3\Win32.Mutex.3

The 3rd batch xcopies data from the xFiles2Copyx.txt list of files from the working directory to the next backup location which I simulated as c:\test2

Problem I found is that it copies the files from /L output to c:\test2\ and doesnt maintain the folder/file structure and I think I might have painted myself in a corner with this one because xcopy isnt able to see the path to the files because the path is specifically stated in the /L output, so because of this xcopy doesnt write the path structure to c:\test2\ from c:\test3\ and this will be a big problem if I run this live and say there are more than one file with same name that have been edited or added since last backup, it will stomp out all older versions of same file name. Additionally its nice to keep to same folder/file structures to keep files associated to their origin vs all pooled together which is the problem I ran into on this project. *Note all of this could be squeezed into a single batch, but I kept each part isolated as i was working out issues with them, then when I got each part working I started fresh with the next part to keep it small and manageable to look at.

So the data copied ends up at

Quote
C:\test2\Win32.Event.3
C:\test2\Win32.IPC.3
C:\test2\Win32.Mutex.3

instead of

Quote
C:\test2\Perl\man\man3\Win32.Event.3
C:\test2\Perl\man\man3\Win32.IPC.3
C:\test2\Perl\man\man3\Win32.Mutex.3

Currently I have test data at

C:\test3\perl\man\man3\

and data acting as the last backup at

C:\test1\perl\man\man3\

and I removed 3 files from C:\test1\perl\man\man3 so that when running the first batch it lists these 3 files as missing from the last backup, so it will back them up to the new backup location at C:\test2\   BUT ... its not keeping folder structure from the /L output because the /L output is direct and xcopy didnt discover its path to create its path I am thinking.


Code: [Select]
c:
cd\.
REM Make List of Files to copy between last backup and current data
xcopy c:\test3\*.* c:\test1\*.* /s/d/y/h/l>>c:\testing\Files2Copy.txt
Code: [Select]
REM Remove Last Line from xcopy /L output to prepare for next read-in list

@echo off & setlocal EnableDelayedExpansion
set row=
for /F "delims=" %%j in (files2copy.txt) do (
  if  defined row echo.!row!>> Xfiles2copyX.txt
  set row=%%j
)
Code: [Select]
REM Process xcopy of all new and edited files from xcopy /L output
set dst_folder=c:\test2\
for /f %%i in (xfiles2copyx.txt) DO xcopy /S/D/Y/H "%%i" "%dst_folder%"
pause

Once I get this worked out I will be blending this together with the making backup locations todays date from the other thread, but since this is different than the last thread I decided to keep them separate to keep them to their titles.

The goal in the end of this is to have the oldest backup with the majority of the data, and each backup thereafter only backing up extra copies of newly edited files and new files as for I have been using a space wasting backup method that copies all data from one location to another and that consumes space fast and is wasteful of drive space when only data that has changed or been added should be saved for a data trail that is minimalist in storage capacity usage as well as every edit of specific files that are edited regularly is saved at each backup. My current backup method is flawed and this corrects for an issue I had which was that a file got corrupted but it had a newer date stamp and so the 7 day backup in which every day it overwrites the backup from 7 days ago the problem wasnt caught for about 2 weeks when someone needed to open the document and it was a bad file. When they contacted me to get the last good copy of the file I found that for the last 7 days that file hasnt changed and every daily backup had the same corrupt file. I got lucky in getting them an older copy because monthly I backup to a 1TB hard drive and each backup never overwrites the old, but i realized that i need to fix this overwriting backup method which has been wasteful of drive space and flawed because if a file that is a problem isnt caught soon enough then the latest backups overwrite the last good copy of that file at the daily backup level and if its not on the monthly backup then that's not good.  ::)

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #1 on: January 16, 2018, 12:38:51 PM »
Quote
Here is test data from the output of /L
I'm a bit puzzled... what is "/L"?

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #2 on: January 16, 2018, 12:55:09 PM »
Sorry about that confusion... its the output of xcopy switch /L as in

Quote
/L           Displays files that would be copied.
  :)

Which I use that output as a means to know what the difference is between current data and the last backup, which I condition to remove the last line which is file count from this appended to text file in the 2nd batch. Then in the last batch it uses this list of files with full pathing to backup the latest backup to c:\test2\. However the problem I have is that I am puzzled in how I can maintain a way to keep the path so that data xcopied to latest backup which doesnt overwrite the old backup, maintains its folder structure vs everything thrown into the root of the C:\test2\   location.

So I would like to have the data placed into its original folder structure like this:
Quote
C:\test2\Perl\man\man3\Win32.Event.3
C:\test2\Perl\man\man3\Win32.IPC.3
C:\test2\Perl\man\man3\Win32.Mutex.3

Instead of currently it wants to throw all files into the same folder but it doesnt create the perl\man\man3\ structure so if restoring from a backup files it makes for a mess as well as if there are files of same name from multiple users whichever is newest will overwrite the oldest.

The perl stuff is just test data. I copied files and folders to c:\test3 which is acting as the current data, which then I copied to c:\test1 which is the last backup. I then removed 3 files from the last backup manually so that these 3 files would look like new data to pass to c:\test2   however the problem I hit was that the folder structure is flattened and I need to maintain the structure so that at c:\test2 it looks like what is quoted above vs what is quoted below. The pathing is as taken from the output of xcopy /L which is appended to text file to be used in the 3rd batch which species which files need to xcopy to C:\test2  pathing is dynamic and unknown and batch3 runs with whatever paths are within the xfiles2copyx.txt  , batch3 needs to be tweaked to maintain creation of the folder structure from source to destination, and not sure if i will need to parse out this information from each line of this text file and have it create the structure first and then populate the files to the folders. - or- if there is a better way to do this.

But currently getting this. This below is bad and is currently happening where the folder structure isnt created at the destination, and all files no matter of where they came from get pooled together.  :-\
Quote
C:\test2\Win32.Event.3
C:\test2\Win32.IPC.3
C:\test2\Win32.Mutex.3

*Lastly in case anyone catches this set row= in the second batch as to not making such sense and it looks like broken scripting because usually you put something for something else to = to. Its intentional that row= nothing with no characters following it. I took a batch example that has a replace last line with some other information and by setting row to nothing then it wipes out the last line which is what i want.  ;D



Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #3 on: January 16, 2018, 01:20:26 PM »
Quote
not sure if i will need to parse out this information from each line of this text file and have it create the structure first and then populate the files to the folders
At first look, this is what I would be thinking of doing.

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #4 on: January 16, 2018, 02:10:03 PM »
dunno if this helps

@echo off
REM you need this for the exclamation points!
setlocal enabledelayedexpansion

set source=c:\test3
set destin=c:\test2

REM You have a list of these?
set sfile=C:\test3\Perl\man\man3\Win32.Event.3

REM Replace source folder with destination folder
set "dfile=!sfile:%source%=%destin%!"

REM %dfile% becomes c:\test2\Perl\man\man3\Win32.Event.3

echo source      %sfile%
echo destination %dfile%

REM get the folder the file is to go to
for /f "delims=" %%A in ("%dfile%") do set destdir=%%~dpA
REM destdir is c:\test2\Perl\man\man3\

REM now you copy "%sfile%" to %destdir%"


DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #5 on: January 16, 2018, 03:43:01 PM »
Looking at what you shared and thank you for helping with this... kind of curious on this one part:

Quote
REM You have a list of these?
set sfile=C:\test3\Perl\man\man3\Win32.Event.3

For the fact that the list is contained within xfiles2copyx.txt I would think that I would need to run a loop to run line for line until end of file vs setting many static paths. So sfile I would think would take an input from the list file for each line run through setting sfile dynamically to whatever is contained on the line in the list file which xfiles2copyx.txt is.  :-\

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #6 on: January 17, 2018, 01:04:35 AM »
This seems to work on specimen file
You might want copy /y for silent overwrite... As usual, change echo copy to copy to make it actually do the copying. Or leave it and put a real copy line underneath so the user sees what is being copied. Much could be added to make it user friendly.

This is xfiles2copyx.txt

Code: [Select]
C:\test3\Perl\man\man3\Win32.Event.3
C:\test3\Perl\man\man3\Win32.IPC.3
C:\test3\Perl\man\man3\Win32.Mutex.3

Code: [Select]
@echo off
setlocal enabledelayedexpansion
set source=c:\test3
set destin=c:\test2
For /f "delims=" %%F in (xfiles2copyx.txt) do (
set sfile=%%F
set "dfile=!sfile:%source%=%destin%!"
for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA
echo copy "!sfile!" "!destdir!"
)

Output:

Code: [Select]
copy "C:\test3\Perl\man\man3\Win32.Event.3" "c:\test2\Perl\man\man3\"
copy "C:\test3\Perl\man\man3\Win32.IPC.3" "c:\test2\Perl\man\man3\"
copy "C:\test3\Perl\man\man3\Win32.Mutex.3" "c:\test2\Perl\man\man3\"
« Last Edit: January 17, 2018, 01:38:58 AM by Salmon Trout »

jakel



    Starter

    • Yes
  • Experience: Experienced
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #7 on: January 17, 2018, 06:03:54 AM »
want to help , dates are important i see
« Last Edit: January 17, 2018, 06:18:47 AM by jakel »

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #8 on: January 17, 2018, 12:20:17 PM »
Hello Salmon ... so i tried what you made and found an issue in which the directory structure if not already created at C:\test2 then it wont copy to the destination. If I manually go in and make the directory structure to match to the source it copies, but if the directory structure isnt created first, then it fails to copy to the destination.

I tested this out on my wifes computer in which I copied some junk data to c:\test1 ( which acts like the last backup location ).
Then copied the same data to c:\test3 ( which acts as the working directory for all data as if its a NAS )
Then deleted 6 files from c:\test1 ( this way c:\test3 contains data that is not present at the last backup )
I then run the first batch and it sends the output of xcopy /L to files2copy.txt

This creates this list:

Quote
C:\test3\CrystalDiskInfo7_1_1\DiskInfo.ini
C:\test3\CrystalDiskInfo7_1_1\DiskInfo32.exe
C:\test3\CrystalDiskInfo7_1_1\DiskInfo64.exe
C:\test3\CrystalDiskInfo7_1_1\License\CAlphaButton.txt
C:\test3\CrystalDiskInfo7_1_1\License\COPYRIGHT-ja.txt
C:\test3\CrystalDiskInfo7_1_1\License\COPYRIGHT.txt
6 File(s)

I then run the second batch which removes the last line of this to remove the 6 File(s) portion and this is written to xfiles2copyx.txt with this list:

Quote
C:\test3\CrystalDiskInfo7_1_1\DiskInfo.ini
C:\test3\CrystalDiskInfo7_1_1\DiskInfo32.exe
C:\test3\CrystalDiskInfo7_1_1\DiskInfo64.exe
C:\test3\CrystalDiskInfo7_1_1\License\CAlphaButton.txt
C:\test3\CrystalDiskInfo7_1_1\License\COPYRIGHT-ja.txt
C:\test3\CrystalDiskInfo7_1_1\License\COPYRIGHT.txt

Now this third batch as is with c:\test2 empty of directory/folder structure with echo enabled shows this:

Quote
copy "C:\test3\CrystalDiskInfo7_1_1\DiskInfo.ini" "c:\test2\CrystalDiskInfo7_1_1
\"
copy "C:\test3\CrystalDiskInfo7_1_1\DiskInfo32.exe" "c:\test2\CrystalDiskInfo7_1
_1\"
copy "C:\test3\CrystalDiskInfo7_1_1\DiskInfo64.exe" "c:\test2\CrystalDiskInfo7_1
_1\"
copy "C:\test3\CrystalDiskInfo7_1_1\License\CAlphaButton.txt" "c:\test2\CrystalD
iskInfo7_1_1\License\"
copy "C:\test3\CrystalDiskInfo7_1_1\License\COPYRIGHT-ja.txt" "c:\test2\CrystalD
iskInfo7_1_1\License\"
copy "C:\test3\CrystalDiskInfo7_1_1\License\COPYRIGHT.txt" "c:\test2\CrystalDisk
Info7_1_1\License\"
Press any key to continue . . .

And with echo removed to actually copy I get this:

Quote
The system cannot find the path specified.
        0 file(s) copied.
The system cannot find the path specified.
        0 file(s) copied.
The system cannot find the path specified.
        0 file(s) copied.
The system cannot find the path specified.
        0 file(s) copied.
The system cannot find the path specified.
        0 file(s) copied.
The system cannot find the path specified.
        0 file(s) copied.
Press any key to continue . . .

I then thought that what if I create the directory structure at c:\test2 to see if that will fix this and I then got this:

Quote
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
Press any key to continue . . .

Checked c:\test2 and now c:\test2 has the data and there are 3 files now at c:\test2\CrystalDiskInfo7_1_1\ as well as 3 others in c:\test2\CrystalDiskInfo7_1_1\license

So it looks like the fix to this will be to create the folder structure before the files are copied. Scratching my head I had an idea to add MD "!destdir!" before the copy instruction and that fixed the issue, but not sure if its the best method or not. It works now but it attempts to create the structure for each line read in and when it already exists it ignores rewriting the folder structure but complains as seen below:

Quote
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\ already exists.
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\ already exists.
        1 file(s) copied.
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\License\ already exists.
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\License\ already exists.
        1 file(s) copied.
Press any key to continue . . .

 ;D

Here is what I did so you can see what I did from what you helped me with:

Code: [Select]
@echo off
setlocal enabledelayedexpansion
set source=c:\test3
set destin=c:\test2
For /f "delims=" %%F in (xfiles2copyx.txt) do (
set sfile=%%F
set "dfile=!sfile:%source%=%destin%!"
for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA
        MD "!destdir!"
copy "!sfile!" "!destdir!"
)
pause

I can continue to run with this as the solution since it works, but if there is a better way to only create the structure once and move on vs this hitting it with a hammer method that I got to work, I'd be interested in learning how to do it once for making each directory branch to get rid of the A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\License\ already exists message.  :)

Maybe its something like

REM place this where it will be updated with latest path
set currentpath=!destdir!

REM place this at the tail end of the loop
set lastpath=!destdir!

Then if the next iteration ( IF statement tests to see if currentpath and lastpath are equal and if equal skip the MD "!destdir!" process until there is an inequality to which it runs the MD "!destdir!" process.

I put this together below in an attempt to get rid of the folder already existing message, but unsuccessful ...  :-\

Code: [Select]
@echo off
setlocal enabledelayedexpansion
set source=c:\test3
set destin=c:\test2
For /f "delims=" %%F in (xfiles2copyx.txt) do (
set sfile=%%F
set "dfile=!sfile:%source%=%destin%!"
for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

REM updated with latest path
set currentpath="!destdir!"

if lastpath==currentpath (echo "skip over making directory") else (MD "!destdir!")
       
copy "!sfile!" "!destdir!"

REM keeps track of last path for comparison
set lastpath="!destdir!"
)
pause

The output when run is:

Quote
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\ already exists.
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\ already exists.
        1 file(s) copied.
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\License\ already exists.
        1 file(s) copied.
A subdirectory or file c:\test2\CrystalDiskInfo7_1_1\License\ already exists.
        1 file(s) copied.
Press any key to continue . . .

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #9 on: January 17, 2018, 12:42:51 PM »
I guess I kind of assumed that if xcopy wrote a list of files to be copied that the corresponding backup directory (mirror to the original source directory) must exist for that to happen but as it happens I was looking at the batch I posted yesterday and noticed... no error checking around the copy operation. What if the source file does not exist? What if the destination directory does not exist? So I was already thinking of adding a few lines... now it checks that the source file actually exists, and if not, it does nothing at all with that listed file. If the listed source file does actually exist, it checks of the dest folder exists too, and if it doesn't, creates it. Then copies the file.

Code: [Select]
@echo off
setlocal enabledelayedexpansion
set source=c:\test3
set destin=c:\test2
For /f "delims=" %%F in (xfiles2copyx.txt) do (
set sfile=%%F
set "dfile=!sfile:%source%=%destin%!"
for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA
if exist "!sfile!" (
if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
copy "!sfile!" "!destdir!"
) else (
echo Source file "!sfile!" does not exist!
)
)
 

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #10 on: January 17, 2018, 12:50:01 PM »
Quote
set currentpath="!destdir!"

if lastpath==currentpath (echo "skip over making directory") else (MD "!destdir!")

This:

if lastpath==currentpath

will always evaluate to "false".

Batch variable names need % or ! symbols surrounding them so you would be using %lastpath% and %currentpath% or in a multiline parenthesis structure (FOR, IF, etc) you would be reading the changing values using exclamation points

if !lastpath!==!currentpath!

In fact, to guard against either value being empty (lastpath will be the first time you run, I think) people very often use guard characters such as quotes or dots or anything as long as they are same both sides of the == . IF will bork and crash the script hard if you try to compare something with nothing (a blank).

like this

IF "!lastpath!"=="!currentpath!"

Square brackets used to be popular also:

IF [!lastpath!]==[!currentpath!]

The idea is that "" or [] is a nonblank and won't break the script.

However, I haven't used the lastpath/currentpath thing in my little script.


« Last Edit: January 17, 2018, 01:17:11 PM by Salmon Trout »

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #11 on: January 17, 2018, 01:40:11 PM »
COOL!    And I forgot about the IF EXIST option that removes the need to compare paths from prior and current.  8)

Thanks for your help on this as well as explaining stuff so that I learned from it. Printing this for future use and immediate implementation with the other batch that creates folders with current date. This is going to save so much space not having each backup containing all the data but the first backup being all data and then consecutive backups only backing up what has changed or been added since last backup.  :)

Quote
Creating directory "c:\test2\CrystalDiskInfo7_1_1\"
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
Creating directory "c:\test2\CrystalDiskInfo7_1_1\License\"
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
Press any key to continue . . .

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #12 on: January 18, 2018, 07:14:22 PM »
Just updating the batches and sharing below to reflect the merger of what I learned in https://www.computerhope.com/forum/index.php/topic,164302.0.html with this backup thread.

Some additional changes ... I removed some stuff that I realized wasn't necessary such as setting path to C: and targeting root of C: by use of CD\. as well as made a correction to the first batch to write and overwrite by use of > instead of >> which would append files2copy.txt which would be bad. The single write and overwrite by use of > corrects for this. Otherwise the list will continue to build every time the batch is run. One other part that I forgot to add to this was the backup mirroring between the NAS which is simulated at c:\test3 and the local backup at c:\test1. Data backed up to c:\test2 contains differences between current data at c:\test3 and c:\test1 containing data from the last full backup and creates an archive trail of data that has changed or been added so that if needed an older file could be accessed at c:\test2 if say for example all of a sudden a corrupt file at c:\test3 overwrites c:\test1, I could dig through c:\test2 to find an older copy of it that is not corrupt and restore that to c:\test3 and c:\test1 . Lastly I added copy c:\testing\files2copy.txt "c:\test2\%datefix%\*.*" so that a list of contents is placed inside the by-date folders so that I can quickly look through list for what data is contained here without having to open every folder to see if it has what I am looking for.

Thanks to Salmontrout for the batch scripting help and others sharing info in the threads to learn from.  :)

Batch 1 of 3 - Create List of Files that Changed or have been added to files2copy.txt

Code: [Select]
@echo off
cls
REM Compare current data test3 to last backup at test1 and create list of files to write to c:\test2\%date% in batch3
xcopy "c:\test3\*.*" "c:\test1\*.*" /s/d/y/h/l>c:\testing\Files2Copy.txt
pause

Batch 2 of 3 - Remove last line from files2copy.txt which contains a file count to avoid crashing batch 3 of 3 and write this change to xfiles2copyx.txt

Code: [Select]
cls
@echo off & setlocal EnableDelayedExpansion
set row=
for /F "delims=" %%j in (c:\testing\files2copy.txt) do (
  if  defined row echo.!row!>> c:\testing\Xfiles2copyX.txt
  set row=%%j
)
pause

Batch 3 of 3 - Process xfiles2copyx.txt to copy all files to the archive backup location at c:\test2 after first confirming and creating if needed the tree/directory structure for the data to be copy populated. Lastly make a mirror copy of all data between main data source such as shared NAS or network storage location "simulated as C:\test3" and local backup location at c:\test1

Code: [Select]
@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=c:\test3
set destin=c:\test2\%datefix%
For /f "delims=" %%F in (c:\testing\xfiles2copyx.txt) do (
set sfile=%%F
set "dfile=!sfile:%source%=%destin%!"
for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA
if exist "!sfile!" (
if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
copy "!sfile!" "!destdir!"
) else (
echo Source file "!sfile!" does not exist!
)
)
REM Write Log of what files were backed up to dated folders at isolated partial archive backup location
copy c:\testing\files2copy.txt "c:\test2\%datefix%\*.*"

REM Mirror Complete Source to Complete Local Backup
xcopy c:\test3\*.* c:\test1\*.* /s/d/y/h
pause

Some info in case anyone wants to use this for their own use ( you will need to alter these to your pathing ):

Quote
C:\test1 = The Local Complete Backup Location
C:\test2 = The Local Backup Location that retains changed and new files as an archival / "shadow copy like" backup, and data is isolated into folders that are dated
C:\test3 = The Data Server, NAS, etc that everyone uses and where latest live data is accessed.
C:\testing = This is the location I chose to run the batches and keep isolated from the other 3 directories.


Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #13 on: January 19, 2018, 02:03:56 AM »
Batch 2 of 3 - Remove last line from files2copy.txt which contains a file count to avoid crashing batch 3 of 3 and write this change to xfiles2copyx.txt

Code: [Select]
cls
@echo off & setlocal EnableDelayedExpansion
set row=
for /F "delims=" %%j in (c:\testing\files2copy.txt) do (
  if  defined row echo.!row!>> c:\testing\Xfiles2copyX.txt
  set row=%%j
)
pause


I was a bit puzzled by this... at first I didn't quite see how it works, but I stared at it, and it's actually a clever way of skipping the last line of the file, by delaying the write by one line. Since you are using >> (append mode) to write to the output file, you might want to preface the FOR /F line with this...

if exist "c:\testing\Xfiles2copyX.txt" del "c:\testing\Xfiles2copyX.txt"

... to avoid the "growing file" problem.

I did previously think that if all you wanted to do was eliminate the final XX File(s) line you could use FIND to do something like this

type c:\testing\files2copy.txt | find /v "File(s)" > c:\testing\Xfiles2copyX.txt

or even, in your later batch (3) just do this*

For /f "delims=" %%F in (' type c:\testing\files2copy.txt ^| find /v "File^(s^)" ') do (

... which would eliminate batch (2) altogether, BUT... this would only be safe if you were sure you would NEVER also have a file in the list to be copied which had the string "File(s)" in the name, so the method you used is maybe safer because it only ever removes the last line.

* In the FOR line, you have to escape (using a caret character ^) certain control characters including the pipe symbol | and the open and close brackets in the FIND string, as these would otherwise break the script at the FOR line and give you an "xyz was unexpected at this time" message. The /V switch for FIND inverts the filter operation from "allow through lines containing this string" to "allow through lines NOT containing this string".

You could do the "if defined row" trickery in the loop in batch (3) anyhow,

[UPDATE} BUT... see below.
« Last Edit: January 19, 2018, 02:48:30 AM by Salmon Trout »

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #14 on: January 19, 2018, 02:39:40 AM »
I stepped back a bit from code-staring, and realized that surely this IF block in batch (3) would sanitize that last line anyhow, because there (hopefully) isn't a file anywhere in the current directory called "3 File(s)" or whatever, so the inside of the IF block wouldn't be executed. And maybe you don't really need the ELSE portion (do you need to see the message?)

if exist "!sfile!" (
...
)


A lesson there, to be sure!

[UPDATE again!] see next...

« Last Edit: January 19, 2018, 03:24:18 AM by Salmon Trout »

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #15 on: January 19, 2018, 03:23:40 AM »
Stepping back even more...

When you start in with batch scripting, there comes a "lightbulb moment" (it happened to me) when you realize that you don't need all those intermediate files.

Instead of this

REM Redirect lines of console output of some_command to a file
some_command > output.txt


then later

for /f "delims=" %%A in (output.txt) do (
    process lines of output.txt
)


You can just do this. You can get FOR /F to process the output of a program or command, by surrounding the command with single quotes:
The spaces between the single quotes and the command are not mandatory; I use them for clarity

for /f "delims=" %%A in ( ' some_command ' ) do (
    swallow and process lines of console output of some_command
)


So you can just process the lines of output of your original Xcopy command (the one which generates Files2Copy.txt in your batch (1) )

Thus you could eliminate batch (1) and batch (2) altogether. You can take care of the logging as well with echo >> lines in the loop.



« Last Edit: January 19, 2018, 04:10:01 AM by Salmon Trout »

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #16 on: January 19, 2018, 03:41:28 AM »
General tip: when FOR processes lines, it ignores and silently skips over any line that starts with a semicolon. (you can change this)
« Last Edit: January 19, 2018, 03:55:26 AM by Salmon Trout »

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #17 on: January 19, 2018, 07:19:10 AM »
How does this work?

@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=c:\test3
set destin=c:\test2\%datefix%

set logdir=c:\test2\%datefix%
if not exist "%logdir%" md "%logdir%"
if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

REM Original command to get Xcopy list
REM Compare current data test3 to last backup at test1 and create list of files to write to c:\test2\%date% in batch3
REM xcopy "c:\test3\*.*" "c:\test1\*.*" /s/d/y/h/l>c:\testing\Files2Copy.txt
REM pause

For /f "delims=" %%F in (' xcopy "c:\test3\*.*" "c:\test1\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      ) else (
      echo Source file "!sfile!" does not exist!
   )
)

REM Write Log of what files were backed up to dated folders at isolated partial archive backup location
REM copy c:\testing\files2copy.txt "c:\test2\%datefix%\*.*"

REM Mirror Complete Source to Complete Local Backup
xcopy c:\test3\*.* c:\test1\*.* /s/d/y/h
pause
 

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #18 on: January 19, 2018, 07:35:10 AM »
Oh cool you combined it all... I will check that out and test it  :)

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #19 on: January 19, 2018, 07:38:06 AM »
When I wrote "How does this work?" I meant, "How well does this work", as in "does this single batch below replace the three that you were using before, and duplicate their functionality?"

[UPDATE] I saw your edit.

« Last Edit: January 19, 2018, 08:13:31 AM by Salmon Trout »

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #20 on: January 19, 2018, 10:37:23 AM »
Quote
[UPDATE] I saw your edit.
  ;D

As I was half awake the light bulb finally came on after some extensive typing and I looked back and saw that you had combined it all into one. Then had a duh moment and quickly corrected via edit to my shorted response ever here at CH :-[ ;D

Took it initially as to "how does this work?" in which I didnt look at the differences in the script where you combined it all, then the lightbulb moment was I better check back at the script and see if thats my last 3rd batch or something you edited because the question is rather odd in the context that I ran with initially. Then I felt dumb that I didnt see that initially and ran with wrong context haha.  :-[

Time for more coffee for me  ::)  ;D

Now testing. It looks correct to me. Nothing stands out as incorrect.

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #21 on: January 19, 2018, 11:16:36 AM »
Works Perfect!  Thank You

Made a minor change to use same path as destin for logdir pointing logdir to %destin% . I was almost going to replace all the %logdir% with %destin% and remove [ set logdir = ] but decided to leave that be in case I ever wanted the path of the logging to go elsewhere [ set logdir=%destin% ] can easily be changed for an alternate path and its easier to read and follow with the naming convention you have vs reuse of %destin% in an area of the script that would make it look odd.  :)

Now to just remove the pause from tail end of it and add it to scheduled task on both computers.  8)

@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=z:
set destin=c:\archiveshadowcopy\%datefix%
set logdir=%destin%

if not exist "%logdir%" md "%logdir%"
if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

For /f "delims=" %%F in (' xcopy "z:\*.*" "c:\zcopy\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      ) else (
      echo Source file "!sfile!" does not exist!
   )
)

REM Mirror Complete Source to Complete Local Backup
xcopy Z:\*.* c:\zcopy\*.* /s/d/y/h
pause

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #22 on: January 19, 2018, 11:34:00 AM »
On the second computer I paid more attention to what was happening and saw that it chokes on the last line from files2copy.txt, but continues chugging along and completes, so I guess to remove that error message I could add the removal of last line from files2copy.txt by using the method I have in the second batch file written into this current batch to write to xfiles2copyx.txt , then run with that file which wouldnt have a line that is invalid to the copy instruction of file count.  :)

When I put together the initial batches for each group of instructions, I expected the batch to fail on hitting the file count. I never tested leaving it in there. So it was interesting that it didnt crash the batch it simply rejected the file count and moved on.  :)

Source file "37 File(s)" does not exist

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #23 on: January 19, 2018, 11:38:52 AM »
I  took your lead on where the log should go from this in your batch (3)

REM Write Log of what files were backed up to dated folders at isolated partial archive backup location
copy c:\testing\files2copy.txt "c:\test2\%datefix%\*.*"


... as c:\test2\%datefix% is the same as %destin%, I hard coded it but that meant it persisted after changes you made. Sorry for the confusion.

I have noticed something else... I made it so each time the new all-in-one batch is run, it blanks the log file and starts a new one, so that if you run it more than once on the same date, you will only see the last run's log results, so you might want to remove this line

if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

On the second computer I paid more attention to what was happening and saw that it chokes on the last line, but continues chugging along and complete, so I guess to remove that error message I could add the removal of last line and the xfiles2copyx.txt method written, then it runs with that file which wouldnt have a line that is invalid to the copy instruction.  :)

Source file "37 File(s)" does not exist

That's just the error message I coded in to the IF block surrounding the copy operation. It isn't really necessary to show that in the console. See below.

Anyhow, here's an update....


@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=z:
set destin=c:\archiveshadowcopy\%datefix%
set logdir=%destin%

if not exist "%logdir%" md "%logdir%"
if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

For /f "delims=" %%F in (' xcopy "z:\*.*" "c:\zcopy\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      REM ) else (
      REM echo Source file "!sfile!" does not exist!

   )
)

REM Mirror Complete Source to Complete Local Backup
xcopy Z:\*.* c:\zcopy\*.* /s/d/y/h
pause
 



Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #24 on: January 19, 2018, 11:42:26 AM »
When I put together the initial batches for each group of instructions, I expected the batch to fail on hitting the file count. I never tested leaving it in there. So it was interesting that it didnt crash the batch it simply rejected the file count and moved on.  :)

Source file "37 File(s)" does not exist

That's because if exist "!sfile!" checks if that line corresponds to a real, actual file.(As I mentioned above ;) (U need cofffee!!!)


DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #25 on: January 19, 2018, 11:44:23 AM »
 ;D Time for another coffee break. haha Thank You  :)

Salmon Trout



    Genius

    Thanked: 930
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #26 on: January 19, 2018, 11:49:08 AM »
Did you see this?

I have noticed something else... I made it so each time the new all-in-one batch is run, it blanks the log file and starts a new one, so that if you run it more than once on the same date, you will only see the last run's log results, so you might want to remove this line

if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #27 on: January 19, 2018, 03:14:59 PM »
Thanks.. I only run the backup once a day, but just in case it is run more than once I removed that so it will append to the log file.

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #28 on: January 19, 2018, 04:48:59 PM »
Decided to make it more user friendly for altering paths, so that the 4 paths just need to be set (Source, Full_Local_Backup, Archive, and Logging ) *In bold are changes made.

Might even set the log file name and make it dynamic below too. Undecided on that.  :-\


Quote
@echo off
cls
setlocal enabledelayedexpansion

REM Passes current Date in [ Fri 01-19-2018 ] format to datefix then datefix substitites / for - with :/=-
set datefix=%date%
set datefix=%datefix:/=-%

REM Source Drive that all data backed up comes from
set source=z:

REM Path for full local backup
set Full_Local_Backup=c:\zcopy

REM Destination for Archive Data
set destin=c:\archiveshadowcopy\%datefix%

REM Logging Destination specified after logdir=
set logdir=%destin%

if not exist "%logdir%" md "%logdir%"

For /f "delims=" %%F in (' xcopy "%source%\*.*" "%Full_Local_Backup%\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      REM ) else (
      REM echo Source file "!sfile!" does not exist!
   )
)

REM Mirror Complete Source to Complete Local Backup
xcopy "%source%\*.*" "%Full_Local_Backup%\*.*" /s/d/y/h
pause

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 610
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #29 on: January 22, 2018, 08:10:01 AM »
Hello salmon ... I dont need to see message about file count. ((( Update: Just realized when i pulled up this page and selected ALL it somehow showed the last item on page 1. So I thought this was a recent question as seen in pic. Time for more coffee  ;D  )))

Some updates to this that dont really affect what was created is that I added another location that isnt in a hidden directory and gave the users the ability to have access to a copy of what is archived in c:\archiveshadowcopy at a location of c:\shadowcopyarchive. No one but me knows about the hidden full local backup at c:\zcopy and the archive by date at c:\archiveshadowcopy . I figured I would trust the 12 users to have access to a copy of the archive data which they have been instructed not to use any files here as active documents to build on from until a copy of whatever file they need is moved away from C:\shadowcopyarchive to their workspace on the NAS. I was going to flag all data as read-only, but decided to hold off on that because the users arent skilled enough to flip it back to read/write permissions. So going to see how it goes with read/write enabled archive space.

If they foul up their archive by accidentally deleting something or altering something, I still have the hidden hands off location to get the original data from. The users are very excited that any data saved before 2am is recoverable in the future as for lots of users have lost data before.

The setup is just 2 computers sharing a low cost NAS for our union here. And because of my skills I was asked to make things better. Data integrity was the first importance to make corrections to. There were users not even using the NAS at times and data stored locally etc, it was a big mess. I set everyone up with their own user accounts and mapped space to their private folders and shared folders on the NAS, and set the default save locations to point to the NAS so that they have to intentionally tell it to save elsewhere.

So the backup solution I have in place is to use the extra space on the workstations in hidden folders to mirror the NAS data, then implement this batch that you helped greatly with to establish an archive that can be navigated by dated folders to have access to older versions  of files if need be. Because everyone who has access to the computers is trusted and everyone is aware that their data can be seen by others who have the computer access, having everyones data accessible in one location in the archive where normally they only see their W: drive as their space isnt an issue. Each user has a folder in the archive that is named for easy identification.

One last final piece to the project will be to set up a google drive and have that mirror to the Z: drive, but it would need to be selective in what if backed up to the cloud as for we have more than 15GB of data. This way if there was ever a fire etc, the most important recent data isnt wiped out.  :)

[attachment deleted by admin to conserve space]