Welcome guest. Before posting on our computer help forum, you must register. Click here it's easy and free.

Author Topic: Looking for a way to maintain path in this backup method  (Read 1014 times)

0 Members and 1 Guest are viewing this topic.

Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #15 on: January 19, 2018, 03:23:40 AM »
Stepping back even more...

When you start in with batch scripting, there comes a "lightbulb moment" (it happened to me) when you realize that you don't need all those intermediate files.

Instead of this

REM Redirect lines of console output of some_command to a file
some_command > output.txt


then later

for /f "delims=" %%A in (output.txt) do (
    process lines of output.txt
)


You can just do this. You can get FOR /F to process the output of a program or command, by surrounding the command with single quotes:
The spaces between the single quotes and the command are not mandatory; I use them for clarity

for /f "delims=" %%A in ( ' some_command ' ) do (
    swallow and process lines of console output of some_command
)


So you can just process the lines of output of your original Xcopy command (the one which generates Files2Copy.txt in your batch (1) )

Thus you could eliminate batch (1) and batch (2) altogether. You can take care of the logging as well with echo >> lines in the loop.



« Last Edit: January 19, 2018, 04:10:01 AM by Salmon Trout »

Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #16 on: January 19, 2018, 03:41:28 AM »
General tip: when FOR processes lines, it ignores and silently skips over any line that starts with a semicolon. (you can change this)
« Last Edit: January 19, 2018, 03:55:26 AM by Salmon Trout »

Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #17 on: January 19, 2018, 07:19:10 AM »
How does this work?

@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=c:\test3
set destin=c:\test2\%datefix%

set logdir=c:\test2\%datefix%
if not exist "%logdir%" md "%logdir%"
if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

REM Original command to get Xcopy list
REM Compare current data test3 to last backup at test1 and create list of files to write to c:\test2\%date% in batch3
REM xcopy "c:\test3\*.*" "c:\test1\*.*" /s/d/y/h/l>c:\testing\Files2Copy.txt
REM pause

For /f "delims=" %%F in (' xcopy "c:\test3\*.*" "c:\test1\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      ) else (
      echo Source file "!sfile!" does not exist!
   )
)

REM Write Log of what files were backed up to dated folders at isolated partial archive backup location
REM copy c:\testing\files2copy.txt "c:\test2\%datefix%\*.*"

REM Mirror Complete Source to Complete Local Backup
xcopy c:\test3\*.* c:\test1\*.* /s/d/y/h
pause
 

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #18 on: January 19, 2018, 07:35:10 AM »
Oh cool you combined it all... I will check that out and test it  :)

Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #19 on: January 19, 2018, 07:38:06 AM »
When I wrote "How does this work?" I meant, "How well does this work", as in "does this single batch below replace the three that you were using before, and duplicate their functionality?"

[UPDATE] I saw your edit.

« Last Edit: January 19, 2018, 08:13:31 AM by Salmon Trout »

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #20 on: January 19, 2018, 10:37:23 AM »
Quote
[UPDATE] I saw your edit.
  ;D

As I was half awake the light bulb finally came on after some extensive typing and I looked back and saw that you had combined it all into one. Then had a duh moment and quickly corrected via edit to my shorted response ever here at CH :-[ ;D

Took it initially as to "how does this work?" in which I didnt look at the differences in the script where you combined it all, then the lightbulb moment was I better check back at the script and see if thats my last 3rd batch or something you edited because the question is rather odd in the context that I ran with initially. Then I felt dumb that I didnt see that initially and ran with wrong context haha.  :-[

Time for more coffee for me  ::)  ;D

Now testing. It looks correct to me. Nothing stands out as incorrect.

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #21 on: January 19, 2018, 11:16:36 AM »
Works Perfect!  Thank You

Made a minor change to use same path as destin for logdir pointing logdir to %destin% . I was almost going to replace all the %logdir% with %destin% and remove [ set logdir = ] but decided to leave that be in case I ever wanted the path of the logging to go elsewhere [ set logdir=%destin% ] can easily be changed for an alternate path and its easier to read and follow with the naming convention you have vs reuse of %destin% in an area of the script that would make it look odd.  :)

Now to just remove the pause from tail end of it and add it to scheduled task on both computers.  8)

@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=z:
set destin=c:\archiveshadowcopy\%datefix%
set logdir=%destin%

if not exist "%logdir%" md "%logdir%"
if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

For /f "delims=" %%F in (' xcopy "z:\*.*" "c:\zcopy\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      ) else (
      echo Source file "!sfile!" does not exist!
   )
)

REM Mirror Complete Source to Complete Local Backup
xcopy Z:\*.* c:\zcopy\*.* /s/d/y/h
pause

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #22 on: January 19, 2018, 11:34:00 AM »
On the second computer I paid more attention to what was happening and saw that it chokes on the last line from files2copy.txt, but continues chugging along and completes, so I guess to remove that error message I could add the removal of last line from files2copy.txt by using the method I have in the second batch file written into this current batch to write to xfiles2copyx.txt , then run with that file which wouldnt have a line that is invalid to the copy instruction of file count.  :)

When I put together the initial batches for each group of instructions, I expected the batch to fail on hitting the file count. I never tested leaving it in there. So it was interesting that it didnt crash the batch it simply rejected the file count and moved on.  :)

Source file "37 File(s)" does not exist

Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #23 on: January 19, 2018, 11:38:52 AM »
I  took your lead on where the log should go from this in your batch (3)

REM Write Log of what files were backed up to dated folders at isolated partial archive backup location
copy c:\testing\files2copy.txt "c:\test2\%datefix%\*.*"


... as c:\test2\%datefix% is the same as %destin%, I hard coded it but that meant it persisted after changes you made. Sorry for the confusion.

I have noticed something else... I made it so each time the new all-in-one batch is run, it blanks the log file and starts a new one, so that if you run it more than once on the same date, you will only see the last run's log results, so you might want to remove this line

if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

On the second computer I paid more attention to what was happening and saw that it chokes on the last line, but continues chugging along and complete, so I guess to remove that error message I could add the removal of last line and the xfiles2copyx.txt method written, then it runs with that file which wouldnt have a line that is invalid to the copy instruction.  :)

Source file "37 File(s)" does not exist

That's just the error message I coded in to the IF block surrounding the copy operation. It isn't really necessary to show that in the console. See below.

Anyhow, here's an update....


@echo off
cls
setlocal enabledelayedexpansion

set datefix=%date%
set datefix=%datefix:/=-%
set source=z:
set destin=c:\archiveshadowcopy\%datefix%
set logdir=%destin%

if not exist "%logdir%" md "%logdir%"
if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

For /f "delims=" %%F in (' xcopy "z:\*.*" "c:\zcopy\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      REM ) else (
      REM echo Source file "!sfile!" does not exist!

   )
)

REM Mirror Complete Source to Complete Local Backup
xcopy Z:\*.* c:\zcopy\*.* /s/d/y/h
pause
 



Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #24 on: January 19, 2018, 11:42:26 AM »
When I put together the initial batches for each group of instructions, I expected the batch to fail on hitting the file count. I never tested leaving it in there. So it was interesting that it didnt crash the batch it simply rejected the file count and moved on.  :)

Source file "37 File(s)" does not exist

That's because if exist "!sfile!" checks if that line corresponds to a real, actual file.(As I mentioned above ;) (U need cofffee!!!)


DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #25 on: January 19, 2018, 11:44:23 AM »
 ;D Time for another coffee break. haha Thank You  :)

Salmon Trout



    Genius

    Thanked: 915
    • Yes
  • Computer: Specs
  • Experience: Experienced
  • OS: Other
Re: Looking for a way to maintain path in this backup method
« Reply #26 on: January 19, 2018, 11:49:08 AM »
Did you see this?

I have noticed something else... I made it so each time the new all-in-one batch is run, it blanks the log file and starts a new one, so that if you run it more than once on the same date, you will only see the last run's log results, so you might want to remove this line

if exist "%logdir%\files2copy.txt" del "%logdir%\files2copy.txt"

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #27 on: January 19, 2018, 03:14:59 PM »
Thanks.. I only run the backup once a day, but just in case it is run more than once I removed that so it will append to the log file.

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #28 on: January 19, 2018, 04:48:59 PM »
Decided to make it more user friendly for altering paths, so that the 4 paths just need to be set (Source, Full_Local_Backup, Archive, and Logging ) *In bold are changes made.

Might even set the log file name and make it dynamic below too. Undecided on that.  :-\


Quote
@echo off
cls
setlocal enabledelayedexpansion

REM Passes current Date in [ Fri 01-19-2018 ] format to datefix then datefix substitites / for - with :/=-
set datefix=%date%
set datefix=%datefix:/=-%

REM Source Drive that all data backed up comes from
set source=z:

REM Path for full local backup
set Full_Local_Backup=c:\zcopy

REM Destination for Archive Data
set destin=c:\archiveshadowcopy\%datefix%

REM Logging Destination specified after logdir=
set logdir=%destin%

if not exist "%logdir%" md "%logdir%"

For /f "delims=" %%F in (' xcopy "%source%\*.*" "%Full_Local_Backup%\*.*" /s/d/y/h/l ' ) do (
   set sfile=%%F

   REM Do the logging
   echo !sfile! >> "%logdir%\files2copy.txt"

   set "dfile=!sfile:%source%=%destin%!"
   for /f "delims=" %%A in ("!dfile!") do set destdir=%%~dpA

   if exist "!sfile!" (
      if not exist "!destdir!" echo Creating directory "!destdir!" & md "!destdir!"
      copy "!sfile!" "!destdir!"
      REM ) else (
      REM echo Source file "!sfile!" does not exist!
   )
)

REM Mirror Complete Source to Complete Local Backup
xcopy "%source%\*.*" "%Full_Local_Backup%\*.*" /s/d/y/h
pause

DaveLembke

    Topic Starter


    Sage
  • Inventor of the Magna-Broom 3000 =)
  • Thanked: 590
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 7
Re: Looking for a way to maintain path in this backup method
« Reply #29 on: January 22, 2018, 08:10:01 AM »
Hello salmon ... I dont need to see message about file count. ((( Update: Just realized when i pulled up this page and selected ALL it somehow showed the last item on page 1. So I thought this was a recent question as seen in pic. Time for more coffee  ;D  )))

Some updates to this that dont really affect what was created is that I added another location that isnt in a hidden directory and gave the users the ability to have access to a copy of what is archived in c:\archiveshadowcopy at a location of c:\shadowcopyarchive. No one but me knows about the hidden full local backup at c:\zcopy and the archive by date at c:\archiveshadowcopy . I figured I would trust the 12 users to have access to a copy of the archive data which they have been instructed not to use any files here as active documents to build on from until a copy of whatever file they need is moved away from C:\shadowcopyarchive to their workspace on the NAS. I was going to flag all data as read-only, but decided to hold off on that because the users arent skilled enough to flip it back to read/write permissions. So going to see how it goes with read/write enabled archive space.

If they foul up their archive by accidentally deleting something or altering something, I still have the hidden hands off location to get the original data from. The users are very excited that any data saved before 2am is recoverable in the future as for lots of users have lost data before.

The setup is just 2 computers sharing a low cost NAS for our union here. And because of my skills I was asked to make things better. Data integrity was the first importance to make corrections to. There were users not even using the NAS at times and data stored locally etc, it was a big mess. I set everyone up with their own user accounts and mapped space to their private folders and shared folders on the NAS, and set the default save locations to point to the NAS so that they have to intentionally tell it to save elsewhere.

So the backup solution I have in place is to use the extra space on the workstations in hidden folders to mirror the NAS data, then implement this batch that you helped greatly with to establish an archive that can be navigated by dated folders to have access to older versions  of files if need be. Because everyone who has access to the computers is trusted and everyone is aware that their data can be seen by others who have the computer access, having everyones data accessible in one location in the archive where normally they only see their W: drive as their space isnt an issue. Each user has a folder in the archive that is named for easy identification.

One last final piece to the project will be to set up a google drive and have that mirror to the Z: drive, but it would need to be selective in what if backed up to the cloud as for we have more than 15GB of data. This way if there was ever a fire etc, the most important recent data isnt wiped out.  :)