Welcome guest. Before posting on our computer help forum, you must register. Click here it's easy and free.

Author Topic: Batch: xcopy /D repeatedly copies old files  (Read 21027 times)

0 Members and 1 Guest are viewing this topic.

donOpenHydro

    Topic Starter


    Starter

    • Experience: Experienced
    • OS: Windows XP
    Batch: xcopy /D repeatedly copies old files
    « on: July 28, 2011, 07:42:43 AM »
    I have a script that performs a delayed synchronisation of data files from an industrial pc running Windows XP to a NAS box (QNAP TS-110). I have been testing this script on my computer (also running XP [-pro 2002 S.P.3]), with the NAS mounted as a network drive (192.168.10.122 -> Z:\).

    I have run the script sequentially to make sure it behaves as expected. My "test" folder contains a few sub directories and a number of files of different types. However, from a reading of my "Log.txt" file, while it copies most files from my "test" folder only once, some files it copies repeatedly. On a colleague's computer (also XP pro) it consistently copied a number of .jpg files, on mine it consistently copies one .txt file.

    As far as I can tell, there does not appear to be any pattern to this behaviour. We have tried tweaking the clock on the NAS to see if this makes any difference, but to no avail. We have also tried various combinations of the switches to see if some interaction between them is causing the unexpected behaviour, but this has not worked either.

    We have considered installing a version of RoboCopy and using that, but I am kinda hoping that this will not be necessary, and that I have just missed something simple.

    My code is:
    Code: [Select]
    @echo off
    title NAScopy02

    REM note: Timeout.exe must be in the same directory as this file to work correctly!

    REM (1)
    Timeout 5
    set today=%date%

    REM (2)
    ping 192.168.10.122 -n 1 | find /i "bytes=" || goto Fail01
    set now=%time%
    echo %today%, %now%, 192.168.0.122 >> "Log.txt
    xcopy test Z:\test01 /D /E /I /H /Y >> "Log.txt
    :Fail01

    REM (3)
    ping 192.168.10.1 -n 1 | find /i "bytes=" || goto Fail02
    set now=%time%
    echo %today%, %now%, 192.168.0.12 >> "Log.txt
    xcopy test test01 /DEIHY >> "Log.txt
    :Fail02

    REM (4)
    ping 192.168.10.1 -n 1 | find /i "bytes=" || goto Fail03
    set now=%time%
    echo %today%, %now%, 192.168.0.13 >> "Log.txt
    xcopy test test01 /DEIHY >> "Log.txt
    :Fail03

    REM : use loops!
    @echo on


    My Log.txt reads:

    27/07/2011, 17:51:41.65, 192.168.0.11
    test\2011 06 22 0000 (Float).DAT
    test\test_files\DSC_2074.JPG
    test\test_files\DSC_2077.JPG
    test\test_files\DSC_2081.JPG
    test\test_files\DSC_2083.JPG
    test\test_files\DSC_2084.JPG
    test\test_files\DSC_2096.JPG
    test\test_files\DSC_2099.JPG
    test\test_files\subfolder1\Subfolder1dash1\yo_mama.txt
    test\test_files\test files 2\DSC_2074.JPG
    test\test_files\test files 2\DSC_2077.JPG
    test\test_files\test files 2\DSC_2081.JPG
    test\test_files\test files 2\DSC_2083.JPG
    test\test_files\test files 2\DSC_2084.JPG
    test\test_files\test files 2\DSC_2096.JPG
    test\test_files\test files 2\DSC_2099.JPG
    test\test_files\test_folder\test.txt
    17 File(s) copied
    27/07/2011, 17:51:59.72, 192.168.0.12
    0 File(s) copied
    27/07/2011, 17:52:00.06, 192.168.0.13
    0 File(s) copied
    27/07/2011, 17:53:16.19, 192.168.0.11
    test\2011 06 22 0000 (Float).DAT
    1 File(s) copied
    27/07/2011, 17:53:16.69, 192.168.0.12
    0 File(s) copied
    27/07/2011, 17:53:17.01, 192.168.0.13
    0 File(s) copied
    27/07/2011, 17:53:27.06, 192.168.0.11
    test\2011 06 22 0000 (Float).DAT
    1 File(s) copied
    27/07/2011, 17:53:27.51, 192.168.0.12
    0 File(s) copied
    27/07/2011, 17:53:27.83, 192.168.0.13
    0 File(s) copied


    Any help with this (now frustrating) problem would be Appreciated.

    Dan




    Sidewinder



      Guru

      Thanked: 139
    • Experience: Familiar
    • OS: Windows 10
    Re: Batch: xcopy /D repeatedly copies old files
    « Reply #1 on: July 28, 2011, 08:55:58 AM »
    This comes from the XCOPY help:

    /D:m-d-y   Copies files changed on or after the specified date.
                     If no date is given, copies only those files whose
                     source time is newer than the destination time.


    Perhaps things have changed, but is this even legal: /DEIHY Thought each switch had to be preceded by a forward slash so the interpreter could parse them correctly.

    Don't see any glaring errors, so fix up the switches (the first XCOPY is correct) and see how it goes. 8)

    The true sign of intelligence is not knowledge but imagination.

    -- Albert Einstein

    donOpenHydro

      Topic Starter


      Starter

      • Experience: Experienced
      • OS: Windows XP
      Re: Batch: xcopy /D repeatedly copies old files
      « Reply #2 on: July 28, 2011, 10:08:04 AM »
      Your right of course, but the first xcopy is the only one that's used at the moment, the others don't get used coz the IP's to be tested before they are executed are dummies, and so the script skips to the Fail point. The Log.txt output is consistant with this, and shows that it is the first xcopy that is giving the problem in any case.

      Sidewinder



        Guru

        Thanked: 139
      • Experience: Familiar
      • OS: Windows 10
      Re: Batch: xcopy /D repeatedly copies old files
      « Reply #3 on: July 28, 2011, 10:32:23 AM »
      Your right of course, but the first xcopy is the only one that's used at the moment, the others don't get used coz the IP's to be tested before they are executed are dummies, and so the script skips to the Fail point. The Log.txt output is consistant with this, and shows that it is the first xcopy that is giving the problem in any case.

      Actually, using bytes as the search argument for the find command is a poor choice.

      Unsuccessful Ping
      Code: [Select]
      Pinging 192.168.2.4 with 32 bytes of data:

      Request timed out.
      Request timed out.
      Request timed out.
      Request timed out.

      Ping statistics for 192.168.2.4:
          Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

      Successful Ping
      Code: [Select]
      Pinging 192.168.2.3 with 32 bytes of data:

      Reply from 192.168.2.3: bytes=32 time<1ms TTL=128
      Reply from 192.168.2.3: bytes=32 time<1ms TTL=128
      Reply from 192.168.2.3: bytes=32 time<1ms TTL=128
      Reply from 192.168.2.3: bytes=32 time<1ms TTL=128

      Ping statistics for 192.168.2.3:
          Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
      Approximate round trip times in milli-seconds:
          Minimum = 0ms, Maximum = 0ms, Average = 0ms

      Bytes is present on the first line issued by ping no matter what the state of the connection. Try using a better argument so you can differentiate the connection status. (TTL will work).

       8)

      The true sign of intelligence is not knowledge but imagination.

      -- Albert Einstein

      donOpenHydro

        Topic Starter


        Starter

        • Experience: Experienced
        • OS: Windows XP
        Re: Batch: xcopy /D repeatedly copies old files
        « Reply #4 on: July 28, 2011, 12:00:01 PM »
        Well, I didn't really know what I was doing with that! Anyway I fixed the switches, and changed the find argument to "TTL", and I still had the same problem. I trimmed the code down to one IP sequence for testing, and it still behaves the same way!

        Code: [Select]
        @echo off
        title NAScopy03

        REM note: Timeout.exe must be in the same directory as this file to work correctly!

        REM (1)
        Timeout 5
        set today=%date%

        REM (2)
        ping 192.168.10.122 -n 1 | find /i "TTL=" || goto Fail01
        set now=%time%
        echo %today%, %now%, 192.168.0.122 >> "Log03.txt
        xcopy test Z:\test01 /D /E /I /H /Y >> "Log03.txt
        :Fail01

        @echo on


        Log03.txt:

        28/07/2011, 18:50:26.18, 192.168.0.122
        test\2011 06 22 0000 (Float).DAT
        test\test_files\DSC_2074.JPG
        test\test_files\DSC_2077.JPG
        test\test_files\DSC_2081.JPG
        test\test_files\DSC_2083.JPG
        test\test_files\DSC_2084.JPG
        test\test_files\DSC_2096.JPG
        test\test_files\DSC_2099.JPG
        test\test_files\subfolder1\Subfolder1dash1\yo_mama.txt
        test\test_files\test files 2\DSC_2074.JPG
        test\test_files\test files 2\DSC_2077.JPG
        test\test_files\test files 2\DSC_2081.JPG
        test\test_files\test files 2\DSC_2083.JPG
        test\test_files\test files 2\DSC_2084.JPG
        test\test_files\test files 2\DSC_2096.JPG
        test\test_files\test files 2\DSC_2099.JPG
        test\test_files\test_folder\test.txt
        17 File(s) copied
        28/07/2011, 18:50:48.82, 192.168.0.122
        test\2011 06 22 0000 (Float).DAT
        1 File(s) copied
        28/07/2011, 18:50:52.46, 192.168.0.122
        test\2011 06 22 0000 (Float).DAT
        1 File(s) copied
        28/07/2011, 18:50:55.51, 192.168.0.122
        test\2011 06 22 0000 (Float).DAT
        1 File(s) copied

        Sidewinder



          Guru

          Thanked: 139
        • Experience: Familiar
        • OS: Windows 10
        Re: Batch: xcopy /D repeatedly copies old files
        « Reply #5 on: July 28, 2011, 12:39:11 PM »
        Still have a few ideas:

        Add the /L switch to the XCOPY and run it from the command line. This will produce a list of files on a what-if basis but not actually copy any files.

        Run a dir against the souce directory on the local machine and the destination directory on the NAS. Be sure to use both the /s and /tc switches on both.

        This should give you enough data to manually check which files should be copied and which shouldn't. Remember the /d switch on the XCOPY without a date only selects files with a source time newer than the destination time.

        Good luck. 8)
        The true sign of intelligence is not knowledge but imagination.

        -- Albert Einstein

        donOpenHydro

          Topic Starter


          Starter

          • Experience: Experienced
          • OS: Windows XP
          Re: Batch: xcopy /D repeatedly copies old files
          « Reply #6 on: July 28, 2011, 02:53:34 PM »
          dir /tc shows that the source file is older than the directory file
          Incidentally /ta & /tw do no better at highlighting any problems.
          In fact, testing seems to indicate that xcopy seems to respond to changes reported by dir /w.
          xcopy /L just shows up the same poor behaviour in the log file.
          If I modify the destination file, the script behaves as it should.
          The only ray of light is that if I leave a gap of 5 mins between sequential runs of the script, dir on the destination shows no changes in the dates of the files. Is it possible that the output from xcopy is buggy, and that there is no actual copying going on? I think th only way to show that would be to monitor network use on the NAS box.
          Maybe its time to call it a day and use a different tool.

          Sidewinder



            Guru

            Thanked: 139
          • Experience: Familiar
          • OS: Windows 10
          Re: Batch: xcopy /D repeatedly copies old files
          « Reply #7 on: July 28, 2011, 03:37:31 PM »
          If I understand this correctly, you want the NAS files to sync to the local machine on a regular basis.

          If you were to create an exact copy of the files and structure of the files on the NAS, you would then have a baseline. As files change on the local machine, the modification date will get updated making the source time newer than the destination time. The file should then be eligible for inclusion the next time XCOPY is run with the /d switch.

          Another method might be to key off the archive bit. When a file is changed, the archive bit is set on, and XCOPY can recognize this with the /m switch.  After the copy the /m switch will flip the archive bit off saving you a separate attribute operation.

          Hope this gives you some alternative ideas.  8)
          The true sign of intelligence is not knowledge but imagination.

          -- Albert Einstein

          hopingForAnswers



            Rookie

            • Experience: Beginner
            • OS: Unknown
            Re: Batch: xcopy /D repeatedly copies old files
            « Reply #8 on: August 01, 2011, 11:19:34 AM »
            I see nothing obvious - but I don't always trust incremental backups.  At least once a month, I would do a complete backup of all of the data,  whatever the file date or whether the archive bit is set on-or-off.   That way you never loose more than a months work at a time. 

            I would also check the results - carefully at least the first few times. 
            I would use dir *.* > afile.prn  for both directories then import into excel and check that both exist, File Size, File Dates. 

            Regarding the batchfile,  can you add a couple of lines to it  to copy  1) starting files (both From and To)  and the resulting files  (both From and To)   and see it anything popouts.  And save about a weeks work of these files for analysis.

            TheShadow



              Hopeful

            • Retiree in Florida
            • Thanked: 13
              • Yes
              • The Doctor
            • Certifications: List
            • Computer: Specs
            • Experience: Guru
            • OS: Windows XP
            Re: Batch: xcopy /D repeatedly copies old files
            « Reply #9 on: August 09, 2011, 02:01:32 PM »
            For years, I've been setting up backup solutions for myself and certain customers.

            The Idea:  Is to make sure that every file created, saved or changed on the C: drive is backed up to the second hard drive as an active backup.  So I wrote the most simple batch file I could, using XCOPY to make sure that no data file went un-backed up.

            The batch file can be run manually from a shortcut, or from a shortcut in the Startup folder or from a simple batch file that runs MS-Word (for instance) and then runs a backup to save the new file to the backup drive.
            The task scheduler can be used to run the batch file too, however that's never been my favorite way to do anything.

            My batch file to run WordPerfect and then backup the result, goes like this:

            @Echo off
            cls
            call "C:\Corel\Suite8\Programs\WPWIN8.EXE"
            "C:\Documents and Settings\Alexi\desktop\Update\XCOPYDocs-Mail to D.bat"

            the last line, just runs an existing batch file to backup all my documents, when WordPerfect finishes.

            The backup batch file run above actually looks like this:

            @Echo off
            cls
            xcopy "C:\Documents and Settings\Alexi\My Documents\*.*" "D:\My Documents\" /s /y /H /R /D
            Rem:  Backup my WordPerfect files
            xcopy "C:\MyFiles\*.*" "D:\MyFiles\" /s /y /H /R /D
            Rem:  Backup all my saved eMails
            xcopy "C:\Documents and Settings\Alexi\Local Settings\Application Data\Identities\{DA2A9331-88CA-4BF6-9B3D-22290B4CD50D}\Microsoft\Outlook Express\*.dbx"  "D:\MyEmailFiles-Backup\" /s /y /H /R /D
            pause



            As Y'all can see, I don't do anything fancy, but I try to keep it as simple as possible.  The first time the batch file runs, ALL files are backed up (copied) from the Source Directories, but on subsequent runs, only files that have changed or have just been created or saved, will be backed up.  A daily backup only takes a few short seconds.

            Cheers Mates!
             8)
            Experience is truly the best teacher.
            Backup! Backup! Backup!  Ghost Rocks!

            Craggy



              Newbie

              • Experience: Beginner
              • OS: Windows 7
              Re: Batch: xcopy /D repeatedly copies old files
              « Reply #10 on: January 06, 2014, 04:07:02 PM »
              Hi donOpenHydro,

              Did this ever get resolved?  I have same issue.  I use Xcopy to copy files locally with no issues.. but up to a NAS I have same copy problem as you .. the same subset of files repeatedly get copied.  I see this issue with xcopy on Windows XP, Vista and Windows 7 also tried various versions of robocopy.

              There is definitely something odd happening when copying up to NAS (linux based).  I'm quite familiar with Networking, OS's, batch files etc .. but this is odd.  Like you listing directly with dir /tc/ta /tw do not show any difference between NAS and local directory listing .. but still continue to recopy certain (not all) files.

              Any shining light would be appreciated  :)

              foxidrive



                Specialist
              • Thanked: 268
              • Experience: Experienced
              • OS: Windows 8
              Re: Batch: xcopy /D repeatedly copies old files
              « Reply #11 on: January 06, 2014, 09:07:31 PM »
              Maybe it has to do with the timestamp granularity as Linux boxes do things differently.

              Test it with a Windows box on the LAN and if it doesn't recopy that file then you know where the problem is.

              mrawler



                Newbie

                • Experience: Experienced
                • OS: Unknown
                Re: Batch: xcopy /D repeatedly copies old files
                « Reply #12 on: January 27, 2018, 08:38:52 PM »
                I decided to register just to put a few updates on this as it's still highly indexed by a google search.

                I'm having exactly the same problem with a much simpler batch script that's going windows -> windows on an old win7 desktop. The backup script seems to randomly choose how many files to copy over again or not. I've done similar troubleshooting steps to the above, including manually creating a new file (sometimes it gets copied, sometimes it doesn't). My only guess is that there's a process (AV? Windows defender/etc) that is doing the linux equiv of 'touching' the files which throws the timestamp off.

                For my purposes it's not a big deal - my script runs daily after hours, so the only 'net loss' is a little more disk activity every night. Sorry for the gravedig, and I've been reading the forum for years!