Subtract images in batch process(leap year and regular year)

[_unknown_]

Just to clarify - for each file you want to determine the correct month in that year, using the day of year.

A high level computer language will automatically handle leap years and non leap years - and there are batch script that can also handle this.

All I know is that if the day from the file name(C2011061.A1_ABC.ABCD.tif) is 061, wherein the day 061 falls on the month of march it will be processed/subtracted also to the sample_file_march.tif. I really don't know can you help me?

[foxidrive]

I think you are right.

Spot on Sqaushman!  Ritchie Lawrence has done some fantastic work for scripters - a very talented fellow.

[foxidrive]

This is untested - it displays the command to the screen only.

It is right then you can thank Squashman for supplying the meat of the batch file on the 3rd post in this thread.

Code: [Select]

@echo off
setlocal enabledelayedexpansion

for /f "delims=" %%a in ('dir /b /on c*.tif ') do (
set "year=%%~na"
set "daynum=!year:~5,3!"
set "year=!year:~1,4!"

call :OrdinalToDate !year! !daynum! yy month dd
echo %%a matches to !yy!-!month!-!dd!

if !Month!==01 set Month=JAN
if !Month!==02 set Month=FEB
if !Month!==03 set Month=MAR
if !Month!==04 set Month=APR
if !Month!==05 set Month=MAY
if !Month!==06 set Month=JUN
if !Month!==07 set Month=JUL
if !Month!==08 set Month=AUG
if !Month!==09 set Month=SEP
if !Month!==10 set Month=OCT
if !Month!==11 set Month=NOV
if !Month!==12 set Month=DEC

echo gdal_calculate --outfile=D:\path\to\file\difference.tif --calc="((image1-image2)/(image1+image2))" --image2=D:\path\to\subtrahend\sample_file_!Month!.tif image1=D:\path\to\minuend\%%a --extent=INTERSECT

pause
)
echo done
pause
goto :EOF


@echo off&setlocal
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%
goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


[Squashman]

I suppose my month number to month abbreviation code wouldn't work because of the way the expansion works. But other than that, all the code that he needed was given to him here and on DosTips. From there it seemed pretty clear what needed to be done.

[Squashman]

Foxidrive, I think you missed removing some of my code.

Right above the Ordinal date label.

Code: [Select]

@echo off&setlocal
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%
goto :EOFDoesn't hurt anything as it will never be executed.

[_unknown_]

This is untested - it displays the command to the screen only.

It is right then you can thank Squashman for supplying the meat of the batch file on the 3rd post in this thread.

Code: [Select]

@echo off
setlocal enabledelayedexpansion

for /f "delims=" %%a in ('dir /b /on c*.tif ') do (
set "year=%%~na"
set "daynum=!year:~5,3!"
set "year=!year:~1,4!"

call :OrdinalToDate !year! !daynum! yy month dd
echo %%a matches to !yy!-!month!-!dd!

if !Month!==01 set Month=JAN
if !Month!==02 set Month=FEB
if !Month!==03 set Month=MAR
if !Month!==04 set Month=APR
if !Month!==05 set Month=MAY
if !Month!==06 set Month=JUN
if !Month!==07 set Month=JUL
if !Month!==08 set Month=AUG
if !Month!==09 set Month=SEP
if !Month!==10 set Month=OCT
if !Month!==11 set Month=NOV
if !Month!==12 set Month=DEC

echo gdal_calculate --outfile=D:\path\to\file\difference.tif --calc="((image1-image2)/(image1+image2))" --image2=D:\path\to\subtrahend\sample_file_!Month!.tif image1=D:\path\to\minuend\%%a --extent=INTERSECT

pause
)
echo done
pause
goto :EOF


@echo off&setlocal
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%
goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


Tried to run and it just says file not found.
By the way, I forgot to modify the gdal_calculate command since the directory has numerous files:

Code: [Select]

echo gdal_calculate --outfile=%out_path%\C!yearDay!.A1_ABC.ABCD.tif --calc="((image1-image2))" --image2=%sample_path%\sample_file_!Month!.tif image1=%in_path%\C!yearDay!.A1_ABC.ABCD.tif --extent=INTERSECT
From your script, where should the out_path, in_path, sample_path and processed_files_path directory be inserted?

[Squashman]

From your script, where should the out_path, in_path, sample_path and processed_files_path directory be inserted?

Isn't that kind of self explanatory from the code that we gave you? It shows in three different places where you need to substitute the path to where your files are and will be going.  It is up to you to put the correct paths into your GDAL command.

Code: [Select]

echo gdal_calculate --outfile=D:\path\to\file\difference.tif --calc="((image1-image2)/(image1+image2))" --image2=D:\path\to\subtrahend\sample_file_!Month!.tif image1=D:\path\to\minuend\%%a --extent=INTERSECT
At some point you gotta start being clear on your instructions.  None of us have ever used your program so you have to define how it needs to work and define where you where the input and the output is.

[_unknown_]

Isn't that kind of self explanatory from the code that we gave you? It shows in three different places where you need to substitute the path to where your files are and will be going.  It is up to you to put the correct paths into your GDAL command.

Code: [Select]

echo gdal_calculate --outfile=D:\path\to\file\difference.tif --calc="((image1-image2)/(image1+image2))" --image2=D:\path\to\subtrahend\sample_file_!Month!.tif image1=D:\path\to\minuend\%%a --extent=INTERSECT

I am pointing out the file names template. Because in this outfile, the output file name has already been set "difference.tif". What if the file names used from the input path will also be the file name of the output? Like the image1 a %%a was just inserted to the last "D:\path\to\minuend\%%a". The file name template should be (C2011061.A1_ABC.ABCD.tif) that is why I modified it into C!yearDay!.A1_ABC.ABCD.tif. Since I have numerous files from the input directory and they have different !yearDay!. The image1 and the outfile will have the same file name template they will just differ with their paths.  I hope you're getting my point. And another thing, how will I set the processed files path and move only the processed files to the said directory?

At some point you gotta start being clear on your instructions.  None of us have ever used your program so you have to define how it needs to work and define where you where the input and the output is.

I'm so sorry about that. My fault.

gdal_calculate Perform simple tiled raster calculations (AKA "map algebra")from the command line.
--calc     : calculation in numpy syntax, rasters specified as using any legal python variable name syntax, band numbers are specified using square brackets (zero based indexing)
--outfile  : output filepath

[patio]

I predict 6 pages for this Topic...

[foxidrive]

Foxidrive, I think you missed removing some of my code.

Doesn't hurt anything as it will never be executed.

Yep, thanks, I wasn't paying close attention.  This thread does that to you.

I predict 6 pages for this Topic...

I'll bet $2 on 8 pages.

[Squashman]

I am finding it hard to fathom that you still do not understand the code enough to make modifications to it. We already did the heavy lifting for you. I hate to quote religious sayings but God helps those who help themselves.
I really dont want to see another technical definition of what you are trying to do because it doesnt help at all. If you are smart enough to understand what your software does I think with a little learning and research of your own you should be able to modify the code.

[_unknown_]

This is untested - it displays the command to the screen only.

It is right then you can thank Squashman for supplying the meat of the batch file on the 3rd post in this thread.

Code: [Select]

@echo off
setlocal enabledelayedexpansion

for /f "delims=" %%a in ('dir /b /on c*.tif ') do (
set "year=%%~na"
set "daynum=!year:~5,3!"
set "year=!year:~1,4!"

call :OrdinalToDate !year! !daynum! yy month dd
echo %%a matches to !yy!-!month!-!dd!

if !Month!==01 set Month=JAN
if !Month!==02 set Month=FEB
if !Month!==03 set Month=MAR
if !Month!==04 set Month=APR
if !Month!==05 set Month=MAY
if !Month!==06 set Month=JUN
if !Month!==07 set Month=JUL
if !Month!==08 set Month=AUG
if !Month!==09 set Month=SEP
if !Month!==10 set Month=OCT
if !Month!==11 set Month=NOV
if !Month!==12 set Month=DEC

echo gdal_calculate --outfile=D:\path\to\file\difference.tif --calc="((image1-image2)/(image1+image2))" --image2=D:\path\to\subtrahend\sample_file_!Month!.tif image1=D:\path\to\minuend\%%a --extent=INTERSECT

pause
)
echo done
pause
goto :EOF


@echo off&setlocal
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%
goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


Hi sorry for bothering you again. I tried to modified the script and run but it only calculate one image the rest of the images wasn't processed and I can't seem to figure out what is the problem. But I know I'm getting close. Another thing how will I moved only the processed files to the processed_path? Thank you!

This is the error:

Code: [Select]

C:\Users\Name\Desktop> gdalcalc.bat
A2014336.L2_LAC.SeAHABS.tif matches to 2014-12-02
Running calculation
0...10..Saving output
.20...30...40...50...60...70...80...90...100 - done.

C:\Users\Name\Desktop>(
set "year= C2011060.A1_ABC.ABCD"
 set "daynum=!year:~5,3!"
 set "year=!year:~1,4!"
 call :OrdinalToDate !year! !daynum! yy month dd
 echo  C2011060.A1_ABC.ABCD.tif matches to !yy!-!month!-!dd!
 if !month! == 01 set Month=jan
 if !month! == 02 set Month=feb
 if !month! == 03 set Month=mar
 if !month! == 04 set Month=apr
 if !month! == 05 set Month=may
 if !month! == 06 set Month=jun
 if !month! == 07 set Month=jul
 if !month! == 08 set Month=aug
 if !month! == 09 set Month=sep
 if !month! == 10 set Month=oct
 if !month! == 11 set Month=nov
 if !month! == 12 set Month=dec
gdal_calculate --outfile=%out_path%\%%a --calc="((image1-image2))" --image2=%sample_path%\sample_file_!month!.tif --image1=%in_path%\%%a --extent=INTERSECT
)
Invalid attempt to call batch label outside of batch script.
C2011060.A1_ABC.ABCD.tif matches to !yy!-!month!-!dd!

Modified:

Code: [Select]

@echo off
setlocal enabledelayedexpansion

set "in_path=E:\Proc\Mer"
set "out_path=E:\Proc\Abcde"
set "sample_path=E:\Proc\Me"
set "proc_path=E:\Proc\Proc_Mer_Fi"

md %out_path%
md %proc_path%

cd /d "%in_path%"
for /f "tokens=* delims=" %%a in ('dir /b /on ????????*.tif ') do (
   set "year=%%~na"
   set "daynum=!year:~5,3!"
   set "year=!year:~1,4!"

call :OrdinalToDate !year! !daynum! yy month dd
echo %%a matches to !yy!-!month!-!dd!

if !month!==01 set Month=jan
if !month!==02 set Month=feb
if !month!==03 set Month=mar
if !month!==04 set Month=apr
if !month!==05 set Month=may
if !month!==06 set Month=jun
if !month!==07 set Month=jul
if !month!==08 set Month=aug
if !month!==09 set Month=sep
if !month!==10 set Month=oct
if !month!==11 set Month=nov
if !month!==12 set Month=dec
gdal_calculate --outfile=%out_path%\%%a --calc="((image1-image2))" --image2=%sample_path%\sample_file_!month!.tif --image1=%in_path%\%%a --extent=INTERSECT
)

echo done
goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

[_unknown_]

Error:

Code: [Select]

RuntimeError: `E:\Proc\Me\sample_file_!Month!.tif' does not exist in the file system,and is not recognised as a supported dataset name.
I think the !Month! is causing the error because it couldn't find a file with sample_file_!Month!.tif from the %sample_path% directory. When I tried to modify it with this %sample_path%\sample_file_!jan! it run(but with also an error which is the Invalid attempt thing) since I already changed the !Month! to a specific month which is January. How to fix that since I have 12 months? 

[foxidrive]

How did you launch this batch file?

I often see people pasting batch code into the cmd prompt these days - it makes me weep.

Is gdal_calculate a batch file too?

[Squashman]

Another thing how will I moved only the processed files to the processed_path? Thank you!

http://lmgtfy.com/?q=batch+move+files