MSDOS Bat File program to Copy the Next Line if Findstr Line is True

[jbelmhicco]

Hello, I am a novice in Bat File making and I need to make this in Bat File since Excel Macro would take a long time in summarizing it.

I have a many csv files to summarize and only need to get its specific line along with its filename. See below for the line:

RawFile:
line64: "Rank","Factor","Over","Time","Serial No.","Disk No.","","",""
line65: 1,2,0,"2013/11/19 13:51:39",44787,44787,"","",""

SummaryOutput:
00044787.csv:1,2,0,"2013/11/19 13:51:39",44787,44787,"","",""
00045276.csv:3,6,0,"2013/11/19 15:30:16",45276,45276,"","",""
00045718.csv:2,14,0,"2013/11/19 16:50: 1",45718,45718,"","",""
00048002.csv:3,10,1,"2013/11/21  2:55: 2",48002,48002,"","",""
00049871.csv:3,10,0,"2013/11/21 15:34:15",49871,49871,"","",""

Can someone, please help me. i only know how to findstr. v_v

[gsnidow]

jbelmhicco, what makes you think an Excel macro would take too long?  I've got some macros that iterate through many files, summarize the data, close each workbook, and write the results to a new workbook, and it only takes a few seconds.  You could probably iterate through hundreds of files in only a few minutes with a VBA macro.  I don't know a lot about batch programming/dos, but any option you use is going to have to open and close the workbook.  Plus, if you write something like that, chances are there are going to be few people in your workplace able to edit it, i.e. you own it forever.  My humble advice would be to stick with VBA as much as possible for anything having to do with MS Office applications.

Greg

[jbelmhicco]

hello gsnidow! thank you for the advice...

excel macro would take too long since i will manipulate thousands of csv files. bat file would take less time than macro since it wont need to open the file. have you tried comparing the two?

[gsnidow]

I have not compared the two, but I would be interested to see what some of the dos/bat programmers suggest.  Good luck.

[Squashman]

I have seen stuff like this done both ways. I  personally think the larger the file the better off your are doing this with VB. Batchfiles become exponentially slower the larger the file it has to read is because it has to read in the entire file into memory before it can start processing it.

[Salmon Trout]

Check the date of the last post previous to yours, Squashman.

[briandams]

Check the date of the last post previous to yours, Squashman.

what about the date?

[Salmon Trout]

what about the date?

Over one month before. Effectively, a dead thread, although the topic is interesting. I may play with some scripts. I am not sure about Squashman's assertion that batch scripts read the whole of a file into memory before processing it. It is true that cmd.exe reads the whole of a batch script into memory before executing it, but as far as I know FOR /F reads files line by line. I am also not convinced that reading a file into memory is actually slower.

[briandams]

for very large file typically I would look for memory mapping instead of line by line processing. eg Perl module File::Map does the job.

[Salmon Trout]

If the number of lines preceding the wanted stuff is a thousand or less, batch can find the wanted info is less than a second on my machine, but bigger amounts of data to sift can increase times rapidly...

In VBScript you have ReadAll, which reads an entire file into one long string, then you can get it into a an array of lines by using the Split function with crlf as the split delimiter.

In pure batch, reading a file line by line is the crazy slow way to do it, for anything over a few lines.

I created test files thus:

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
Trigger for Next line
Show this line

... a block of Lorem Ipsum of 100 1,000, 10,000 and 100,000 lines before a "previous line" and a "wanted line".

First used a crude batch method (FOR /F) to read lines until the trigger line was found, counting lines as I went, and got these times

Lines   Time
   100    4.10 sec
  1000   41.72 sec
 10000  459.14 sec
100000 6465.01 sec (!)

However using FIND /C to get the line number of the trigger line, and then using FOR /F with that number as a skip value, to read the wanted line, I got these times

100 lines to skip
FIND     0.08 secs   
FOR /F   0.03 secs
Total    0.11 secs

1,000 lines to skip
FIND     0.08 secs
FOR /F   0.20 secs
Total    0.28 secs

10,000 lines to skip
FIND     0.31 secs
FOR /F  17.64 secs     
Total   17.95 secs

100,000 lines to skip
FIND      2.65 secs
FOR /F 2031.63 secs (!)
Total  2034.28 secs


Next VBScript...

[Squashman]

Try reading in a 700MB file in a for loop on a 1gb memory system. Open up task manager and watch your memory consumption. You will see it spike 700mb that is if you even have enough memory to read it at all at which point you will get an out of memory error. I just went over this with another user not to long ago.

[Salmon Trout]

I think it is clear that cmd.exe is not meant for 700 MB text files! My test file of 100,000 lines is only 12.5 MB in size. There would have to be nearly 6 million lines of that average length to get it to 700 MB, and the time taken to chug through it would be measured in days (to get through only 100,000 lines took over an hour on my system)

However, I have been timing Visual Basic Script and got these figures

Loading entire file into an array, splitting it at each CRLF and then iterating through the lines in a loop until the required line is found. Much faster than batch!

Times showing as 0.0000 secs represent intervals below the resolution of the VBScript Timer function (about 15 milliseconds)

100 lines to search past
Read file   0.0078 secs
Split array 0.0000 secs
Find line   0.0000 secs

1,000 lines to search past
Read file   0.0234 secs
Split array 0.0000 secs
Find line   0.0000 secs

10,000 lines to search past
Read file   0.1250 secs
Split array 0.0078 secs
Find line   0.0078 secs

100,000 lines to search past
Read file   1.2500 secs
Split array 0.3281 secs
Find line   0.1094 secs

[Squashman]

I used to have a script that was pure batch that would append a code to the end of every line and output to a combined file. Some of the input files were close to that size and I knew that if they were much bigger I would get memory errors. Now it would take a couple of hours but not much more than that. I then switched to using a 3rd party utility called Swiss File Knife. No more memory errors and it runs much faster.

[briandams]

I think it is clear that cmd.exe is not meant for 700 MB text files! My test file of 100,000 lines is only 12.5 MB in size. There would have to be nearly 6 million lines of that average length to get it to 700 MB, and the time taken to chug through it would be measured in days (to get through only 100,000 lines took over an hour on my system)

you are using type + for loop? or other methods? how about timing? what are you using to time the script?

[briandams]

I then switched to using a 3rd party utility called Swiss File Knife. No more memory errors and it runs much faster.

most probably done with C or some lower level lang + compiled. its almost a fact that lower level (in the likes of C/C++ ) and how its coded (algorithm) to read big files play a part in performance.