Welcome guest. Before posting on our computer help forum, you must register. Click here it's easy and free.

Author Topic: Want help with SED for Windows.  (Read 6336 times)

0 Members and 1 Guest are viewing this topic.

Geek-9pm

    Topic Starter

    Mastermind
  • Geek After Dark
  • Thanked: 1026
    • Gekk9pm bnlog
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 10
Want help with SED for Windows.
« on: November 24, 2017, 09:40:45 PM »
This is not really very important, eventually I would find a workaround. But some of you aire really good at this, so I thought I would ask.  :)

This ought to be super simple with SED for windows.
http://gnuwin32.sourceforge.net/packages/sed.htm

Here is what I have. A text file with a lot of domain names, but lacking the http:// prefix that ought to go in front of each domain. Each line has just one name. No white space. File is notepad compatible.

File names.txt has:

Code: [Select]
abcnews.com
bbcnews.com
cbsreport.com
denverpost.com
And so on. If it was only a few nme, I would just add teh prefic by hand. But the lit is very long and it is subject to change. So I would like to have a easy way to but the prefix in front of each name.
Like  this:

Code: [Select]
sed s#^http://#g# names.txt
It just echos the lines in the names.txt file and does not add the prefix I want.
No error message.

What did I do wrong?   :(


BC_Programmer


    Mastermind
  • Typing is no substitute for thinking.
  • Thanked: 1140
    • Yes
    • Yes
    • BC-Programming.com
  • Certifications: List
  • Computer: Specs
  • Experience: Beginner
  • OS: Windows 11
Re: Want help with SED for Windows.
« Reply #1 on: November 24, 2017, 10:33:03 PM »
No need to use SED:
Code: [Select]
FOR /F "tokens=*" %P in (names.txt) do echo http://%P >> httpnames.txt

(use %%P instead of %P if using in a batch file)

I'd guess that the issues with sed involve / having a special meaning which you need to escape.
I was trying to dereference Null Pointers before it was cool.

Geek-9pm

    Topic Starter

    Mastermind
  • Geek After Dark
  • Thanked: 1026
    • Gekk9pm bnlog
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 10
Re: Want help with SED for Windows.
« Reply #2 on: November 24, 2017, 11:56:53 PM »
WOW! That works!
Thanks.

Salmon Trout

  • Guest
Re: Want help with SED for Windows.
« Reply #3 on: November 25, 2017, 01:51:18 AM »
Dunno if this helps. For some reason, I seem to prefer awk for this kind of thing. If you are using sed on Windows, maybe you also have awk or gawk?

In Linux Bash: this works in Ubuntu WSL on Windows 10; Debian 8 Jessie; ARM Linux on my NAS:

Code: [Select]
$ cat input.txt
apple
bear
cat
dog
egg
franklin
ghost
$ awk '{print "http://"$0}' input.txt
http://apple
http://bear
http://cat
http://dog
http://egg
http://franklin
http://ghost

In Win32 cmd, using GNU awk (gawk), the shell passes single quotes unchanged, so use doubles, and escape the doubles in the prefix string with 2 extra:

Code: [Select]
D:\>type input.txt
apple
bear
cat
dog
egg
franklin
ghost
D:\>gawk "{print """http://"""$0}" input.txt
http://apple
http://bear
http://cat
http://dog
http://egg
http://franklin
http://ghost



Salmon Trout

  • Guest
Re: Want help with SED for Windows.
« Reply #4 on: November 25, 2017, 03:46:15 AM »
These are very similar

for /f "delims=" %0 in (input.txt) do @echo http://%0
gawk "{print """http://"""$0}" input.txt


Of course in Linux (Bash) you can just do this

while read line; do echo "http://$line"; done < input.txt




« Last Edit: November 25, 2017, 04:45:26 AM by Salmon Trout »

Salmon Trout

  • Guest
Re: Want help with SED for Windows.
« Reply #5 on: November 25, 2017, 05:03:55 AM »
Of course in Linux (Bash) you can just do this

while read line; do echo "http://$line"; done < input.txt
if you did not have/use WSL but you have a bash.exe in Windows (e.g. Cygwin) you could do this (your path to bash.exe will be different!)

c:\utils\winbash\bash.exe -c "while read line; do echo "http://$line"; done < input.txt"

But if you have bash why *censored* around in CMD? (could apply to whole topic).



DaveLembke



    Sage
  • Thanked: 662
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 10
Re: Want help with SED for Windows.
« Reply #6 on: November 25, 2017, 12:27:39 PM »
I've used Perl to take a list of numbers and generate URL's out of them such as

Item #'s

11111
22222
33333
44444

and write to text file as


https://www.somedomainname.com/item=11111
https://www.somedomainname.com/item=22222
https://www.somedomainname.com/item=33333
https://www.somedomainname.com/item=44444


Also wrote one that places the items from a list that is read in into the middle of a URL path when farming information online and I know unique identifier that I am targeting gathering information on and need to prepend and append info to make it into a URL

Looks like your all set though with what you have. Many different script/language methods to get to the same result.  ;D

When gathering information on 37,000 items, this was sweet. It took a while to read in each line and write out to file the prepend and appended info and hard drive was very busy, but was better than what I had been doing prior which was running a macro loop to edit a text file to add information for each line which took forever. The Perl method was so much faster and not as buggy as the macro that would without any reason go berserk and start writing information in wrong places and make a huge mess running as a TSR on the system. The Perl method is flawless and precise vs a keyboard macro running in a loop for x-many iterations which had a flaw in it somewhere that caused it to foul everything up.

After getting my full list of proper URL's, I had a team of systems running on their own gathering information on all of these items unattended which took a while. For 1 computer it would have taken about 154.17 hours at one items information gathered every 15 seconds. I used 5 systems and split the list up into 5 shorter lists to get this done in around 31 hours time. the 15 seconds was because the gathering automation i had I had to ensure that the webpage was fully loaded before information gathered before moving on to the next item to gather info from. The webpage had lots of elements that caused for a not so fast load time even with a broadband internet connection, so some of the 15 seconds was wasted time as I made a delay for full loading of webpages before the quick gathering and moving on to the next item.

Then all that info gathered I was able to look further into that information quickly locally as well as generate reports and graphs and stuff and look for patterns and other items of interest with the data that was gathered that is public domain free for all.  ;D

A professional programmer who could specifically target a pages element could probably gather the information a lot faster reading in the exact information from a specific location in a webpage knowing it populated with info before moving on and no delay padding used to ensure its there before gathering, but thats beyond my programming skills with gathering information online from websites.  :)   So my methods are Rube Goldberg masterpieces that work, but not as efficient.  ;D

One of my C# books had a section with browser hooks which I suppose could be used for parsing information from websites, but I never dug deep into this. Maybe some day since I still have the book.  ;D

Geek-9pm

    Topic Starter

    Mastermind
  • Geek After Dark
  • Thanked: 1026
    • Gekk9pm bnlog
  • Certifications: List
  • Computer: Specs
  • Experience: Expert
  • OS: Windows 10
Re: Want help with SED for Windows.
« Reply #7 on: November 25, 2017, 05:11:11 PM »
Thanks all for the education.
I have SED installed because it is a lightweight tool for the few times I need to do search and replace. The issues was ho to define the null string at the start of a line. Th use of ^ has me confused.
As to BASH, I did not know there was BASH for windows.
Quote
Bash for Windows

Everything You Can Do With Windows 10’s New Bash Shell.  This isn’t a virtual machine, a container, or Linux software compiled for Windows (like Cygwin). Instead, Windows 10 gains a Windows Subsystem for Linux, which is based on Microsoft’s abandoned Project Astoria work for running Android apps on Windows.
https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/
He goes on to say:
Quote
This isn’t a virtual machine, a container, or Linux software compiled for Windows (like Cygwin). Instead, Windows 10 gains a Windows Subsystem for Linux, which is based on Microsoft’s abandoned Project Astoria work for running Android apps on Windows.

Think of it as the opposite of Wine. While Wine allows you to run Windows applications directly on Linux, the Windows Subsystem for Linux allows you to run Linux applications directly on Windows.
Really, that is not new. I just did not grasp the impact.

So, if I feel the need to bash windows 10, I can bash windows 10.  ::)

Learn something almost new every day.   ;D

Again, thanks to all. I will bookmark this for future reference.

Salmon Trout

  • Guest
Re: Want help with SED for Windows.
« Reply #8 on: November 26, 2017, 02:27:57 AM »
I have SED installed because it is a lightweight tool for the few times I need to do search and replace.
Had to do 2 things to your command to make it work in GNU sed in Windows shell:

(1) Omit the extra delimiter # at the end.
(2) In Windows cmd shell, double quote the sed command. (In Linux Bash, double or single quotes work the same).

Note: In this case, you don't need the g option (which means "replace every occurrence of the regex, not just the first"), because each line only has one beginning to replace.

sed "s#^#http://#g" names.txt

In Windows 10 command prompt:

C:\>type names.txt
abcnews.com
bbcnews.com
cbsreport.com
denverpost.com

C:\>sed "s#^#http://#g" names.txt
http://abcnews.com
http://bbcnews.com
http://cbsreport.com
http://denverpost.com


« Last Edit: November 26, 2017, 02:55:51 AM by Salmon Trout »

Salmon Trout

  • Guest
Re: Want help with SED for Windows.
« Reply #9 on: November 26, 2017, 03:00:01 AM »
Note: In this case, you don't need the g option (which means "replace every occurrence of the regex, not just the first"), because each line only has one beginning to replace.

Example:

C:\>type a.txt
aaaa
aaaa
aaaa
aaaa
aaaa
aaaa

C:\>sed "s#a#b#" a.txt
baaa
baaa
baaa
baaa
baaa
baaa

C:\>sed "s#a#b#g" a.txt
bbbb
bbbb
bbbb
bbbb
bbbb
bbbb