TIME NOW
World current time now,
CALENDAR
Calendar monthly, yearly
login CONVERT LENGTH
login CONVERT TEMPERATURE
login DICTIONARIES, LISTS
login SCIENCE EDUCATION RELIGION
login WORK CALCULATOR
login CALCULATE LIFE

Linux commands, scripts, examples, tutorial

Previous articlePage bottomNext article  ALL TOPICS

Linux. wget. Random user agent, random proxy bash script, example

 load.scr file:
for i in {1..13}
do

    echo ">>>>>>>>>>>>>>>>>>>>>>>>";
    echo "Executing wget $i ......";
    echo ">>>>>>>>>>>>>>>>>>>>>>>>";
    
    #random sleep time 1-10 sec
    sleep_time=$[ ( $RANDOM % 10 ) + 1 ];
    echo "Sleep time: $sleep_time seconds";
    sleep $sleep_time

    #random user-agent header 1-13 line
    user_agent_number=$[ ( $RANDOM % 13 ) + 1 ];
    #echo $user_agent_number;
    user_agent=`awk -v r=$user_agent_number ' NR==r {print} ' user_agents.list`;
    
    echo ">>>>>>>>>>>>>>>>>>>>>>>";
    echo "User agent: $user_agent";
    echo ">>>>>>>>>>>>>>>>>>>>>>>>";

    #random http proxy 1-21 line
    proxy_number=$[ ( $RANDOM % 21 ) + 1 ];
    use_proxy=`awk -v r=$proxy_number ' NR==r {print} ' proxy.list`;
    
    echo ">>>>>>>>>>>>>>>>>>>>>>>";
    echo "Using proxy: $use_proxy";
    echo ">>>>>>>>>>>>>>>>>>>>>>>";
    export http_proxy=`$use_proxy`;
    
    do_dir="./$i";
    mkdir "$do_dir";

    wget -e robots=off -nd -r -l 10 --user-agent="`$user_agent`" -P $do_dir -A jpg,jpeg,gif,png,bmp http://www.google.lt/images/srpr/logo1w.png;

done

proxy.list file:
173.219.189.95:3128
212.72.56.21:80
111.161.30.237:80
111.161.30.232:80
111.161.30.231:80
41.89.130.4:3128
168.144.97.202:3128
78.134.255.43:8080
110.138.162.171:3128
202.4.120.10:3128
189.15.88.20:3128
94.126.17.68:3128
202.166.205.91:80
75.126.176.161:80
180.178.98.78:8080
210.51.59.144:443
117.3.66.115:80
201.240.59.106:80
115.249.63.243:3128
79.101.37.133:8080
190.81.197.97:8080

user_agents.list file:
Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1
Mozilla/5.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1
Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:14.0) Gecko/20120405 Firefox/14.0a1
Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20120405 Firefox/14.0a1
Mozilla/5.0 (Windows NT 5.1; rv:14.0) Gecko/20120405 Firefox/14.0a1
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/24.0.1290.1 Safari/537.13
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/24.0.1290.1 Safari/537.13
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/24.0.1290.1 Safari/537.13
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/24.0.1290.1 Safari/537.13
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3
 
Previous articlePage topNext article  ALL TOPICS



 Use username: Guest, Anonymous, Programmer






QUOTES:
To insist on strength ... is not war-mongering. It is peace-mongering.
Barry Goldwater
Nimble thought can jump both sea and land.
William Shakespeare
Silence at the proper season is wisdom, and better than any speech.
Plutarch