newlispfanclub.alh.net

Posted: **Sat Jan 26, 2013 4:57 am**

first:
I have wirted a multiporcess scanner.
i use ten process.
All of them blocked sometimes later!!!

(define (check_path host , url url2 result str  socket)
            (if-not (starts-with host "http://") (setf url2  (string "http://" host)))
            (dolist (port portlst)
                (when (and 
                            (setf socket (net-connect host (int port) TIMEOUT)) ;have question
                            (close socket)
                            (find "^ERR: server code 404" (get-url (append url2 ":" port FOOLCHECK) ) 1)
                            )
                            (dolist (path ALLPATH)
                            (setf url (append url2 ":" port  path));notice here 
                            ;(setf result (get-url url  "list debug"TIMEOUT AGENT))
                            (setf result (get-url url  TIMEOUT AGENT))
                            (if-not (find "^ERR: Operation timed out" result 1)
                                (if-not (find "^ERR.*\n*" result 1)
                                    (and
                                    (println "\r\n\r\n----------------------Bingo----------------------\r\n" url "\r\n")
                                    (set 'str  (string  "<a href=\"" url " \">" url "</a>        200 </br>"))
                                    (write FID_R  str)
                                    )
                                   ; (
                                    ;can add other err log ex:503 forbiden
                                    ;)
                              )
                              (and
                                (println "\r\n\r\n----------------------Time out----------------------\r\n" url "\r\n")
                                (set 'str  (string  "<a href=\"" url "\">" url "</a>        time out </br>"))
                                (write FID_R  str)
                              )
                        )
                    ) 

                )
            )
)
(set 'pnum (min pnum allipnum))
 (dotimes (n pnum)
    (spawn (sym (allip pcount)) (brute_path  (allip pcount)))
    (inc pcount)
    (println "")
)

(define (report pid)
    (semaphore sid -1)
    (when (< pcount allipnum)
        (spawn (sym (allip  pcount)) (check_path (allip pcount)))
    )
    (inc pcount)
    (semaphore sid 1)
)

i strace process find :
then always thus:

!netstat -anp -c 1| grep newlisp

tcp 1 0 192.168.51.103:46220 115.36.76.90:80 CLOSE_WAIT 9771/newlisp
tcp 0 0 192.168.51.103:52180 113.17.16.64:8000 ESTABLISHED 23789/newlisp
tcp 1 0 192.168.51.103:46192 115.26.76.90:80 CLOSE_WAIT 9771/newlisp
tcp 1 0 192.168.51.103:52520 115.26.76.90:80 CLOSE_WAIT 9771/newlisp
tcp 0 0 192.168.51.103:35091 121.1.45.14:8080 ESTABLISHED 24664/newlisp
tcp 1 0 192.168.51.103:46187 115.36.76.90:80 CLOSE_WAIT 9771/newlisp
tcp 0 0 192.168.51.103:48741 115.8.11.11:8001 ESTABLISHED 24305/newlisp
tcp 0 0 192.168.51.103:36782 58.9.19.181:80 ESTABLISHED 24934/newlisp
tcp 0 0 192.168.51.103:32917 121.4.45.10:8002 ESTABLISHED 24575/newlisp
tcp 0 0 192.168.51.103:37811 58.24.19.13:8080 ESTABLISHED 24926/newlisp
tcp 1 0 192.168.51.103:46200 115.23.76.90:80 CLOSE_WAIT 9771/newlisp
tcp 0 0 192.168.51.103:44936 121.14.45.12:80 ESTABLISHED 24607/newlisp
tcp 0 0 192.168.51.103:45617 115.28.1.206:80 ESTABLISHED 24423/newlisp
tcp 0 0 192.168.51.103:41055 121.1.45.121:8002 ESTABLISHED 24596/newlisp
tcp 0 0 192.168.51.103:43155 121.4.45.240:80 ESTABLISHED 24720/newlisp

strace -v -p 24664
select(6, [5], NULL, NULL, {0, 1000}) = 0 (Timeout)
select(6, [5], NULL, NULL, {0, 1000}) = 0 (Timeout)
select(6, [5], NULL, NULL, {0, 1000}) = 0 (Timeout)
.
.
.

second:

Code: Select all

(set 'socket (net-connect "www.baidu.com" 80))
(net-send socket "GET /\r\n\r\n")
(net-receive socket buffer 10000);have issuses
(println buffer)
(exit)

Posted: **Fri Feb 01, 2013 3:58 am**

I found where the problem.

Code: Select all

get-url

Function get-url always not return on child process.
Even if you set the timeout does not work。
Comment out the line that contains the “get-url”, will be able to perform good.

Posted: **Fri Feb 01, 2013 9:54 am**

Whenever you use spawn, you also have to use sync. This function processes the return signals of child-processes. Without doing this, they will stay in memory.

See here for a programs using spawn to retrieve multiple web pages in parallel using get-url with a timeout:

http://www.nuevatec.com/query.html
and here for the complete source:
http://www.nuevatec.com/syntax.cgi?query-cgi.txt

this critical snippet is towards the bottom of the program:

Code: Select all

; spawn a childprocess for each link
(dolist (lnk (0 20 links))
	(set 'pid (spawn 'page (get-url lnk 4000)))
	(push (list pid lnk) SITES -1))

; this gets executed whenever a page has been retrieved
(define (page-ready-event pid)
    (let (link (0 80 (lookup pid SITES)))
        (set 'link (url-decode link))
        (println (inc cnt) " pid:" pid " " (- (time-of-day) START_TIME) " ms " link BRK)
        (push (lower-case (clean-html page)) CONTENT -1)
        (inc xferred (length page)))
)

; start waiting for pages
(println "waiting: ..." BRK BRK)
(unless (sync 10000  page-ready-event)
	(println BRK "timeout" BRK))

see the sync statement at the end - scroll up.

Posted: **Sun Feb 17, 2013 7:18 am**

thx lutz...

冏
I forgot to paste the code about "sync" portion.
I use "sync"from the outset. --!

Post it to github now:
https://github.com/freewinger/wsec_tool ... canner.lsp

Posted: **Sun Feb 17, 2013 3:44 pm**

Perhaps you have some unknown error thrown in brute_path. Try the following to catch anything occurring in that function:

Code: Select all

(spawn (sym (allip  pcount)) 
    (begin (catch (brute_path (allip pcount)) 'result) result) 
)

I also wonder about another spawn in the event funtion report handling the returns from spawns. The event function is called from inside sync in the same process. Semaphores would not be necessary in that case.

Posted: **Sat Feb 23, 2013 5:21 am**

Code: Select all

;i change report function to  following :
(define (report pid)
    (semaphore sid -1)
    (when (< pcount allipnum)
        (spawn (sym (allip  pcount)) 
            (if (begin (catch (brute_path (allip pcount)) 'result) result) (println  "----:----> "result  (read-key)))
        )
    )
    (inc pcount)
    (semaphore sid 1)
)

But code

Code: Select all

(println  "----:----> "result  (read-key))

never execute (This should be a syntax error--!) .

Three days i satart script use

newlisp svnscan.lsp -n 60 -f ip2.txt -sc

Find script just output

" ."

when i come back today .
It is work done after I press the “Enter” key four times . (it output num 10 every time . mybe it's number of bytes written by the function "write" ?).

........................................................................................................................................................................10
.......................................................................................
10
10
10
Begin process scan result
Spend 0 5397 minutes 27 seconds
Result file: svnscan_result2013年02月19日17时48分56秒.html

# ps -A |grep newlisp
12626 pts/11 3-17:38:38 newlisp
21272 pts/11 00:00:12 newlisp
27581 pts/11 00:00:11 newlisp
28713 pts/11 00:00:11 newlisp

#strace -v -p 12626
....
gettimeofday({1361591102, 962949}, NULL) = 0
waitpid(-1, 0xbfeff0e8, WNOHANG) = 0
gettimeofday({1361591102, 962993}, NULL) = 0
waitpid(-1, 0xbfeff0e8, WNOHANG) = 0
gettimeofday({1361591102, 963037}, NULL) = 0
....

#strace -v -p 21272
Process 21272 attached - interrupt to quit
read(0,
.........

Posted: **Sat Feb 23, 2013 5:53 am**

sync is not able to continue when waiting on read-key in the event function, it has called. The four times you hit the enter key is for four spawned processes, which had finished, but could not be processed further, because sync was waiting for return from the report event function. Don't put anything blocking into the event function.

newlispfanclub.alh.net

Some issuses about socket function

Some issuses about socket function

Re: Some issuses about socket function

Re: Some issuses about socket function

Re: Some issuses about socket function

Re: Some issuses about socket function

Re: Some issuses about socket function

Re: Some issuses about socket function