get-url question

nikolean · Post by **nikolean** » Mon Nov 21, 2005 8:52 pm

newbie in NewLisp here...greetings to NewLisp creator and community...

I have a question. I'm trying to create (my first more or less serious) newlisp script that should go to password protected page, login there, and fetch some information from the next page after login... I have created similar scripts in perl in the past, and now trying to do the same in newlisp.... and got some problems...
Login page sets cookie that need to be transferred to next page (along with some other info passed in the page body). However, when I try to use get-url (or post-url - doesn't matter in this case), I can read only either page header (with cookie data) or body. I need to get both parts of the request to be able to read cookie AND other variables stored inside HTML/form code. Is it possible?

thanks,
Andy

newdep · Post by **newdep** » Mon Nov 21, 2005 9:31 pm

Hi Nikolean,

I have never tried it but its only possible with your own post/get url handler i think. The version that is used inside newlisp does only fetch the page data directly. But if you know how a cookie is set during the password phase then its not that difficult to build it in newlisp.

regards, Norman.

Lutz · Post by **Lutz** » Tue Nov 22, 2005 12:30 am

You could try to talk with 'net-connect', 'net-send' etc. Look at the following interactive newLISP session:

Code: Select all

> (net-connect "newlisp.org" 80)
3 
> (net-send 3 "GET / HTTP/1.1\r\nHost: anybody.com\r\nUser-Agent: newlisp\r\n\r\n")
54 
> (net-receive 3 'buff 100000)
5305 
> buff
[text]HTTP/1.1 200 OK
Date: Tue, 22 Nov 2005 03:18:42 GMT
Server: Apache/1.3.29 (Unix) mod_python/2.7.10 Python/2.2.2 mod_webapp/1.2.0-dev mod_perl/1.29 mod_throttle/3.1.2 PHP/4.3.4
Transfer-Encoding: chunked
Content-Type: text/html

49 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>

But you would have to handle things like "chunked transfer encoding" yourself.

Lutz

nikolean · Post by **nikolean** » Tue Nov 22, 2005 11:10 am

many thanks for responses and example! I will try to use socket functions...
But I think it would be great to add one more option to get-url and similar functions to be able to get whole response content... or may be return it as a list (header/body) ?

Lutz · Post by **Lutz** » Tue Nov 22, 2005 11:56 am

Such an option may appear in a future release. Internally the header is already parsed out and separated (see "header" and "dbug" options).

Lutz