new webfs, rc based hget

3 files changed, 119 insertions, 275 deletions

diff --git a/sys/man/1/hget b/sys/man/1/hget
index 7cc8c97c5..9e6a4ed3f 100644
--- a/sys/man/1/hget
+++ b/sys/man/1/hget
@@ -4,19 +4,20 @@ hget \- retrieve a web page corresponding to a url
 .SH SYNOPSIS
 .B hget
 [
-.B -dhv
-] [
 .B -o
-.I ofile
+.I file
 ] [
 .B -p
 .I body
 ] [
-.B -x
-.I netmntpt
-] [
 .B -r
 .I header
+] [
+.B -m
+.I method
+] [
+.B -b
+.I baseurl
 ]
 .I url
 .SH DESCRIPTION
@@ -26,7 +27,16 @@ retrieves the web page specified by the URL
 and writes it, absent the
 .B -o
 option, to standard output.
-The known URL types are: http and ftp.
+.PP
+The
+.I url
+can be a relative path like
+.B ../index.html
+if a absolute
+.I baseurl
+was specified with the
+.B -b
+option.
 .PP
 If
 .I url
@@ -47,40 +57,22 @@ but incomplete,
 will fetch the missing bytes.
 .PP
 Option
-.B -h
-causes HTTP headers to be printed to standard output
-in addition to the transferred web page.
-.PP
-Option
 .B -r
 sends an arbitrary HTTP
 .IR header .
 .PP
 Option
-.B -d
-turns on debugging written to standard error.
-.PP
-Normally,
-.I hget
-uses the IP stack mounted under
-.BR /net .
-The
-.B -x
-option can be used to specify the mount point of
-a different IP stack to use.
-.PP
-Option
-.B -v
-writes progress lines to standard error once a second.
-Each line contains two numbers, the bytes transferred so
-far and the total length to be transferred.
-.PP
-If the environment variable
-.B httpproxy
-is set, it is used as a URL denoting an HTTP proxy server.
-All HTTP accesses use this server to get the page instead of
-calling the destination server.
+.B -m
+overrides the HTTP method used for the request.
 .SH SOURCE
-.B /sys/src/cmd/hget.c
+.B /rc/bin/hget
 .SH "SEE ALSO"
+.IR webfs (4),
 .IR ftpfs (4)
+.SH DIAGNOSTICS
+.I Hget
+requires 
+.IR webfs (4)
+service mounted on
+.B /mnt/web
+to work.
diff --git a/sys/man/4/webcookies b/sys/man/4/webcookies
index 63842e90f..b4c6c2168 100644
--- a/sys/man/4/webcookies
+++ b/sys/man/4/webcookies
@@ -147,20 +147,8 @@ If
 .B cookiefs
 decides not to accept the cookie (as outlined in
 RFC2109, section 4.3.4), no indication is given.
-.PP
-.IR Hget (1)
-uses
-.BR /mnt/webcookies/http ,
-when it exists, to manage cookie state.
-.I Webfs
-does not (yet).
 .SH SOURCE
 .B /sys/src/cmd/webcookies.c
 .SH SEE ALSO
+.IR webfs (4),
 .IR hget (1)
-.SH BUGS
-It's not clear what the relationship between
-.I cookiefs
-and something like
-.I webfs
-should be.
diff --git a/sys/man/4/webfs b/sys/man/4/webfs
index 5f0d40720..630f0911f 100644
--- a/sys/man/4/webfs
+++ b/sys/man/4/webfs
@@ -4,10 +4,6 @@ webfs \- world wide web file system
 .SH SYNOPSIS
 .B webfs
 [
-.B -c
-.I cookiefile
-]
-[
 .B -m
 .I mtpt
 ]
@@ -26,10 +22,15 @@ mounts itself at
 .BR /mnt/web ),
 and, if 
 .I service
-is specified, will post a service file descriptor
-in 
+is specified, will post a service file descriptor in 
 .BR /srv/\fIservice .
 .PP
+If the enviroment variable
+.B httpproxy
+is set, all HTTP request initiated by
+.I webfs
+will be made thru that proxy url.
+.PP
 .I Webfs
 presents a three-level file system suggestive
 of the network protocol hierarchies
@@ -37,13 +38,12 @@ of the network protocol hierarchies
 and
 .IR ether (3).
 .PP
-The top level contains three files:
+The top level contains the files files:
 .BR ctl ,
-.BR cookies ,
 and
 .BR clone .
 .PP
-The
+The top level
 .B ctl
 file is used to maintain parameters global to the instance of
 .IR webfs .
@@ -53,72 +53,6 @@ file yields the current values of the parameters.
 Writing strings of the form
 .RB `` attr " " value ''
 sets a particular attribute.
-Attributes are:
-.TP
-.B chatty9p
-The
-.B chatty9p
-flag used by the 9P library, discussed in
-.IR 9p (2).
-.B 0
-is no debugging,
-.B 1
-prints 9P message traces on standard error,
-and values above
-.B 1
-present more debugging, at the whim of the library.
-The default for this and the following debug flags is 
-.BR 0 .
-.TP
-.B fsdebug
-This variable is the level of debugging output about the file system module.
-.TP
-.B cookiedebug
-This variable is the level of debugging output about the cookie module.
-.TP
-.B urldebug
-This variable is the level of debugging output about URL parsing.
-.TP
-.B acceptcookies
-This flag controls whether to accept cookies presented by remote web servers.
-(Cookies are described below, in the discussion of the
-.B cookies
-file.)
-The values
-.B on
-and
-.B off
-are synonymous with
-.B 1
-and
-.BR 0 .
-The default is
-.BR on .
-.TP
-.B sendcookies
-This flag controls whether to present stored cookies to remote web servers.
-The default is
-.BR on .
-.TP
-.B redirectlimit
-Web servers can respond to a request with a message
-redirecting to another page.
-.I Webfs
-makes no effort to determine whether it is in an infinite
-redirect loop.
-Instead, it gives up after this many redirects.
-The default is
-.BR 10 .
-.TP
-.B useragent
-.I Webfs
-sends the value of this attribute in its
-.B User-Agent:
-header in its HTTP requests.
-The default is
-.RB `` "webfs/2.0 (plan 9)" .''
-.PD
-.PP
 The top-level directory also contains
 numbered directories corresponding to connections, which
 may be used to fetch a single URL.
@@ -131,23 +65,10 @@ After opening, the
 .B clone
 file is equivalent to the file
 .IB n /ctl \fR.
-A connection is assumed closed once all files in its directory
-have been closed, and is then will be reallocated.
-.PP
-Each connection has its own private set of
-.BR acceptcookies ,
-.BR sendcookies ,
-.BR redirectlimit ,
-and
-.B useragent
-variables, initialized to the defaults set in the
-root's
-.B ctl
-file.  The per-connection
-.B ctl
-file allows editing the variables for this particular connection.
+A connection is assumed closed once all files in its
+directory have been closed, and is then will be reallocated.
 .PP
-Each connection also has a URL string variable
+Each connection has a URL attribute
 .B url
 associated with it.
 This URL may be an absolute URL such as
@@ -156,153 +77,96 @@ or a relative URL such as
 .IR ../index.html .
 The
 .B baseurl
-string variable sets the URL against which relative URLs
+attribute sets the URL against which relative URLs
 are interpreted.
-Once the URL has been set,
-its pieces can be retrieved via individual files in the
-.B parsed
-directory.
-.I Webfs
-parses the following URL syntaxes; names in italics are
-the names of files in the
+Once the URL has been set by wrting to the
+.B ctl
+file of the connetcion, its pieces can be retrieved via
+individual files in the
 .B parsed
-directory.
-.IP
-\fIscheme\f5:\fIschemedata
-.br
-\f5http://\fIhost\f5/\fIpath\fR[\f5?\fIquery\fR][\f5#\fIfragment\fR]
-.br
-\f5ftp://\fR[\fIuser\fR[\f5:\fIpassword\fR]\f5@\fR]\fP\f5\fIhost\f5/\fIpath\fR[\f5;type=\fIftptype\fR]
-.br
-\f5file:\fIpath
-.LP
-If there is associated data to be
-posted with the request, it can be written to
+directory:
+.de UU
+.TP
+.B parsed/\fI\\$1
+\\$2
+..
+.UU url http://pete:secret@www.example.com:8000/cgi/search?q=kittens#results
+.UU scheme http
+.UU user pete
+.UU pass secret
+.UU host www.example.com
+.UU port 8000
+.UU path /cgi/search
+.UU query q=kittens
+.UU fragment results
+.PP
+If there is associated data to be posted with the request,
+it can be written to
 .BR postbody .
-Finally, opening
+Opening
+.B postbody
+or
 .B body
-initiates the request.
-The resulting data may be read from
+initiates the request. If the request fails,
+then opening the
 .B body
-as it arrives.
-After the request has been executed, the MIME content type
-may be read from the
+or writing to
+.B postbody
+file will fail and return a error string.
+.PP
+When the
+.B body
+file has been opend, response headers appear
+as files in the connection directory. For example
+reading the
 .B contenttype
-file.
+file yields the MIME content type of the body data.
+If the request was redirected, the URL represended
+by the
+.B parsed
+directory will change to the final destination.
 .PP
-The top-level
-.B cookies
-file contains the internal set of HTTP cookies, which
-are used by HTTP servers to associate requests with persistent
-state such as user profiles.
-It may be edited as an ordinary text file.
-Multiple instances of
-.I webfs
-and
-.IR webcookies (4)
-share cookies by keeping their internal set
-consistent with the
-.I cookiefile
-(default
-.BR $home/lib/webcookies ),
-which has the same format.
+The resulting data may be read from
+.B body
+as it arrives.
 .PP
-These files contain one line per cookie;
-each cookie comprises some number of
-.IB attr = value
-pairs.
-Cookie attributes are:
+The following is a list of attributes that can be
+set to to a connection prior initiating the request:
 .TP
-.BI name= name
-The name of the cookie on the remote server.
+.B url,baseurl
+See above.
 .TP
-.BI value= value
-The value associated with that name on the remote server.
-The actual data included when a cookie is sent back
-to the server is
-.IB \fR``\fIname = value\fR''
-(where, confusingly,
-.I name
-and
-.I value
-are the values associated with the
-.B name
-and
-.B value
-attributes.
-.TP
-.BI domain= domain
-If
-.I domain
-is an IP address, the cookie can only be used for URLs
-with
-.I host
-equal to that IP address.
-Otherwise,
-.I domain
-must be a pattern beginning with a dot, and
-the cookie can only be used for URLs with a
-.I host
-having
-.I domain
-as a suffix.
-For example, a cookie with
-.B domain=.bell-labs.com
-may be used on hosts
-.I www.bell-labs.com
-and
-.IR www.research.bell-labs.com
-(but not
-.IR www.not-bell-labs.com ).
-.TP
-.BI path= path
-The cookie can only be used for URLs with a path
-beginning with
-.IR path .
-.TP
-.BI version= version
-The version of the HTTP cookie specification, specified by the server.
-.TP
-.BI comment= comment
-A comment, specified by the server.
-.TP
-.BI expire= expire
-The cookie expires at time
-.IR expire ,
-which is a decimal number of seconds since the epoch.
-.TP
-.B secure=1
-The cookie may only be used over secure
-.RB ( https )
-connections.
-Secure connections are currently unimplemented.
-.TP
-.B explicitdomain=1
-The domain associated with this cookie was set by
-the server (rather than inferred from a URL).
-.TP
-.B explicitpath=1
-The path associated with this cookie was set by the
-server (rather than inferred from a URL).
+.B useragent
+Sets a custom useragent string to be used with the request.
 .TP
-.B netscapestyle=1
-The server presented the cookie in ``Netscape style,'' which
-does not conform to the cookie standard, RFC2109.
-It is assumed that when presenting the cookie to the server,
-it must be sent back in Netscape style as well.
-.PD
+.B contenttype
+Sets the MIME content type of the postbody.
+.TP
+.B request
+Usualy, the HTTP method used is
+.B POST
+when
+.B postbody
+file is opend first or
+.B GET
+otherwise. This can be overriden with the
+.B request
+attribute so send arbitrary HTTP requests.
+.TP
+.B headers
+Adds arbitrary HTTP headers to be send with
+the request.
 .SH EXAMPLE
-.B /sys/src/cmd/webfs/webget.c
+.B /rc/bin/hget
 is a simple client.
 .SH SOURCE
 .B /sys/src/cmd/webfs
-.SH SEE ALSO
-.IR hget (1),
-.IR webcookies (4)
-.SH BUGS
-It's not clear what the relationship between
-.IR hget ,
-.I webcookies
-and
-.I webfs
-should be.
+.SH "SEE ALSO"
+.IR webcookies (4),
+.IR hget (1)
+.SH DIAGNOSTICS
+For cookies to work,
+.IR webcookies (4),
+should be running and mounted on
+.B /mnt/webcookies
+otherwise cookies will be ignored.