[Zope] Vhosting

sean.upton@uniontrib.com sean.upton@uniontrib.com
Fri, 23 Aug 2002 17:27:44 -0700


How 'bout a slightly different approach:

Setup Squid on port 80; use a redirector.  Search freshmeat for pyredir;
it's a good one, written in python, and very simple to use.  The redirector
is your friend, and makes virtual hosting work quite nicely; it acts like
Apache's mod_rewrite, and plays well witha Virtual Host Monster.
Instructions assume you use Zope with a VHM, and perhaps (might) want SSL
and load-balancing for ZEO.  Setup requirements: Squid, Zope, use of a VHM,
use of a redirector, and optionally OpenSSL if you want SSL accel.

Details:
1 - Prerequisites: OpenSSL; realpath and stat commands; Chris McDonough's
squidctl script (find on the squid-users mailing list); pyredir
2 - Go download Squid 2.5pre10 source; untar in a directory of your choice.
3 - ./configure --prefix=/usr/local/squid --enable-dlmalloc
--enable-pthreads --enable-storio=aufs
--with-openssl=/usr/lib/include/openssl --disable-internal-dns
--disable-identd-lookups --enable-ssl
	--> This is for Linux; you may want to adjust
		--enable-pthreads and use of aufs, since I think 
		you want to use diskd instead on BSD
	--> Change path to openssl include files
4 - make && make install
	--> Squid will install in /usr/local/squid ($PREVIX)
		--> Squid.conf will be in /usr/local/squid/etc
	--> Sometime between Squid 2.5pre5 and Squid2.5pre10, the dir layout
switched so that the squid binary is in $PREFIX/sbin
5 - Put the pyredir script in $PREFIX/libexec
6 - Put the pyredir.conf file in $PREFIX/etc
7 - Edit/hack pyredir:
	--> Disable redirector logging in the code if you have a
		big server (perhaps later, after this works?)
	--> Change path to log and pyredir.conf file
	--> Make sure all needed commands are supported (GET,POST,HEAD,etc)
8 - Put a VHM in the folder(s) you want to act as your host.
9 - Edit pyredir.conf:
	#this rule is for the public url mysite1.foo.com, which should 
	#be accessed via SSL on port 443 on Squid...
	#it says ^http:// becuase squid passes the URL to the redirector
	#with the https stripped off so it is more like a normal URL.
	#note: there is a Zope VHM in the MySite1 folder
	^http://mysite1.foo.com[/]?(.*)
=http://zopeserver:8080/MySite1/VirtualHostBase/https/mysite1.foo.com:443/My
Site1/VirtualHostRoot/\1
      #Note that this one is http (not https) in the rewrite rule passed
eventually to the virtual host monster
	^http://mysite2.foo.com[/]?(.*)
=http://zopeserver:8080/MySite2/VirtualHostBase/http/mysite2.foo.com:443/MyS
ite2/VirtualHostRoot/\1
10 - Add "zopeserver" to /etc/hosts
	--> if you have a ZEO cluster, add the name zopeserver for every IP
in your cluster; it will be round-robined
	--> Squid uses dnsserver (a resolver helper child program) for
/etc/hosts support since its internal resolver only works with DNS
		--> this is why we compiled with --disable-internal-dns
11 - Set up any ssl keys/certs you need with openssl.
12 - Setup squid:
http_port 80
#You will need to setup these keys with openssl first:
https_port 443 cert=/usr/local/squid/var/mysite1.foo.com_cert.pem
key=/usr/local/squid/var/mysite1.foo.com_key.pem
cache_dns_program /usr/local/squid/libexec/dnsserver
dns_children 5
redirect_program /usr/local/squid/libexec/pyredir
redirect_children 12
redirect_rewrites_host_header off
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_single_host off
httpd_accel_uses_host_header on
#obviously there are other things to set up, these 
#are the interesting/specific ones

13 - Download and setup the squidctl script in /usr/local/squid
	--> you will have to change the path to reflect that the squid
binary is in $PREFIX/sbin
	--> hack/adjust as necessary
14 - Chreate the cache dir; make sure permissions are okay for
nobody/nogroup. 
	--> Start squid; pay attention to any warnings about misconfig, and
fix
15 - Adjust:
	--> ACLs in squid.conf as needed.
	--> Pyredir rules in pyredir.conf
		'killall -s HUP pyredir' will reload rules without a squid
restart
	--> SSL config, if needed
16 - Once squid is going, make sure DNS or hosts on your client points to
your accelerator box, and try the URL...
17 - Once you have everything working, if you have a ZEO cluster, give
consideration to trying the ICP patches for Zope to allow for true
load-balancing by adding Zope servers to Squid's notion of its caching
fabric...
18 - Email me with questions if anything doesn't work.  

I plan on trying to turn these really rough notes into a howto at some point
on Zope.org, but I'm too busy at the moment.  Perhaps discussion here on the
list will help me with that process.

Questions, thoughts?

Sean

-----Original Message-----
From: Adam Getchell [mailto:AdamG@hrrm.ucdavis.edu]
Sent: Friday, August 23, 2002 3:51 PM
To: 'Charlie Reiman'; Marc Lindahl; Quentin Smith; Adam Getchell
Cc: zope@zope.org
Subject: RE: [Zope] Vhosting


Hello all,

Okay, so I read the links you gave me.

http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/VirtualHosting.s
tx
http://www.zope.org/Members/Jace/apache-vhm
http://www.zope.org/Members/bowerymarc/squid-zserver-virtual

Starting from scratch, I created a zope user on my OpenBSD box, with a home
directory of /home/zope. I then ran 

# /usr/local/bin/zope-instance /home/zope 
as root, to create an instance of Zope. 

# chmod -R zope:nobody /home/zope
# su zope
# /home/zope/start &
To start Zope as user zope. I then logged into the site, created my /site1
and /site2 folders, and created a Virtual Host monster with mappings of:

site1.ucdavis.edu/site1
site2.ucdavis.edu/site2

At this point, going to site1.ucdavis.edu goes to the main website, while
going to site1.ucdavis.edu:8080 goes to the subfolder. So I think my Squid
configuration is off.

I went through /etc/squid/squid.conf and have the following options enabled
(not commented out):

http_port 80
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_dir diskd /var/squid/cache 100 16 256
emulate_httpd_log on
auth_param basic children 5
auth_param basic real Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
httpd_accel_host my.ip.
httpd_accel_port 8080
httpd_accel_single_host on

I'm confused about the ACL's in step 4 of
http://www.zope.org/Members/bowerymarc/squid-zserver-virtual

I don't see an acl of type webserver in squid.conf ... So what should that
line be?

acl ext-ip-addr site1.ucdavis.edu www.xxx.yyy.zzz/255.255.255.255 

And then the ACL is ...?

Http_access allow MATCH ext-ip-addr

Clearly, the zope side of things are working so far. What is the
SiteAccessEnhanced used for? Do I still need a site root in each virtual
folder?

Many thanks,

--Adam



-----Original Message-----
From: Charlie Reiman [mailto:creiman@kefta.com] 
Sent: Friday, August 23, 2002 9:06 AM
To: Marc Lindahl; Quentin Smith; Adam Getchell
Cc: zope@zope.org
Subject: RE: [Zope] Vhosting


You should also read the 2.6 docs on VHM. They're downright tasty.

http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/VirtualHosting.s
tx

They apply to 2.5.1 as far as I noticed.

> -----Original Message-----
> From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of 
> Marc Lindahl
> Sent: Thursday, August 22, 2002 11:42 PM
> To: Quentin Smith; Adam Getchell
> Cc: 'zope@zope.org'
> Subject: Re: [Zope] Vhosting
>
>
> Try: http://www.zope.org/Members/bowerymarc/squid-zserver-virtual
>
>
> on 8/22/02 10:15 PM, Quentin Smith at quentins@comclub.dyndns.org
> scrivened:
>
> > Hi-
> > Use a Virtual Host Monster, one in the root of the site. Ideally, you
> > should put Apache or Pound in front of Zope instead of squid, but you
> > can use the Mappings tab of the Virtual Host Monster to define the
> > virtual hosts. I used to use SiteRoots for my site, and none of the
> > directions I found actually worked.
> > HTH,
> > --Quentin
> > On Thursday, August 22, 2002, at 08:27  PM, Adam Getchell wrote:
> >
> >> Hello all,
> >>
> >> I've read and re-read
> >> http://www.zope.org/Members/4am/SiteAccess2/vhosting a
> >> number of times.
> >>
> >> I have a webserver that needs to host a bunch of different sites,
> >> site1.ucdavis.edu, site2.ucdavis.edu, etc.
> >>
> >> I planned to use name based redirection, so I have c-names to the IP
> >> address
> >> of the server, i.e. site1.ucdavis.edu --> IP, site2.ucdavis.edu --> IP,
> >> etc.
> >>
> >> I installed on OpenBSD, so Zope runs as an instance in /home/zoperoot
> >> under
> >> port 8080, and Squid sits in front of it and redirects from 80 to port
> >> 8080.
> >> This part is working fine.
> >>
> >> Under my main zope site, I created two folders, site1 and site2. I then
> >> created, in each folder, an empty SiteRoot. That is, Title, Base, and
> >> Path
> >> are all blank. At least, that's how I interpreted the directions above.
> >>
> >> I then created a DTML method called host_redirector, with the following
> >> content:
> >>
> >> Extract the part of HTTP_HOST we care about, and declare our rewrite
> >> dictionary.
> >> <dtml-let hostname="_.string.upper(_.string.split(HTTP_HOST, '.')[0])"
> >>           sitemap="{'SITE1': 'site1',
> >>                     'SITE2': 'site2'}">
> >>     Do we have a match?
> >>     <dtml-if expr="sitemap.has_key(hostname)">
> >> Set the logical root: <dtml-call "REQUEST.set('SiteRootPATH', '/')">
> >> Add physical root: <dtml-call
> >> "REQUEST['TraversalRequestNameStack'].append(sitemap[hostname])">
> >>     </dtml-if>
> >> </dtml-let>
> >>
> >> Since I only care about the first part of the name, I rewrote
> the script
> >> from the example. It may be incorrect! I tried the equivalent at a
> >> python
> >> interpreter, but I may certainly have goofed up.
> >>
> >> In the root folder, I Set an Access rule to host_redirector.
> >>
> >> I changed the index pages in each of Site1 and Site2 to differentiate
> >> them.
> >> However, going to site1.ucdavis.edu gives the top level index_html,
> >> while
> >> going to site1.ucdavis.edu/site1 gives index_html in the Site1 folder,
> >> so
> >> clearly I did something wrong.
> >>
> >> Any pointers?
> >>
> >> ***************************
> >> *    Adam Getchell
> >> AdamG@hrrm.ucdavis.edu
> >> *    System Architect/Programmer            (530) 752-1584
> >> *    Human Resources Information Systems
> >> http://www.hr.ucdavis.edu/
> >> ***************************
> >> "Invincibility is in oneself, vulnerability in the opponent."
> -- Sun Tzu
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Zope maillist  -  Zope@zope.org
> >> http://lists.zope.org/mailman/listinfo/zope
> >> **   No cross posts or HTML encoding!  **
> >> (Related lists -
> >>  http://lists.zope.org/mailman/listinfo/zope-announce
> >>  http://lists.zope.org/mailman/listinfo/zope-dev )
> >>
> >
> >
> > _______________________________________________
> > Zope maillist  -  Zope@zope.org
> > http://lists.zope.org/mailman/listinfo/zope
> > **   No cross posts or HTML encoding!  **
> > (Related lists -
> > http://lists.zope.org/mailman/listinfo/zope-announce
> > http://lists.zope.org/mailman/listinfo/zope-dev )
> >
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>

_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )