Introduction

Why muck around with HAProxy and Varnish when you can have nginx do it all for you. The setup is easy and it's a lot easier to maintain.

Installing nginx with buildout

You can setup nginx fairly easily with buildout. The only fancy part of our setup is that we're going to include the nginx cache purge module.

1. Add the cache purge part to your buildout
   

   [ngx_cache_purge]
   recipe = hexagonit.recipe.download
   url = http://labs.frickle.com/files/ngx_cache_purge-1.1.tar.gz
   strip-top-level-dir = true

2. I needed the pcre source to compile also
   

   [pcre-source]
   recipe = hexagonit.recipe.download
   url = ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.00.tar.gz
   strip-top-level-dir = true
   

3. and the nginx part
   

   [nginx-build]
   recipe = hexagonit.recipe.cmmi
   url = http://nginx.org/download/nginx-0.8.45.tar.gz
   configure-options =
       --with-http_stub_status_module
       --conf-path=${buildout:directory}/settings/nginx.conf
       --error-log-path=${buildout:directory}/var/log/nginx-error.log
       --pid-path=${buildout:directory}/var/nginx.pid
       --lock-path=${buildout:directory}/var/nginx.lock
       --with-pcre=${pcre-source:location}
       --with-http_ssl_module
       --add-module=${ngx_cache_purge:location}
   

4. and finally, add it all to the parts directive
   

   parts =
       ...
       pcre-source
       ngx_cache_purge
       nginx-build
       ...
   
   

5. After you re-run buildout, you'll be able to run nginx by issuing a command like this:
   

   ./parts/nginx/sbin/nginx -c /path/to/configuration/nginx.conf

nginx configuration

Here is a simple sample configuration for nginx configured with load balancing and caching. It can obvious get as complicated as you want it, but I definitely think this is easier than managing haproxy, varnish and nginx.

pid /path/to/buildout/var/nginx.pid;
lock_file /path/to/buildout/var/nginx.lock;

worker_processes 2;
daemon off;

events {
    worker_connections 1024;
}

error_log /path/to/buildout/var/log/nginx-error.log warn;

# HTTP server

http {
    server_names_hash_bucket_size 64;

    # this is how you do simple round robin load balancing with nginx.
	# you can define as many backup servers as you'd like here.
    upstream plone {
        server 127.0.0.1:8080;
        server 127.0.0.1:8081;
    }
    
    access_log /path/to/buildout/var/log/main-access.log;


 	# can specify multiple cache paths for different resources/paths/proxies
    # if needed..
    # the levels=1:2 just means it'll store the cache'd files 2 levels down in
    # the folder structure
    proxy_cache_path  /var/www/cache  levels=1:2 keys_zone=thecache:100m max_size=1000m inactive=600m;
    proxy_temp_path /var/www/cache/tmp;

    # Here is the caching purge handling. Purge request come in here
    server {
        listen 8089;
        server_name www.example.com;

        access_log /path/to/buildout/var/log/purge.log;

        location / {
      	  allow			127.0.0.1;
      	  deny			all;
      	  proxy_cache_purge thecache $scheme$proxy_host$request_uri;
        }
    }
    
    server {
        listen 80;
        server_name www.example.com;


		# log for cache hits.
		log_format cache '***$time_local '
		                             '$upstream_cache_status '
		                             'Cache-Control: $upstream_http_cache_control '
		                             'Expires: $upstream_http_expires '
		                             '"$request" ($status) '
		                             '"$http_user_agent" ';

		access_log /path/to/buildout/var/log/cache.log cache;

        # Enable gzip compression
        gzip             on;
        gzip_min_length  1000;
        gzip_proxied     any;
        gzip_types       text/xml text/plain application/xml;

        # Show status information on /_main-status
        location = /_main_status_ {
            stub_status on;
            allow 127.0.0.1;
            deny all;
        }

		# do not cache when users are logged in..
        proxy_cache_bypass $cookie___ac;

	    location / {
	        proxy_redirect                  off;
			proxy_set_header                Host $host;
			proxy_set_header                X-Real-IP $remote_addr;
			proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
			client_max_body_size            0;
			client_body_buffer_size         128k;
			proxy_send_timeout              120;
			proxy_buffer_size               4k;
			proxy_buffers                   4 32k;
			proxy_busy_buffers_size         64k;
			proxy_temp_file_write_size      64k;
			proxy_connect_timeout           75;
			proxy_read_timeout              205;   
	        http://plone/VirtualHostBase/http/www.example.com:80/Plone	/VirtualHostRoot/;

	        proxy_cache_bypass $cookie___ac;
	        proxy_cache thecache;
	        proxy_cache_key $scheme$proxy_host$request_uri;
	    }
    }
}

Overview

Sometimes grep or TextMate's find can take a long time so I decided to give a mini search engine a try using repoze.bfg, repoze.catalog and Pygments. The result is that it works rather well and is very fast. The catalog ends up being rather large once you add groups of packages from Plone and such so this ends up making the RAM usage be very high unfortunately.

It gives you the ability to easily search methods, classes, filenames and full text and then give allows you to view the file with pygments.

You can take a look at search.nathanvangheem.com to see it working with repoze.bfg and Plone 4 being indexed.

You can checkout the pypi page if you're interested in it and I also have it on bitbucket.

Update

Tres has managed to merge his WSGI branch into trunk and Hanno tells me the unofficial plan is to include this in a release for Zope 2.13, in time for Plone 4.1. This is not decided upon yet though.

Overview

I was planning on implementing WSGI for Zope2 during the Penn State Symposium Sprints, and I did a few things to help out; however, Tres Seaver did most of the work on his own before and during part of the Sprint :) The rest of the time I spent just testing it out and helping with the Theme Editor sprint.

Now, it's really quite trivial to get it working now and makes all of the repoze.zope2 nonsense unneeded now. This article is just here for a reference if anyone else is interested in getting it going on their setup.

Guide

This guide assumes you have an existing Plone 4 installation to work from. I don't provide any buildouts here--just modifying an existing buildout to make it work with a branch of Zope2 and creating an ini that Paste can consume to serve WSGI.

Supplying the WSGI'd Zope2

First off, go to the src directory of the installation. If you've installed using the unified installer, that will bin in instance-home/zinstance/src or if you just used straight buildout, it'll be in instance/src. Then checkout the Zope2 branch:

svn co http://svn.zope.org/repos/main/Zope/branches/tseaver-fix_wsgi/ Zope2

Stringing up buildout

Next thing you'll need to do is modify your buildout.cfg file to add the checked out Zope 2 to the develop section:

develop = 
  ...
  Zope2
  ...

Still modifying your buildout.cfg, add Paste, PasteScript, repoze.tm2 and repoze.retry to your eggs section:

eggs =
  Plone
  Paste
  PasteScript
  repoze.tm2
  repoze.retry 

Again, editing your buildout.cfg, add a paster part for the paster script:

parts =
  ...
  paster
  ...

[paster]
recipe = repoze.recipe.egg
scripts = paster
eggs = ${instance:eggs}

Then, you'll need to add the updated Zope2 versions for the WSGI branch. To do this, basically, just add the versions.cfg file provided in the branch after every other version file listed in the extends directive. It'll look like this:

extends =
  ...
  src/Zope2/versions.cfg

Then run your buildout like normal:

./bin/buildout

Creating a WSGI Configuration File

You'll now need to create a WSGI configuration file. Right now, we'll just server it using the Paste server and wsgi ini configuration way. You can also do this to string up Apache's WSGI implementation but that is beyond the scope of this article.

Create a file in the instance directory called, zope2.ini with the contents of:

[app:zope]
use = egg:Zope2#main
zope_conf = %(here)s/parts/instance/etc/zope.conf

[pipeline:main]
pipeline =
    egg:paste#evalerror
    egg:repoze.retry#retry
    egg:repoze.tm2#tm
    zope

[server:main]
use = egg:paste#http
host = localhost
port = 8080

The zope_conf value in the app:zope section can be the path to any zope.conf file. I'm just exampling the standard location of it and not going through the configuration of that file itself.

Fire it up!

If all went well, you should now be able to start up your Plone 4 instance on WSGI like this:

./bin/paster serve zope2.ini

Your server should now be able to visit your site on http://localhost:8080

Caveats

I did run into a snag with the Mac OS X unified installer and the version of python it has configured. Basically, it wouldn't compile the Zope2 dependencies so I had to use my own version of python that I had compiled with the python buildout found in the plone collective svn. The bug is sort of referenced in the zope bug tracker.

Future Considerations

I'm hoping to maybe get a release with this branch implementation out--maybe as an alpha or beta release since I don't think they are planning on merging this to core any time soon; although, I really have no understanding of what that whole process is.

I'd like to see the Zope2 package implement mkzope2instance and other convenience methods so it'd be possible to install Plone 4/Zope2 without buildout at all maybe using pip. I'm looking into how this might be able to happen with a pip versions file and other things. Maybe more on this later.

Post any comments if you run into any issues.

Last night I decided to take some time to move this site over to Plone 4. The repoze.zope2 build I had previously was buggy and I am starting to dislike Deliverance a bit.

Snags

• the search didn't work because of this migration issue http://dev.plone.org/plone/ticket/10360
• old file types aren't migrated to blob http://dev.plone.org/plone/ticket/10365
• archetypes don't want to use the new add views--should probably report this
• If you have the jquery UI package installed, it'll mess with some of Plone's JS for some reason--I just disabled it.
• some other control panel entries and menu items were not fully migrated, still pointing to old templates

Benefits

• FAST--running this on a linode with 512MB of ram. 2 zeo clients and with CacheSetup, this thing is very fast. Much faster than Plone 3.
• Just more polished than Plone 3--with usability enhancements and a new theme, it's great.
• I haven't found a product that wasn't compatible with Plone 4 yet--although, sometimes I needed to do some digging or use an svn checkout to get the compatible version

I was having issues with my repoze.zope2 and Deliverance setup locking up on me. Turns out, since I had some of sub-requests in my Deliverance rules, it was occasionally causing a race condition where all the wsgi server threads were taken up and an existing thread was making a sub-request.

The fix is to remove all sub-requests in the deliverance rules file if possible, add more threads threads in the repoze.zope2 wsgi server configuration, or use a different wsgi server that doesn't restrict the amount of threads--so it'll add a new thread if needed. A better solution, IMO, would be to just have a different server, serve the theme files. Anything to make it not make sub-requests against the same wsgi server.

For some very odd reason, boxee did not include an eject button in their version of the app. Luckily it is pretty easy to add your own eject button. Well, looking back it is easy; however, it took me a while to figure it out...

First off, find a nice fancy eject button and place it into boxee's skin images folder, "skin/boxee/media/icons/". I used this one and found it on google image search.

Next insert these lines after line number 231 of the file skin/boxee/720p/boxee_main_menu.xml

<item>
    <visible>System.HasMediadvd</visible>
    <controlid>1300</controlid>
    <thumb>icons/eject.png</thumb>
    <label>Eject DVD</label>
    <onclick>XBMC.EjectTray()</onclick>
</item>

And you're set. You should now have an eject button for your DVDs.

I like to make my work environments contained and repeatable. Buildout is tool that can help you do that. While buildout is a python tool, with buildout recipes you can compile and install other types of applications relatively easily. 

You can download my buildout here. It also contains patches for it to work on Mac OS X. You'll need python installed for this to work.

If you're not familiar with buildout, these are the steps to get it working after you've extracted it...

python bootstrap.py
./bin/buildout

Then wait for a long time. Then read the README.txt for instructions on how to configure and start apache

First off

collect2: ld returned 1 exit status
lipo: can't open input file: /var/folders/nO/nOf0lGB+GlGrFGnhcyaFPE+++TI/-Tmp-//ccqHwR4o.out (No such file or directory)
error: Setup script exited with error: command 'gcc' failed with exit status 1

Then it's failing because it doesn't have the universal versions of some packages installed. 

You'll first need to install MacPorts. You can download the dmg install file or just go to the MacPorts installation page for instructions on how to get it setup on your system.

Installing Universal Packages

sudo port install libiconv +universal
sudo port upgrade --enforce-variants zlib +universal
sudo port install jpeg +universal
sudo port install freetype2	 +universal

Now Install PIL 

./bin/easy_install http://dist.repoze.org/PIL-1.1.6.tar.gz

 This is a version of PIL that plays nice with setuptools.

While moving all my photos from flickr over to Picasa Web Albums I ran into some problems. The original post explained how the whole process works. 

• Picasa albums have 1000 photos limits
• the python gdata api didn't support video uploads
• Picasa trimmed whitespace from Album names
• other small things