We find that simple systems are easier to build and understand, which make them easier to change, so with that in mind we are always striving to make things as simple as they can be, without being simplistic.  Our setup is a fairly standard in modern web applications with a web server that receives client’s requests then passes those requests back to either the file-system (if static assets are requested) or to the application server which in turn pulls data from a database, a search engine or the file system.

The Bits And Pieces

• The Web Server (Nginx) – (pronounced as “engine X”) is a light weight, high performance web server/reverse proxy. We chose Nginx for it’s light footprint and easy configuration.  It has served us very well and hasn’t got in the way of anything we’ve needed to do. If you are looking for a web server and you’d like to try something beside Apache’s httpd then Nginx is a great alternative.
• The App Server (A Bunch Of Mongrels) – Mongrel is our application server.  If you are from a Java world you can think of it as Apache’s Tomcat for Ruby.  We use Mongrel to host our Rails application and it does a fine job.  Mongrel was the default deployment option for Rails until the original author, Zed Shaw, decided he didn’t want to support it any more and a competing technology, Phusion’s Passenger, came along.  Passenger is the “preferred deployment setup for Rails” – it is under active development but doesn’t really offer us enough to switch from Mongrel.  We’ll probably swap over to it, but we’re in no rush.
• The Meat And Bones (Ruby On Rails) – We are using the Ruby on Rails framework to build the business logic for the site.  So, has it changed our delivery patterns? Yes, it’s made us faster. Some of it is probably due to the fact that we are a lab/test bed group, but RoR has certainly been a major contributing factor to allow us to get features into ‘production’ more quickly (there are some stories and learnings here, but we can explore that some other time).  Ruby is a fantastic language to build web applications with and Rails leverages Ruby’s language features pretty well.
• The Database (PostgreSQL) – PostgreSql is a mature, open source database with clear licensing.  If you haven’t looked at it, then you should.  We use PostgreSql to store business listing data and user generated content (we are sooo web 2.0).  Fairly straight forwards stuff.  We also use a PostgreSql GIS extension, PostGIS, to generate reference data for suburbs that we use during our search.
• How We Find Stuff (FAST ESP) – All Sensis properties use FAST ESP (bought by Microsoft last year) for their search requirements. Being a test bed for new functionality (of which search is a major component), we use it too.  In the open source world Solr (build on Lucene), Sphinx or Xapian may fill the same requirements. We make heavy use of FAST and many of it’s feature and there is much we can say about it as well as all the things we have learned about search from using it, but we’re trying to keep this short and sweet, so moving right along.

That’s our logical design, but physically we have two servers.

Why Two Machines?

Notice that one of our machine is completely devoted to FAST (our search product). While we could use just a single server for delivering search results we need the grunt power of a second machine when we “ingest” our data (load it) into FAST.

If you have a comment or suggestion about our setup or simply want to brag about showcase your own, please leave a comment, we’d love to hear from you.