This page has been accessed 2,813 times since the 30th October 2008.
| View this page in: |
English |
Any language: |
Chinese |
French |
German |
Japanese |
Portuguese |
Russian |
Spanish |
Translation to non-English languages provided by Google Language
You are connecting to the IPv4 version of this website from the IP address 38.107.191.106. You can try the IPv6-only version if you want.
|
|
[Written Autumn 2008] It's taken longer than anyone originally imagined, but
Virtual
Private Servers (VPSs) are finally becoming affordable by the average person
i.e. below US$100 a year. I recently bought one as an extension to nedprod, partially
as the main server for the book
Freeing Growth
but also to give me full control over my spam processing as my hosting provider
was getting sick & tired of me overloading his shared server. It was a fair cop
- but much of the problem was because I was limited to a normal user shell account.
With full root access I could do a lot more.
A VPS is basically a slice of a normal server, so you rent a bit of a full server with the size of that bit dependent on how much you are willing to pay. You get your own full OS installation (which is usually Linux for the cheap VPSs) and you can do anything with them that you like. You can get VPSs up to half the size of a normal server but they cost more than I earn in a year, so realistically I'm in the £5 a month category which doesn't buy you much. With a decent amount of tuning though you would be surprised what you can achieve, and of course you can easily convert your VPS upwards or downwards according to demand. 1. Find a decent VPS providerThis is very easy: go to Low End Box and read through the reviews. Overwhelmingly the US based VPSs are cheapest, then Germany, then the UK - however, remember that you will be spending most of your time in a SSH session so ping times are important unless you like long lags between keypresses. You also get a choice between OpenVZ and Zen - OpenVZ can be better if the sysadmin is competent and doesn't oversell the box, whereas Xen is less performant but much more predictable. I went with buying a full year in advance from cheapvps.co.uk which is the unmanaged arm of the respectable VAServ VPS hosting company. If you buy a full year at once, they will double your VPS spec which meant that I got a UK-based Xen-based 256Mb RAM, 20Gb HD and 300Gb of bandwidth VPS for just £4.75 a month (US$9). It being UK based got me 65ms ping times from Ireland which makes typing much easier, plus I chose Ubuntu server for its OS. With cheapvps.co.uk you actually can choose between OS installations any time you like and there is quite a long list of alternatives including Windows Server 2003. I personally would recommend with hindsight that you need a minimum of 256Mb of RAM unless you are willing to run really ancient server software and/or you don't mind a sluggish web service. As you will see later on, I aim for 128Mb to be in use at any given time with a maximum of 192Mb in burst - this leaves 64-128Mb for caching disc access which you really want if you can, and remember that there will be a spike in memory usage every time a SSL connection is formed. If you approach your RAM slice then your VPS will start swapping and things will slow right down. However, if you can only afford 128Mb of RAM, then my following instructions will at least get you a working server. [Added Summer 2009: If you need something with 1Gb of RAM or more, low end fully dedicated servers are finally becoming affordable - you can get one from €30/month. See page three of this guide] By the end of this, you should have a web server capable of running any PHP and MySQL based CMS, plus the email service will have excellent spam filtering. 2. Free up RAMFor some strange reason most VPSs will preinstall a 64 bit Linux OS. If you possibly possibly can, switch this to a 32 bit installation as 32 bit binaries are significantly smaller and use less memory during use. Chances are that your VPS will have no shortage of bandwidth, CPU nor disk space - its single & biggest bottleneck will be RAM. Even if you get your server running smoothly close to its RAM limit, wait till your first DDoS or bot attack and watch those memory spikes! You can replace openssh with dropbear to save a few Mb of memory - it does no harm and it's not like you need the full openssh. Simply apt-get install dropbear and apt-get remove ssh and then apt-get autoremove to clean up. Also, disable a lot of the default tty processes which hang around eating up half a megabyte each - simply rename /etc/event.d/ttyX to something like /etc/ttyX.off. Don't forget to add universe and hardy-proposed to /etc/apt/sources.lst and do an upgrade. If you're really short of RAM, replace bash with dash or even busybox though you may get shell script incompatibilities. I didn't bother with the latter two and ended up with the following memory free & processes: top - 21:41:50 up 1 min, 1 user, load average: 0.69, 0.33, 0.12 Tasks: 32 total, 2 running, 30 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 262144k total, 41616k used, 220528k free, 2096k buffers Swap: 0k total, 0k used, 0k free, 9728k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 971 root 15 0 2688 1536 1216 S 0.0 0.6 0:00.00 bash 970 root 15 0 2272 1204 784 R 0.0 0.5 0:00.08 dropbear 929 klog 24 0 2164 1132 400 S 0.0 0.4 0:00.01 klogd 984 root 15 0 2148 1032 828 R 0.0 0.4 0:00.00 top 1 root 17 0 1880 964 524 S 0.0 0.4 0:01.61 init 906 syslog 16 0 1776 632 496 S 0.0 0.2 0:00.00 syslogd 358 root 12 -4 2064 628 364 S 0.0 0.2 0:00.00 udevd 927 root 18 0 1708 516 420 S 0.0 0.2 0:00.01 dd 967 root 18 0 1556 508 436 S 0.0 0.2 0:00.00 getty 869 root 16 0 1556 504 436 S 0.0 0.2 0:00.00 getty 966 root 18 0 1552 504 436 S 0.0 0.2 0:00.00 getty 941 root 18 0 1960 468 364 S 0.0 0.2 0:00.00 dropbear 29,792Kb (~30Mb) used is pretty damn good for Ubuntu v8.04 I think :) 3. Replace rootFirstly, create your own user account (preferably with a random bit on the end: the technical name for this technique is salting) and add it either to /etc/sudoers or the admin group. Check that it logs in and that you can sudo as root. Now disable root login in dropbear by adding the -w switch to /etc/default/dropbear extra args. You probably should install ssh keys and disable password logins, but to be honest I have a long randomised alphanumeric password which is good enough in my opinion. 4. Install MySQL + Lighttpd + PHPApache is fine for big fast servers but it sucks for small ones. You must use Lighttpd instead which uses threads instead of forks to handle concurrent connections. You should also disable InnoDB from MySQL BEFORE doing anything else as that will save 100Mb of RAM. You can find RAM tuning guides all over the internet, but basically I ran lighty-enable-mod fastcgi and then I edited /etc/lighttpd/conf-enabled/10-fastcgi.conf: "max-procs" => 1, "PHP_FCGI_CHILDREN" => "4"This basically prevents PHP from handling more than four connections at once - any extra users just have to wait. A PHP child can consume about 10-16Mb of RAM each so you can increase PHP_FCGI_CHILDREN if needs be, but for most low traffic sites this will be fine. You should never increase max-procs past two - it wastes memory by having an extra master php process, on the other hand it adds resiliency should one of the master php processes die. Chances are though you wouldn't try running a heavy user site on a VPS anyway. 4. Secure ALL admin parts of the siteYou don't want to get hacked, so you will need to be paranoid about security - particularly when you are installing, configuring and/or playing around with your website. PHP based admin tools such as phpmyadmin are very convenient but are deadly if they get broken into - you don't even want squirrelmail broken in reality. To ensure full security, put auth digest (not the older auth plain) password blocks around every sensitive area of the website - this requires a user name and password to access. You should also always password protect an entire site if it may be in anyway vulnerable. Also, force the use of HTTPS for all the same sensitive regions - you don't want your emails nor your SQL transactions being readable if you use them to add passwords etc. It took me a long time to figure out the lighttpd.conf to make these work, so here's mine for reference: # Debian lighttpd configuration file
#
############ Options you really have to take care of ####################
## modules to load
# mod_access, mod_accesslog and mod_alias are loaded by default
# all other module should only be loaded if neccesary
# - saves some time
# - saves memory
server.modules = (
"mod_access",
"mod_alias",
"mod_accesslog",
"mod_compress",
"mod_evhost",
"mod_redirect",
# "mod_rewrite",
# "mod_status",
# "mod_evhost",
# "mod_usertrack",
# "mod_rrdtool",
# "mod_webdav",
# "mod_expire",
# "mod_flv_streaming",
# "mod_evasive"
)
## a static document-root, for virtual-hosting take look at the
## server.virtual-* options
server.document-root = "/var/www/default/html"
## where to send error-messages to
server.errorlog = "/var/log/lighttpd/error.log"
## files to check for if .../ is requested
index-file.names = ( "index.php", "index.html",
"index.htm", "default.htm",
"index.lighttpd.html" )
## Use the "Content-Type" extended attribute to obtain mime type if possible
# mimetype.use-xattr = "enable"
#### accesslog module
accesslog.filename = "/var/log/lighttpd/access.log"
## deny access the file-extensions
#
# ~ is for backupfiles from vi, emacs, joe, ...
# .inc is often used for code includes which should in general not be part
# of the document-root
url.access-deny = ( "~", ".inc" )
##
# which extensions should not be handle via static-file transfer
#
# .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
######### Options that are good to be but not neccesary to be changed #######
## bind to port (default: 80)
# server.port = 81
## bind to localhost only (default: all interfaces)
## server.bind = "localhost"
## error-handler for status 404
#server.error-handler-404 = "/error-handler.html"
#server.error-handler-404 = "/error-handler.php"
server.error-handler-404 = "/error404.html"
## to help the rc.scripts
server.pid-file = "/var/run/lighttpd.pid"
##
## Format: <errorfile-prefix><status>.html
## -> ..../status-404.html for 'File not found'
#server.errorfile-prefix = "/var/www/"
## virtual directory listings
dir-listing.encoding = "utf-8"
server.dir-listing = "enable"
## send unhandled HTTP-header headers to error-log
#debug.dump-unknown-headers = "enable"
### only root can use these options
#
# chroot() to directory (default: no chroot() )
#server.chroot = "/"
## change uid to <uid> (default: don't care)
server.username = "www-data"
## change uid to <uid> (default: don't care)
server.groupname = "www-data"
#### compress module
compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ("text/plain", "text/html", "application/x-javascript", "text/css")
#### status module
# status.status-url = "/server-status"
# status.config-url = "/server-config"
#### url handling modules (rewrite, redirect, access)
# url.rewrite = ( "^/$" => "/server-status" )
# url.redirect = ( "^/wishlist/(.+)" => "http://www.123.org/$1" )
#
# define a pattern for the host url finding
# %% => % sign
# %0 => domain name + tld
# %1 => tld
# %2 => domain name without tld
# %3 => subdomain 1 name
# %4 => subdomain 2 name
#
# evhost.path-pattern = "/home/storage/dev/www/%3/htdocs/"
evhost.path-pattern = "/var/www/%0/html/"
#### expire module
# expire.url = ( "/buggy/" => "access 2 hours", "/asdhas/" => "access plus 1 seconds 2 minutes")
#### rrdtool
# rrdtool.binary = "/usr/bin/rrdtool"
# rrdtool.db-name = "/var/www/lighttpd.rrd"
#### variable usage:
## variable name without "." is auto prefixed by "var." and becomes "var.bar"
#bar = 1
#var.mystring = "foo"
## integer add
#bar += 1
## string concat, with integer cast as string, result: "www.foo1.com"
#server.name = "www." + mystring + var.bar + ".com"
## array merge
#index-file.names = (foo + ".php") + index-file.names
#index-file.names += (foo + ".php")
#### external configuration files
## mimetype mapping
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
## load enabled configuration files,
## read /etc/lighttpd/conf-available/README first
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
#### handle Debian Policy Manual, Section 11.5. urls
### by default allow them only from localhost
### (This must come last due to #445459)
$HTTP["remoteip"] == "127.0.0.1" {
alias.url += (
"/doc/" => "/usr/share/doc/",
"/images/" => "/usr/share/images/"
)
$HTTP["url"] =~ "^/doc/|^/images/" {
dir-listing.activate = "enable"
}
}
# Enable SSL
$SERVER["socket"] == ":443" {
ssl.engine = "enable"
ssl.pemfile = "/etc/lighttpd/ssl/vps.nedprod.com/server.pem"
#ssl.ca-file = "/etc/lighttpd/theos.in/CA_issuing.crt"
}
# Force SSL on vps.nedprod.com
$SERVER["socket"] == ":80" {
$HTTP["host"] =~ "vps\.nedprod\.com" {
url.redirect = ( "^/(.*)" => "https://vps.nedprod.com/$1" )
server.name = "vps.nedprod.com"
}
}
# Force phpmyadmin to vps
$HTTP["url"] =~ "/phpmyadmin/" {
$HTTP["host"] !~ "vps.nedprod.com" {
url.redirect = ( "^/(.*)" => "https://vps.nedprod.com/phpmyadmin/")
server.name = "vps.nedprod.com"
}
}
# Make stuff auth protected
auth.backend = "htdigest"
auth.backend.htdigest.userfile = "/etc/lighttpd/lighttpd-htdigest.user"
auth.require = ( "/phpmyadmin/" =>
( "method" => "digest",
"realm" => "admin",
"require" => "user=ned"
),
"/squirrelmail/" =>
( "method" => "digest",
"realm" => "admin",
"require" => "user=ned"
)
)
# Password protect neocapitalism.org
$HTTP["host"] =~ "^(www\.)?neocapitalism\.org$" {
auth.require = ( "/" =>
( "method" => "digest",
"realm" => "admin",
"require" => "user=ned"
)
)
}
Among the many things which this config does, it rewrites any accesses to http://vps.nedprod.com to use https: instead. It also redirects any usage of phpmyadmin in the other virtual domains (as the ubuntu package overlays itself on all virtual domains) to use the vps one. And it sticks auth digest blocks round stuff. After all of that, here's my memory usage on a reasonably loaded server: total used free shared buffers cached Mem: 262144 257672 4472 0 9608 189216 -/+ buffers/cache: 58848 203296 Swap: 0 0 0 Total: 262144 257672 4472 So that's about 58Mb used at this stage - MySQL (sans InnoDB), lighttpd plus four php5-cgi processes eat up 28Mb of RAM. 5. Setup Firewall + Email + SpamassassinThe best thing to do is to follow the excellent guide at http://flurdy.com/docs/postfix/ which covers setting up Shorewall, Postfix, Courier IMAP and integrating both into MySQL. You need to skip the Amavisd and ClamAV steps however because they chew up 170Mb alone and will reduce your poor server to a crawl. Be careful to use Shorewall to block off public access to anything until you are absolutely sure it is secure. Instead, chain in spamassassin manually as follows. Edit /etc/postfix/master.cf: #
# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - y - 1 smtpd
-o content_filter=spamassassin
#submission inet n - - - - smtpd
# -o smtpd_tls_security_level=encrypt
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#smtps inet n - - - - smtpd
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#628 inet n - - - - qmqpd
pickup fifo n - - 60 1 pickup
cleanup unix n - - - 0 cleanup
qmgr fifo n - n 300 1 qmgr
#qmgr fifo n - - 300 1 oqmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - - - - smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay unix - - - - - smtp
-o smtp_fallback_relay=
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# See the Postfix UUCP_README file for configuration details.
#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail unix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2 pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
${nexthop} ${user}
spamassassin unix - n n - 1 pipe
user=spamd argv=/usr/bin/spamc -f -e
/usr/sbin/sendmail -oi -f ${sender} ${recipient}
Edit /etc/default/spamassassin to make sure you set up spamd to run under the spamd user rather than as root and limit its children to one seeing as each child will chew up 50Mb or so. Spamassassin alone eats 30% of my 256Mb VPS - it's the single largest memory hog by far. Run sa-compile after every sa-update to help reduce memory usage and improve throughput, and don't forget to enable loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody in v320.pre to enable the compiled rules. After all of that we get the following major memory hogs: top - 20:25:49 up 2 days, 21:07, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 65 total, 2 running, 63 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.3%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 262144k total, 258992k used, 3152k free, 6416k buffers Swap: 0k total, 0k used, 0k free, 130656k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10103 spamd 15 0 32036 28m 2176 S 0.0 11.2 0:00.09 spamd 10102 root 15 0 31296 28m 2328 S 0.0 11.0 0:00.71 spamd 1962 www-data 15 0 21004 10m 4152 S 0.0 4.1 0:06.34 php-cgi 1886 mysql 15 0 55660 8064 4580 S 0.0 3.1 0:00.23 mysqld 11878 postgrey 18 0 10976 7968 2444 S 0.0 3.0 0:00.02 postgrey 1964 www-data 15 0 18172 7904 4112 S 0.0 3.0 0:02.70 php-cgi 1961 www-data 16 0 17724 7500 4140 S 0.0 2.9 0:03.08 php-cgi 1963 www-data 16 0 17308 7052 4124 S 0.0 2.7 0:03.97 php-cgi 1960 www-data 25 0 16448 4960 3248 S 0.0 1.9 0:00.00 php-cgi 1957 www-data 15 0 6676 2868 1400 S 0.0 1.1 0:00.06 lighttpd 10347 postfix 15 0 5600 2476 1808 S 0.0 0.9 0:00.00 tlsmgr 9134 postfix 15 0 5276 1784 1444 S 0.0 0.7 0:00.00 qmgr And we have the follow memory free: total used free shared buffers cached Mem: 262144 259044 3100 0 6504 130664 -/+ buffers/cache: 121876 140268 Swap: 0 0 0 Total: 262144 259044 3100 You can see the hefty effect of spamassassin - we have roughly doubled our memory usage to ~120Mb which is as far as we want to go if we want to keep our 256Mb server running smoothly. 6. Setup Reverse DNS + SPFNot setting up your reverse DNS will penalise any email you send from your VPS by spam checkers, so here's how. Firstly, most VPSs don't have DNS delegated to them so there is no point setting up your own bind or npd server unless you want a local caching nameserver (not a lot of point seeing as your VPS provider has a perfectly good equivalent nearby) - therefore, you should ask your provider to add the rDNS for your IP, or else use a DNS provider which lets you have access to your host records (e.g. the very reputable though slightly more expensive namecheap.com). For SPF you need to add an IN TXT record to your DNS record detailing every server which could send email for your domain like as follows: "v=spf1 a mx include:ukfsn.org ~all" The include: is for when you may send your email from your ISP's smtp server instead e.g. if you don't set up your SMTP service on your VPS to send email for you. 7a. Content ManagementI have spent the last two months looking for a decent content management system and I have installed over ten different CMSs and database driven systems. I was looking mainly for an integrated forum, wiki, issue tracker and database-driven display plus automatic generation of search engine friendly data as well as disability support and any other easy freebies XML enables. This website, nedprod.com, still runs along a static HTML data system whereby I edit the content on my computer, munge it through some Python scripting and upload it via SSH. nedprod's pages do contain PHP to implement little things like the hit counter and automatic page compression (this halved my bandwidth requirements instantly) but all in all, nedprod's architecture would be entirely familiar to someone in the mid 1990s. I'm happy with this system - it works fine for me and gives me the kind of absolute control that I like without being needlessly annoying on a daily basis. However, Google takes special notice of the main CMSs nowadays and it doesn't understand nedprod's data format. Besides, it was about time I modernised my web programming knowledge - I can't live in the 1990s forever! What follows are only the ones I tried which I thought anything of - as you'll notice, they all have the reputation for being very lean and fast (I didn't even consider them unless they had a reputation for speed). Obviously, I didn't spent a massive amount of time on each CMS so you can take my opinion at the time of writing (Autumn 2008) for what it's worth:
Obviously I was getting a bit annoyed by this stage - everything I tried was good in one or two specific areas, but fairly shocking in everything else and I wanted a good performer in lots of areas at once: a one size fits all solution. For example, Drupal is still using a node based layout and page addressing which makes all the URLs some random node number rather than a URL which describes the page content. I had thought that CMSs had moved on by now, though I guess the fact we still need sitemaps.xml would suggest otherwise. If you're happy with PHP based sites, then great you'll be very happy at this stage - your VPS will handle even fairly heavy loads with ease even without adding a static proxy. Even if you prefer Ruby on Rails, you have ample memory to spare to run two concurrent Rails processes and your VPS will scale moderately well though it'll scale better with multiple logged in users (which are much harder to cache) on PHP. However, I wasn't happy at this stage and I began to wonder what else I could do ... Read all about setting up Plone & Zope to run on a low-end VPS |