The Saltmine Chronicles

Mistah Wheelus requested I write an explanation we could give to our customers about greylisting. And here's my attempt at being lucid.

Greylisting is an easy and effective (so far) anti-spam technique (our current tests show an effective spam-stopping rate of over 97%) but before I explain how it works, I must first explain a bit how the email system works.

Once you click “send” your computer will connect with an outgoing email server (this is the “Outgoing SMTP Server” setting in the configuration of your email account); it then identifies itself to the outgoing email server, sends your email address (technically, the “sender address”), then the email address of who you are sending it to (the “recipient address”) and then finally, the actual email (which may have nothing to do with either the sender or recipient email address, a fact that spammers often exploit for fun and profit).

Once the outgoing email server has accepted the email, it is then queued up for final delivery.

The outgoing email server will then look up where to send the email based upon the recipient's domain name, and once this is done, connects to an incoming email server that handles email for the recipient, and using SMTP, deliver the email to the recipient's email box. And if for any reason the email can't be delivered, or there's an error during the delivery, the outgoing email server queues up the email for another attempt at a later time (which can be a few minutes to maybe an hour later). And this is an important detail to remember.

Once the email has been successfully delivered, it's then saved in the recpients incoming email box, which stays there until the recipient retrieves the email (which is beyond the scope of this entry).

Now, how does Greylisting fit into all of this?

Greylisting works on the recipient side of this. Send me an email, and eventually, some server from your end (“your server”) will contact the server on my end (“my server”) to deliver the email. My server then has three pieces of information: the IP address of your server, your email address (assuming it matches the sender address) and my email address. And for the sake of an example, let's say it's [ 3.4.5.6 , fred@example.net , sean@pickint.net ]. My server will see if it has seen that particular combination before, and if not, record it, and send back to your server “try again later.” And until it's been at least 25 minutes since I first saw that particular combination, my server will keep sending back “try again later.”

After the initial 25 minutes, any email from 3.4.5.6 with sender fred@example.net and recipient sean@pickint.net will be accepted. But other emails from 3.4.5.6 can still experience the delay, if the sender email addresss, recipient email address, or both, are different. It's the combination of all three pieces of information that have to match.

Basically, greylisting delays an initial email by some period of time, only “whitelisting” it after a delay period. And while it seems strange, that simple strategy can easily filter out 97% of all spam, since most spammers don't want to bother with redelivery of non-delivered email. They're trying to get their spam out as fast as possible. Attempting to redeliver their spam will only complicate things on their end.

And it's this delay that causes the biggest complaints. But the delay is for an initial email from an unknown source. Once whitelisted, no delay. Second, email is not (and never was) instant messaging, despite it appearing that way. And third … um … do not talk about Fight Club?

We all bow down to the gods of traffic, but the reality is that not all traffic is created equal. Perhaps if you’re selling junk page views to an ad network for rock-bottom prices, it’s all the same, but for most pages on most “quality” content sites, some kinds of traffic are definitely better than others.

Digg and other social news sites have captured the imagination of publishers because of the massive amounts of referral traffic that Digg in particular can drive, leading to the obvious comparisons to Google and search engine traffic. Danny Sullivan points out that, in terms of raw traffic, sites like Digg seem to beat out non-Google search engines. (TechCrunch’s referral sources was an important reference point.)

But mounting evidence suggests that Digg traffic in particular is less like networking with like-minded individuals at a social event and more like getting attacked by a pack of wild dogs, who leave nothing of value in their wake, other than lessons learned on closing comments and crashed servers.

Via Flutterby, Not All Traffic Is Created Equal

I hadn't given it much thought about the type of traffic coming to a website, but in this interesting (and short) article, a distinction is made between “good” traffic (traffic that leads to sales, say, or people sticking around to view more of your site) and “bad” traffic (a ton of people who aren't interested in your site thundering through and possibly leaving rude comments in their wake).

It's one thing to have your website inundated with traffic (that the server may not be able to handle) with people that are interested in your site; it's another to have your server taken down by people that are overly critical. So, just because you can “digg it,” doesn't mean you want to be “dugged.”

“Do. Or do not. There is no try.”

Yoda

No, I haven't forgotten about this site; it's just that I've been rather busy here at The Office, configuring network equipment and servers that I haven't had a chance to update.

Which is a shame, since without regular updates, no one will bother to come here, and with no one coming here, the point of this blog (or any corporate blog for that matter) to drive readership and to promote the company, is in vain.

So, once you set down this course, you should make sure that it's updated on a regular schedule. And maintain that schedule.

Okay, so it was a little bit longer than a few days, but this site has moved to it's new (and hopefully, final) location and server.

When I originally changed the domain of this blog, just the domain changed, not the server is was on. Since we run Apache, it was a simple matter to configure the webserver as:

<VirtualHost 66.252.227.139>
  ServerName    saltmine.pickint.net
  ServerAlias	pb1.flummux.org

  # A whole bunch of other directives
  # that we needn't get into at this
  # time ... 

</VirtualHost>

Notice how the old domain name was configured as an alias. This told Apache to respond to pb1.flummux.org as well as saltmine.pickint.net. The reason for this is to prevent any links (like there were any at that time) from breaking. It also meant that any search results (like any search engine had actually indexed the site at that time) wouldn't break either, and that's a very important thing.

But since we not only changed names, but server as well, such a trick won't work. And by this time, we have been indexed, and have a few links and whatnot. In this case, to prevent the links (and to keep any Google juice we might have) we have to take a different approach.

So, after setting up the new domain and making sure the blog works on the new server, I went back to the original server, and made a slight change to the configuration:

<VirtualHost 66.252.227.139>
  ServerName	saltmine.pickint.net
  ServerAlias	pb1.flummux.org

  Redirect 	permanent / http://www.saltminechronicles.com/

  # every other directive has been deleted,
  # as they're not required any more

</VirtualHost>

This particular Redirect directive of Apache instructs web browsers that everything on the site has moved permanently to a new location and to use that location from now on (and a browser should update any bookmarks for the old site to use the new site without having the user do anything).

And the Redirect can be used even if you aren't planning on moving your site to a new domain name, it can be used when you reorganize your website. Here's an example from a friend's site after he restructured his site:

Redirect permanent /software/seminole http://gladesoft.com/products/seminole/
Redirect permanent /software/seminole/ http://gladesoft.com/products/seminole/

Nothing else on the site changed except for the renaming of the software folder to products. No links were broken, and more importantly, no Google juice was lost.

Fortunately, the Redirect directive is not restricted to the main Apache configuration file (which you probably don't have access to if you don't host your own site)—they can be placed within an .htaccess file (which is a file that resides within your webhosting space on the server).

So hopefully, this is the latest last location of this blog and it won't be changing any time soon.

I was pointed to this article about why web hosting companies suck. It's worth the read because it does point out a lot of facts about our current market. The author also presents a list of questions you should ask about your web hosting company and I'd like to take the time to answer them about our web hosting company.

“Do you own all of your hardware?”

Yes we do. In fact, we also run our own data center (and all the hassle that involves).

“What type of server hardware/software do you run, and what are the specifications?”

We're primarily a Linux shop here, running the lastest distribution of CentOS with Apache as our web server software. The actual server specs depend upon the server itself but they're all current machines.

“Who is your backbone provider and what is your overall capacity?”

We currently have three backbone providers, Global Crossing, Expedient and Savvis. I'm not sure of our actual network capacity since I don't deal with that part of the network but personally I have never seen us max out our network, unless we were under a DDoS attack, which is another question.

“Where are your servers physically located and are they securely housed?”

Our servers are physically located in two data centers, one located in Boca Raton, Florida (the current location of our offices) and another data center in Charlotte, North Carolina. We run the data center in Boca Raton; we rent several cabinets from a data center company on Charlotte. Both facilities require authorized access to gain entry.

“Can I have the name of your co-location facility?”

Erm … Mistah Wheelus will have to approve me answering this question. I don't have a problem mentioning it, but I don't officially speak for Pick // Internet Services.

“Do your servers have backup power? If so, what type?”

Yes. UPS on each box, and the data centers have on-site generators in case of power failure. I know that last year after Hurricane Wilma hit South Florida, we were down for about three hours durring the worst part of the storm, but a huge web hosting company down the street was out of power for a few days. We were running on generator backup for a week.

“Do you build redundancy into your infrastructure? If so, how?”

We have multiple network backbone providers so the network portion is covered. We also have servers located in two physically different locations so one being wiped off the face of the Earth won't wipe our company out entirely. We don't have full redundancy in our servers, but that is an issue we are currently researching.

“How quickly can you deal with catastrophic hardware failures?”

Pretty darn quickly, usually within the day. We've had several catastrophic hardware failures in the past (always servers, never with a router or switch) and were able to rectify the situation in a day or so. Like I said, we're currently researching methods to add redundancy to our server platforms.

“How much expertise do you have managing servers, and in particular hosting dynamic sites?”

Between Mistah Wheelus and myself, we have over twenty years experience in running web servers under Linux and Apache. We tend to concentrate on smaller sites and have some experience with dynamic sites, but nothing on the level of say, CNN or LiveJournal. Then again, if you have a huge dynamic site like CNN or LiveJournal you probably have the infrastructure in-house to run it.

“What systems are in place to cope with the added demand of dynamic sites?”

Buy bigger hardware? Really, it depends upon the expected level of traffic of the dynamic site and if you expect a huge amount of traffic for your highly dynamic site, then we might not be the best hosting company. Granted, we'll try working with you and seeing if we can do it, but that really depends upon the amount of traffic you get.

“Do you do daily backups?”

Yes. Of all our webservers.

“Do you backup the database?”

If you mean our customers, yes, it's part of our daily backup schedule. If you mean our database, then yes, that's backed up too.

“Do you run dedicated servers for individual tasks (sending email, database, web hosting, etc.) or does every action associated with every site happen on the single server hosting it?”

Yes and no. Web hosting, email and databases for each site run on the server the site is on, which hasn't proved to be a problem for anyone (on performance—there are other issues with email, but that, as Alton Brown would say, is another show), although we do have a separate email server that filters spam that we charge extra for. We also have separate servers for DNS.

“Are you able to meet the demands of enterprise level sites (do you run load-balanced servers, etc.)?”

Heck no. That's not our target market. It's a nice market, but it's not one we're in.

Heck yes!

But seriously, this answer hinges on what exactly is meant by the term “enterprise” and that's why The Powers That Be and I had a discussion about this today (let me reassure you it's nothing to worry about, just a difference of definition).

When I think of “enterprise” I tend to think of sites like CNN or LiveJournal or even Google—really huge sites run by the companies themselves since they have the infrastructure to handle such sites (although given that Google can dropship whole datacenters perhaps they've moved beyond “enterprise” and into the “deathstar” level (joke! It's a geek joke ... Enterprise, Death Star ... oh ... bother)). I'm thinking “insanely huge clusters of machines measured in square yardage.”

Mistah Wheelus has a different measure for “enterprise”—we certainly host sites for Fortune 500 Companies. Yes, we've set up (and currently manage) redundant fail-over servers for clients. And we've had sites that required load balancers (until said sites got bought out by Fortune 500 Companies and moved closer to corporate headquarters—hey, it happens). And as a company, we're not averse to large projects—heck, I think they're fun (I know I had a blast researching running a 100 machine cluster).

So the “official line” here is—yes, we are able to meet the demands of an enterprise level site (excepting for Google—that's way beyond mortal comprehension).

“Do you have systems in place to deal with DoS attacks?”

Yes, as best we can. I have quite a bit of experience in dealing with DoS attacks on the server level and some experience with mitigating it at the network level, but our network engineer, Dan, has more experience with the network side.

“How long have you been in business?”

Mistah Wheelus and I have been in this webhosting business since 1995 when we started in Mistah Wheelus' dining room (at the time, he had no garage, and besides, garages don't typically have A/C—ah, those were the days). I'm not sure when Pick // Internet Services was started, but it was within the past five or six years when we aquired it and started doing business under the name.

“Are your systems set up to notify your staff instantly when your hardware has a problem, or must your clients inform you of a problem?”

Both, but it depends upon the nature of the problem. If a server goes down, or is otherwise inaccessible, we do get notification. Problems with a website, say, a malfunctioning or buggy shopping cart, the customer will inform us of the problem. Network congestion … depends upon the severity of the situation in the data center.

“Do you offer phone support?”

Yes.

“Can your provide a significant list of clients hosted by you, particularly clients with very highly trafficked sites?”

Again, that's a question that Mistah Wheelus can answer.

So there you go. Not half bad and we can provide some answer to each of the questions but I do expect some clarifications will come from higher up in the company, if only for the questions that Mistah Wheelus can answer.

Since I wrote the last article this site has been down a few times, and no, it wasn't because of any networking issues.

The site was actually down.

Currently, this site is hosted on my workstation at Pick // Internet Services—it has yet to be moved to an actual server and in the meantime, the site has been down twice. Technically. Okay, down days at a time, but it was down twice.

Also, I forgot that when I reboot my workstation, the webserver doesn't come up automatically.

Oops.

That will change in the next few days as this site is moved to a proper web server. And it will get its own domain name as well. So the site should be more … available … in the immediate future.

It's not unusual for us to get a trouble ticket that says not much more than “My site is down! Fix it!” And nine times out of ten, we (that is, the Tech Support staff here at Pick // Internet Services) are able to bring the site up in our browsers, as we reply with to the ticket with:

In order to help us more accurately identify the source of the latency you are experiencing, please download the following program to your desktop:

http://support.pickint.net/resources/winmtr.exe

There is no need to “install” the software; simply click on the WinMTR.exe icon and a window will appear. In the “Host” box please type the domain name or the ip address you are having difficulty connecting to.

WinMTR will begin a diagnotic routine that continuously pings each host between your computer and the server you entered in the Host box.

Please let this run for several minutes and then click “Copy Text to clipboard”.

Then paste the results in to this ticket.

Thanks for helping us get the data we need to resolve the latency issue you have reported.

We will get back to you shortly.

(that is the exact text we use, and it's not unusual for most Tech Support departments to have pre-canned responses to common problems)

But it's not that hard to troubleshoot the exact problem and save all of us from having to play Twenty Questions.

Now, given that all of our customers use Windows, the instructions following assume that you too are using Windows (an 88% chance at the time of this writing, but don't worry, if you are using a Mac I'll be telling you what to do, and what you see should be similar enough to what the Window users will see; if you're using Linux, you probably know enough to trouble shoot the problem anyway).

So, the next time you can't get to your website, before calling or submitting a ticket saying “My site is down! Fix it!” take a few moments and do the following. First, click the “Start” button, select ”Run” and type cmd, then click “Okay”. A black window will pop up, with the contents looking something like:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>

(and for this, I'm using The Kids' computer, since I don't have a Windows system of my own. Mac users should run the Terminal program, found under /Applications/Utilities). For this example, I'm using the Pick // Internet Services website, at http://www.pickint.net/. You'll use your own domain name for this, preceeded by www.:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>ping www.pickint.net

Pinging www.pickint.net [204.29.162.248] with 32 bytes of data:

Reply from 204.29.162.248: bytes=32 time=26ms TTL=59
Reply from 204.29.162.248: bytes=32 time=26ms TTL=59
Reply from 204.29.162.248: bytes=32 time=33ms TTL=59
Reply from 204.29.162.248: bytes=32 time=26ms TTL=59

Ping statistics for 204.29.162.248:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 26ms, Maximum = 33ms, Average = 27ms

C:\Documents and Settings\Kids>

(Mac users, the command is the same, although you have to press Ctrl-C to stop it running. The output is slightly different but not enough to worry about it)

This shows that the server is accessible from your computer. The bit that says time=26ms tells you how long it took for a packet of data to make a round trip from your computer to the server and back again, in milliseconds. A double digit number is very good, and low triple digits is okay. If you still have problems pulling your website up in a browser, at this point it's probably a problem on the server itself, so you can call or submit a ticket with this information. But, if the times are above 400ms or so, then it's likely to be a network problem somewhere along the way (which we'll get to in a bit).

Now, if you get the following:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>ping www.pickint.net
Ping request could not find host www.pickint.net. Please check the name and try again.

C:\Documents and Settings\Kids>

(Mac users: the message will be: ping: unknown host www.pickint.net)

There are two possible problems. One, your ISP is having DNS issues and we can't help you. Other symptoms of this problem is that you can't get to other, or any website. The other problem might be: your domain registration expired, so yes, your site is down, but what happens is different than if the server is down.

The other result from running that command:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>ping www.pickint.net

Pinging www.pickint.net [204.29.162.248] with 32 bytes of data:

Reply from 10.0.1.1: Destination net unreachable.
Reply from 10.0.1.1: Destination net unreachable.
Reply from 10.0.1.1: Destination net unreachable.
Reply from 10.0.1.1: Destination net unreachable.

Ping statistics for 204.29.162.248:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Kids>

(Mac users: there will be no output at all, so after a bit, just press Ctrl-C)

This is a networking issue (and don't worry if the reply comes from some other IP address—this is an example, remember?) and if you are getting the impression that most of the time, the problems are due to networking issues, that's because it's probably the case.

Now, to trouble shoot that, you can download WinMTR and run that, but Windows also comes with a similar program to that, tracert (under just about everything else, including the Mac, this is traceroute).

C:\Documents and Settings\Kids>tracert www.pickint.net

Tracing route to www.pickint.net [204.29.162.248]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  hobbes.hangar18.area51 [10.0.2.1]
  2     4 ms     3 ms     4 ms  janet.dreamland.area51 [10.0.1.1]
  3     5 ms     4 ms     5 ms  spc.bct.dsl.pickint.net [66.252.226.49]
  4    25 ms    27 ms    70 ms  core.bct.rt.pickint.net [66.252.227.33]
  5    26 ms    25 ms    26 ms  204.29.162.248

Trace complete.

C:\Documents and Settings\Kids>

(for Mac users: the command is traceroute and the output is reversed—the host or router is listed first, then the timing information. Other than that, it's pretty much the same)

This command (much like WinMTR) will show each point along the Internet data from your computer to your website will take, and how long it takes to get to each point (skip).

Occasionally, you'll see something like:

C:\Documents and Settings\Kids>tracert www.pickint.net

Tracing route to www.pickint.net [204.29.162.248]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  hobbes.hangar18.area51 [10.0.2.1]
  2     4 ms     3 ms     4 ms  janet.dreamland.area51 [10.0.1.1]
  3     5 ms     4 ms     5 ms  spc.bct.dsl.pickint.net [66.252.226.49]
  4    25 ms    27 ms    70 ms  core.bct.rt.pickint.net [66.252.227.33]
  5     *        *        *     Request timed out
  6    26 ms    25 ms    26 ms  204.29.162.248

Trace complete.

C:\Documents and Settings\Kids>

Where one of the hops doesn't report anything. Sometimes a router is programmed not to send back an error, or some other router on the way back filters such error messages, or there's too much traffic at that instance on that router (or host) for it to bother sending back an error. One or two occasional such lines are fine and normal.

But when you start seeing three, four, five such lines in a row, there's a problem. And depending upon how far along the problem is, it could be an issue with your ISP, or with some network provider between you and your website, or with your webhosting company.

But then sometimes you'll see something like:

C:\Documents and Settings\Kids>tracert www.pickint.net

Tracing route to www.pickint.net [204.29.162.248]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  hobbes.hangar18.area51 [10.0.2.1]
  2     4 ms     3 ms     4 ms  janet.dreamland.area51 [10.0.1.1]
  3   100 ms   104 ms   102 ms  spc.bct.dsl.pickint.net [66.252.226.49]
  4   125 ms   127 ms   170 ms  core.bct.rt.pickint.net [66.252.227.33]
  5   126 ms   125 ms   126 ms  204.29.162.248

Trace complete.

C:\Documents and Settings\Kids>

Note the rather large jump in times between hops 2 and 3? (in this case, it's my DSL router). This means the problem is with my ISP (which, in this case, is Pick // Internet Services as a perk of working there).

For another example, here's the output from WinMTR (the actual output is text, I converted it into a table format, and changed the host/hop names so they were less cryptic and shorter!):

WinMTR statistics

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
customer.mi.comcast.net	0	72	72	0	140	710	170
r-alpha.mi.comcast.net	0	72	72	0	137	280	110	*
r-bravo.mi.comcast.net	0	72	72	0	143	1150	1150	*
r-charlie.mi.comcast.net	2	72	71	0	144	1150	160
12.116.16.25	0	71	71	0	133	330	110
r-alpha.cgcil.att.net	0	71	71	0	135	330	170
r-alpha.phlpa.att.net	0	71	71	50	135	220	160
r-beta.phlpa.att.net	0	71	71	50	136	330	110
12.119.53.118	0	71	71	0	137	330	170
r-alpha.pitb.telcove.net	0	71	71	0	146	330	160
r-alpha.atln.telcove.net	0	71	71	50	140	330	170
24.56.107.70	0	71	71	110	154	390	110
No response from host	100	71	0	0	0	0	0
r-alpha.cm1.peak-10.net	0	71	71	50	196	330	220
66.129.112.148	0	71	71	50	140	220	110
www.example.com	0	71	71	50	149	330	160

This is an actual WinMTR sent to us by a customer. WinMTR works similarly to tracert but keeps sending packets until stopped.

Now, notice the two marked lines. These are still within the network of the customer's ISP so the problem was not something we could handle—the customer has to call his ISP to complain.

We have another customer, on the other side of the world, that will complain about the site being down, or being slow (another indication of a possible network issue) and we always have to remind this customer to send in a WinMTR trace and the majority of the times, it's due to some trans-Atlantic connection that is slow, and not us.

So the next time you can't get to your site, you may want to run ping and tracert and see if it's our problem, your ISP's problem, or something going on between the two.

On the birth of Mistah Wheelus' first son this evening at 5:30 pm.

It's reported that all are doing well.