The Saltmine Chronicles

A small primer on Greylisting email

2007-10-22T23:37:12-05:00

Mistah Wheelus requested I write an explanation we could give to our customers about greylisting. And here's my attempt at being lucid.

Greylisting is an easy and effective (so far) anti-spam technique (our current tests show an effective spam-stopping rate of over 97%) but before I explain how it works, I must first explain a bit how the email system works.

Once you click “send” your computer will connect with an outgoing email server (this is the “Outgoing SMTP Server” setting in the configuration of your email account); it then identifies itself to the outgoing email server, sends your email address (technically, the “sender address”), then the email address of who you are sending it to (the “recipient address”) and then finally, the actual email (which may have nothing to do with either the sender or recipient email address, a fact that spammers often exploit for fun and profit).

Once the outgoing email server has accepted the email, it is then queued up for final delivery.

Technobabble

Actually, I'm describing the most commonly used configuration, where all outgoing emails for an organization (or an ISP) are funneled through a so-called “relay host” or “smart host” because of security issues or as a means of preventing outgoing spam.

Some ISPs go so far as to block all outgoing email traffic from their subscriber base, only allowing connections to their outgoing email server.

If an outgoing email server isn't required, then your computer may very well connect directly to the server responsible for the recipient's email and deliver the email directly. But then, your computer becomes responsible for redelivery in case the recpient server can't accept the email at that time.

There are more details of this in the next Technobabble section.

The outgoing email server will then look up where to send the email based upon the recipient's domain name, and once this is done, connects to an incoming email server that handles email for the recipient, and using SMTP, deliver the email to the recipient's email box. And if for any reason the email can't be delivered, or there's an error during the delivery, the outgoing email server queues up the email for another attempt at a later time (which can be a few minutes to maybe an hour later). And this is an important detail to remember.

Technobabble

I glossed over quite a few details here. The computer sending the email to the recipient first does a DNS lookup for a special type of record, the MX record. This returns a list of servers than handle email for that domain. For instance, at this moment in time, the following servers handle incoming email for gmail.com:

Incoming email servers for Gmail
Server name	Server Priority
gmail-smtp-in.l.google.com	5
alt1.gmail-smtp-in.l.google.com	10
alt2.gmail-smtp-in.l.google.com	10
gsmtp163.google.com	50
gsmtp183.google.com	50

The server(s) with the lowest priority is checked first. If more than one server has the same priority, then one is picked randomly. So, in this case, if for some reason gmail-smtp-in.l.google.com is not responding, then the sending computer picks either alt1.gmail-smtp-in.l.google.com or alt2.gmail-smtp-in.l.google.com.

Oh, and what if there isn't an MX for the domain in question? Then the sending computer looks up the IP address associated with the domain and delivers the email to that machine.

Once the email has been successfully delivered, it's then saved in the recpients incoming email box, which stays there until the recipient retrieves the email (which is beyond the scope of this entry).

Now, how does Greylisting fit into all of this?

Greylisting works on the recipient side of this. Send me an email, and eventually, some server from your end (“your server”) will contact the server on my end (“my server”) to deliver the email. My server then has three pieces of information: the IP address of your server, your email address (assuming it matches the sender address) and my email address. And for the sake of an example, let's say it's [ 3.4.5.6 , fred@example.net , sean@pickint.net ]. My server will see if it has seen that particular combination before, and if not, record it, and send back to your server “try again later.” And until it's been at least 25 minutes since I first saw that particular combination, my server will keep sending back “try again later.”

After the initial 25 minutes, any email from 3.4.5.6 with sender fred@example.net and recipient sean@pickint.net will be accepted. But other emails from 3.4.5.6 can still experience the delay, if the sender email addresss, recipient email address, or both, are different. It's the combination of all three pieces of information that have to match.

Basically, greylisting delays an initial email by some period of time, only “whitelisting” it after a delay period. And while it seems strange, that simple strategy can easily filter out 97% of all spam, since most spammers don't want to bother with redelivery of non-delivered email. They're trying to get their spam out as fast as possible. Attempting to redeliver their spam will only complicate things on their end.

And it's this delay that causes the biggest complaints. But the delay is for an initial email from an unknown source. Once whitelisted, no delay. Second, email is not (and never was) instant messaging, despite it appearing that way. And third … um … do not talk about Fight Club?

There's good traffic, and then there's bad traffic

2007-01-26T16:57:08-05:00

We all bow down to the gods of traffic, but the reality is that not all traffic is created equal. Perhaps if you’re selling junk page views to an ad network for rock-bottom prices, it’s all the same, but for most pages on most “quality” content sites, some kinds of traffic are definitely better than others.

Digg and other social news sites have captured the imagination of publishers because of the massive amounts of referral traffic that Digg in particular can drive, leading to the obvious comparisons to Google and search engine traffic. Danny Sullivan points out that, in terms of raw traffic, sites like Digg seem to beat out non-Google search engines. (TechCrunch’s referral sources was an important reference point.)

But mounting evidence suggests that Digg traffic in particular is less like networking with like-minded individuals at a social event and more like getting attacked by a pack of wild dogs, who leave nothing of value in their wake, other than lessons learned on closing comments and crashed servers.

Via Flutterby, Not All Traffic Is Created Equal

I hadn't given it much thought about the type of traffic coming to a website, but in this interesting (and short) article, a distinction is made between “good” traffic (traffic that leads to sales, say, or people sticking around to view more of your site) and “bad” traffic (a ton of people who aren't interested in your site thundering through and possibly leaving rude comments in their wake).

It's one thing to have your website inundated with traffic (that the server may not be able to handle) with people that are interested in your site; it's another to have your server taken down by people that are overly critical. So, just because you can “digg it,” doesn't mean you want to be “dugged.”

“I'm not dead yet!”

2006-11-10T17:21:29-05:00

“Do. Or do not. There is no try.”

Yoda

No, I haven't forgotten about this site; it's just that I've been rather busy here at The Office, configuring network equipment and servers that I haven't had a chance to update.

Which is a shame, since without regular updates, no one will bother to come here, and with no one coming here, the point of this blog (or any corporate blog for that matter) to drive readership and to promote the company, is in vain.

So, once you set down this course, you should make sure that it's updated on a regular schedule. And maintain that schedule.

“My site is down! Fix it!” Redux part II

2006-10-14T22:29:37-05:00

Okay, so it was a little bit longer than a few days, but this site has moved to it's new (and hopefully, final) location and server.

Technobabble and an apology

To the one person who's currently subscribed to the Atom feed, I must apologize for screwing up the entry IDs and throwing a whole batch of previously read entries as new. The software used for this blog, mod_blog has very minimal support for entry IDs and no support for preserving IDs across domain moves. But I wasn't expecting ~~the Spanish Inquisition~~ Mistah Wheelus to get this blog its own domain name.

Sorry.

When I originally changed the domain of this blog, just the domain changed, not the server is was on. Since we run Apache, it was a simple matter to configure the webserver as:

<VirtualHost 66.252.227.139>
  ServerName    saltmine.pickint.net
  ServerAlias	pb1.flummux.org

  # A whole bunch of other directives
  # that we needn't get into at this
  # time ... 

</VirtualHost>

Notice how the old domain name was configured as an alias. This told Apache to respond to pb1.flummux.org as well as saltmine.pickint.net. The reason for this is to prevent any links (like there were any at that time) from breaking. It also meant that any search results (like any search engine had actually indexed the site at that time) wouldn't break either, and that's a very important thing.

But since we not only changed names, but server as well, such a trick won't work. And by this time, we have been indexed, and have a few links and whatnot. In this case, to prevent the links (and to keep any Google juice we might have) we have to take a different approach.

So, after setting up the new domain and making sure the blog works on the new server, I went back to the original server, and made a slight change to the configuration:

<VirtualHost 66.252.227.139>
  ServerName	saltmine.pickint.net
  ServerAlias	pb1.flummux.org

  Redirect 	permanent / http://www.saltminechronicles.com/

  # every other directive has been deleted,
  # as they're not required any more

</VirtualHost>

This particular Redirect directive of Apache instructs web browsers that everything on the site has moved permanently to a new location and to use that location from now on (and a browser should update any bookmarks for the old site to use the new site without having the user do anything).

Technobabble

The web protocol, HTTP, includes a mechanism to inform the client side (that's the browser) that a resource (say, an image) has moved to a new location—a “redirect.” The two most common types are the “temporary redirect” and the “permanent redirect” and they are pretty much what they say.

From the actual protocol (which seems pretty straightforward to me):

10.3.2 301 Moved Permanently

The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs. Clients with link editing capabilities ought to automatically re-link references to the Request-URI to one or more of the new references returned by the server, where possible. …

10.3.3 302 Found

The requested resource resides temporarily under a different URI. Since the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests. …

Status Code Definitions

(Just for the record, a URI is a more technically “pure” term for a URL and has a specific meaning. If you really want to be pedantic about it, all URLs are URIs, but not all URIs are URLs. But this is a bit too much heavy wizardry even for the Technobabble sections.)

And the Redirect can be used even if you aren't planning on moving your site to a new domain name, it can be used when you reorganize your website. Here's an example from a friend's site after he restructured his site:

Redirect permanent /software/seminole http://gladesoft.com/products/seminole/
Redirect permanent /software/seminole/ http://gladesoft.com/products/seminole/

Nothing else on the site changed except for the renaming of the software folder to products. No links were broken, and more importantly, no Google juice was lost.

Fortunately, the Redirect directive is not restricted to the main Apache configuration file (which you probably don't have access to if you don't host your own site)—they can be placed within an .htaccess file (which is a file that resides within your webhosting space on the server).

So hopefully, this is the latest last location of this blog and it won't be changing any time soon.

Why Most Web Hosting Companies Suck

2006-10-14T22:30:11-05:00

I was pointed to this article about why web hosting companies suck. It's worth the read because it does point out a lot of facts about our current market. The author also presents a list of questions you should ask about your web hosting company and I'd like to take the time to answer them about our web hosting company.

“Do you own all of your hardware?”: Yes we do. In fact, we also run our own data center (and all the hassle that involves).
“What type of server hardware/software do you run, and what are the specifications?”: We're primarily a Linux shop here, running the lastest distribution of CentOS with Apache as our web server software. The actual server specs depend upon the server itself but they're all current machines.
“Who is your backbone provider and what is your overall capacity?”: We currently have three backbone providers, Global Crossing, Expedient and Savvis. I'm not sure of our actual network capacity since I don't deal with that part of the network but personally I have never seen us max out our network, unless we were under a DDoS attack, which is another question.
“Where are your servers physically located and are they securely housed?”: Our servers are physically located in two data centers, one located in Boca Raton, Florida (the current location of our offices) and another data center in Charlotte, North Carolina. We run the data center in Boca Raton; we rent several cabinets from a data center company on Charlotte. Both facilities require authorized access to gain entry.
“Can I have the name of your co-location facility?”: Erm … Mistah Wheelus will have to approve me answering this question. I don't have a problem mentioning it, but I don't officially speak for Pick // Internet Services.
“Do your servers have backup power? If so, what type?”: Yes. UPS on each box, and the data centers have on-site generators in case of power failure. I know that last year after Hurricane Wilma hit South Florida, we were down for about three hours durring the worst part of the storm, but a huge web hosting company down the street was out of power for a few days. We were running on generator backup for a week.
“Do you build redundancy into your infrastructure? If so, how?”: We have multiple network backbone providers so the network portion is covered. We also have servers located in two physically different locations so one being wiped off the face of the Earth won't wipe our company out entirely. We don't have full redundancy in our servers, but that is an issue we are currently researching.
“How quickly can you deal with catastrophic hardware failures?”: Pretty darn quickly, usually within the day. We've had several catastrophic hardware failures in the past (always servers, never with a router or switch) and were able to rectify the situation in a day or so. Like I said, we're currently researching methods to add redundancy to our server platforms.
“How much expertise do you have managing servers, and in particular hosting dynamic sites?”: Between Mistah Wheelus and myself, we have over twenty years experience in running web servers under Linux and Apache. We tend to concentrate on smaller sites and have some experience with dynamic sites, but nothing on the level of say, CNN or LiveJournal. Then again, if you have a huge dynamic site like CNN or LiveJournal you probably have the infrastructure in-house to run it.
“What systems are in place to cope with the added demand of dynamic sites?”: Buy bigger hardware? Really, it depends upon the expected level of traffic of the dynamic site and if you expect a huge amount of traffic for your highly dynamic site, then we might not be the best hosting company. Granted, we'll try working with you and seeing if we can do it, but that really depends upon the amount of traffic you get.
“Do you do daily backups?”: Yes. Of all our webservers.
“Do you backup the database?”: If you mean our customers, yes, it's part of our daily backup schedule. If you mean our database, then yes, that's backed up too.
“Do you run dedicated servers for individual tasks (sending email, database, web hosting, etc.) or does every action associated with every site happen on the single server hosting it?”: Yes and no. Web hosting, email and databases for each site run on the server the site is on, which hasn't proved to be a problem for anyone (on performance—there are other issues with email, but that, as Alton Brown would say, is another show), although we do have a separate email server that filters spam that we charge extra for. We also have separate servers for DNS.
“Are you able to meet the demands of enterprise level sites (do you run load-balanced servers, etc.)?”: ~~Heck no. That's not our target market. It's a nice market, but it's not one we're in.~~
Heck yes!

But seriously, this answer hinges on what exactly is meant by the term “enterprise” and that's why The Powers That Be and I had a discussion about this today (let me reassure you it's nothing to worry about, just a difference of definition).

When I think of “enterprise” I tend to think of sites like CNN or LiveJournal or even Google—really huge sites run by the companies themselves since they have the infrastructure to handle such sites (although given that Google can dropship whole datacenters perhaps they've moved beyond “enterprise” and into the “deathstar” level (joke! It's a geek joke ... Enterprise, Death Star ... oh ... bother)). I'm thinking “insanely huge clusters of machines measured in square yardage.”

Mistah Wheelus has a different measure for “enterprise”—we certainly host sites for Fortune 500 Companies. Yes, we've set up (and currently manage) redundant fail-over servers for clients. And we've had sites that required load balancers (until said sites got bought out by Fortune 500 Companies and moved closer to corporate headquarters—hey, it happens). And as a company, we're not averse to large projects—heck, I think they're fun (I know I had a blast researching running a 100 machine cluster).

So the “official line” here is—yes, we are able to meet the demands of an enterprise level site (excepting for Google—that's way beyond mortal comprehension).
“Do you have systems in place to deal with DoS attacks?”: Yes, as best we can. I have quite a bit of experience in dealing with DoS attacks on the server level and some experience with mitigating it at the network level, but our network engineer, Dan, has more experience with the network side.
“How long have you been in business?”: Mistah Wheelus and I have been in this webhosting business since 1995 when we started in Mistah Wheelus' dining room (at the time, he had no garage, and besides, garages don't typically have A/C—ah, those were the days). I'm not sure when Pick // Internet Services was started, but it was within the past five or six years when we aquired it and started doing business under the name.
“Are your systems set up to notify your staff instantly when your hardware has a problem, or must your clients inform you of a problem?”: Both, but it depends upon the nature of the problem. If a server goes down, or is otherwise inaccessible, we do get notification. Problems with a website, say, a malfunctioning or buggy shopping cart, the customer will inform us of the problem. Network congestion … depends upon the severity of the situation in the data center.
“Do you offer phone support?”: Yes.
“Can your provide a significant list of clients hosted by you, particularly clients with very highly trafficked sites?”: Again, that's a question that Mistah Wheelus can answer.

So there you go. Not half bad and we can provide some answer to each of the questions but I do expect some clarifications will come from higher up in the company, if only for the questions that Mistah Wheelus can answer.

“My site is down! Fix it!” Redux

2006-10-11T01:16:46-05:00

Since I wrote the last article this site has been down a few times, and no, it wasn't because of any networking issues.

The site was actually down.

Currently, this site is hosted on my workstation at Pick // Internet Services—it has yet to be moved to an actual server and in the meantime, the site has been down twice. Technically. Okay, down days at a time, but it was down twice.

Technobabble

The first time was due to a slight design misfeature of the blogging software I use (which I wrote). I can't really classify it as a bug as more of a debugging feature that should not have been enabled, was enabled and the blogging engine could not open up the debug log file since the system had automatically deleted it, and in such an instance, the blogging software won't run if it can't open the log file.

Also, I forgot that when I reboot my workstation, the webserver doesn't come up automatically.

Oops.

That will change in the next few days as this site is moved to a proper web server. And it will get its own domain name as well. So the site should be more … available … in the immediate future.

“My site is down! Fix it!”

2006-09-27T23:52:04-05:00

It's not unusual for us to get a trouble ticket that says not much more than “My site is down! Fix it!” And nine times out of ten, we (that is, the Tech Support staff here at Pick // Internet Services) are able to bring the site up in our browsers, as we reply with to the ticket with:

In order to help us more accurately identify the source of the latency you are experiencing, please download the following program to your desktop:

http://support.pickint.net/resources/winmtr.exe

There is no need to “install” the software; simply click on the WinMTR.exe icon and a window will appear. In the “Host” box please type the domain name or the ip address you are having difficulty connecting to.

WinMTR will begin a diagnotic routine that continuously pings each host between your computer and the server you entered in the Host box.

Please let this run for several minutes and then click “Copy Text to clipboard”.

Then paste the results in to this ticket.

Thanks for helping us get the data we need to resolve the latency issue you have reported.

We will get back to you shortly.

(that is the exact text we use, and it's not unusual for most Tech Support departments to have pre-canned responses to common problems)

But it's not that hard to troubleshoot the exact problem and save all of us from having to play Twenty Questions.

Now, given that all of our customers use Windows, the instructions following assume that you too are using Windows (an 88% chance at the time of this writing, but don't worry, if you are using a Mac I'll be telling you what to do, and what you see should be similar enough to what the Window users will see; if you're using Linux, you probably know enough to trouble shoot the problem anyway).

So, the next time you can't get to your website, before calling or submitting a ticket saying “My site is down! Fix it!” take a few moments and do the following. First, click the “Start” button, select ”Run” and type cmd, then click “Okay”. A black window will pop up, with the contents looking something like:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>

(and for this, I'm using The Kids' computer, since I don't have a Windows system of my own. Mac users should run the Terminal program, found under /Applications/Utilities). For this example, I'm using the Pick // Internet Services website, at http://www.pickint.net/. You'll use your own domain name for this, preceeded by www.:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>ping www.pickint.net

Pinging www.pickint.net [204.29.162.248] with 32 bytes of data:

Reply from 204.29.162.248: bytes=32 time=26ms TTL=59
Reply from 204.29.162.248: bytes=32 time=26ms TTL=59
Reply from 204.29.162.248: bytes=32 time=33ms TTL=59
Reply from 204.29.162.248: bytes=32 time=26ms TTL=59

Ping statistics for 204.29.162.248:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 26ms, Maximum = 33ms, Average = 27ms

C:\Documents and Settings\Kids>

(Mac users, the command is the same, although you have to press Ctrl-C to stop it running. The output is slightly different but not enough to worry about it)

This shows that the server is accessible from your computer. The bit that says time=26ms tells you how long it took for a packet of data to make a round trip from your computer to the server and back again, in milliseconds. A double digit number is very good, and low triple digits is okay. If you still have problems pulling your website up in a browser, at this point it's probably a problem on the server itself, so you can call or submit a ticket with this information. But, if the times are above 400ms or so, then it's likely to be a network problem somewhere along the way (which we'll get to in a bit).

Now, if you get the following:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>ping www.pickint.net
Ping request could not find host www.pickint.net. Please check the name and try again.

C:\Documents and Settings\Kids>

(Mac users: the message will be: ping: unknown host www.pickint.net)

There are two possible problems. One, your ISP is having DNS issues and we can't help you. Other symptoms of this problem is that you can't get to other, or any website. The other problem might be: your domain registration expired, so yes, your site is down, but what happens is different than if the server is down.

The other result from running that command:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Kids>ping www.pickint.net

Pinging www.pickint.net [204.29.162.248] with 32 bytes of data:

Reply from 10.0.1.1: Destination net unreachable.
Reply from 10.0.1.1: Destination net unreachable.
Reply from 10.0.1.1: Destination net unreachable.
Reply from 10.0.1.1: Destination net unreachable.

Ping statistics for 204.29.162.248:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Kids>

(Mac users: there will be no output at all, so after a bit, just press Ctrl-C)

This is a networking issue (and don't worry if the reply comes from some other IP address—this is an example, remember?) and if you are getting the impression that most of the time, the problems are due to networking issues, that's because it's probably the case.

Now, to trouble shoot that, you can download WinMTR and run that, but Windows also comes with a similar program to that, tracert (under just about everything else, including the Mac, this is traceroute).

C:\Documents and Settings\Kids>tracert www.pickint.net

Tracing route to www.pickint.net [204.29.162.248]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  hobbes.hangar18.area51 [10.0.2.1]
  2     4 ms     3 ms     4 ms  janet.dreamland.area51 [10.0.1.1]
  3     5 ms     4 ms     5 ms  spc.bct.dsl.pickint.net [66.252.226.49]
  4    25 ms    27 ms    70 ms  core.bct.rt.pickint.net [66.252.227.33]
  5    26 ms    25 ms    26 ms  204.29.162.248

Trace complete.

C:\Documents and Settings\Kids>

(for Mac users: the command is traceroute and the output is reversed—the host or router is listed first, then the timing information. Other than that, it's pretty much the same)

This command (much like WinMTR) will show each point along the Internet data from your computer to your website will take, and how long it takes to get to each point (skip).

Technobabble

traceroute shows each hop packets take from your computer to the destination (and yes, “hop” is a technical term). It does this by using a neat hack.

Each packet that a computer sends out has a “time-to-live” field, which is the maximum number of hops it can take. At each hop, the router will subtract one from this field and if it's equal to zero, the packet is dropped and an error is sent back to the originating computer that the packet “died” enroute.

Typically, the operating system will set this field to a large enough value to ensure that the packet makes it to the destination before the “time-to-live” field hits zero (this value is typically set to 60, although in practice, no two points on the Internet has been greater than 30 hops apart). But traceroute will send the first packet with a “time-to-live” set to 1 (it actually sends three such packets). The immediate next hop will decrement the counter, see that it's zero, and send back an error. Then the next packet with a “time-to-live” set to 2, so the second such hop will return the error.

And so on until a packet reaches the destination, at which point a different error is returned (since the packet is sent to a non-existent program).

Occasionally, you'll see something like:

C:\Documents and Settings\Kids>tracert www.pickint.net

Tracing route to www.pickint.net [204.29.162.248]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  hobbes.hangar18.area51 [10.0.2.1]
  2     4 ms     3 ms     4 ms  janet.dreamland.area51 [10.0.1.1]
  3     5 ms     4 ms     5 ms  spc.bct.dsl.pickint.net [66.252.226.49]
  4    25 ms    27 ms    70 ms  core.bct.rt.pickint.net [66.252.227.33]
  5     *        *        *     Request timed out
  6    26 ms    25 ms    26 ms  204.29.162.248

Trace complete.

C:\Documents and Settings\Kids>

Where one of the hops doesn't report anything. Sometimes a router is programmed not to send back an error, or some other router on the way back filters such error messages, or there's too much traffic at that instance on that router (or host) for it to bother sending back an error. One or two occasional such lines are fine and normal.

But when you start seeing three, four, five such lines in a row, there's a problem. And depending upon how far along the problem is, it could be an issue with your ISP, or with some network provider between you and your website, or with your webhosting company.

But then sometimes you'll see something like:

C:\Documents and Settings\Kids>tracert www.pickint.net

Tracing route to www.pickint.net [204.29.162.248]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  hobbes.hangar18.area51 [10.0.2.1]
  2     4 ms     3 ms     4 ms  janet.dreamland.area51 [10.0.1.1]
  3   100 ms   104 ms   102 ms  spc.bct.dsl.pickint.net [66.252.226.49]
  4   125 ms   127 ms   170 ms  core.bct.rt.pickint.net [66.252.227.33]
  5   126 ms   125 ms   126 ms  204.29.162.248

Trace complete.

C:\Documents and Settings\Kids>

Note the rather large jump in times between hops 2 and 3? (in this case, it's my DSL router). This means the problem is with my ISP (which, in this case, is Pick // Internet Services as a perk of working there).

For another example, here's the output from WinMTR (the actual output is text, I converted it into a table format, and changed the host/hop names so they were less cryptic and shorter!):

WinMTR statistics
Host	%	Sent	Recv	Best	Avrg	Wrst	Last
customer.mi.comcast.net	0	72	72	0	140	710	170
r-alpha.mi.comcast.net	0	72	72	0	137	280	110	*
r-bravo.mi.comcast.net	0	72	72	0	143	1150	1150	*
r-charlie.mi.comcast.net	2	72	71	0	144	1150	160
12.116.16.25	0	71	71	0	133	330	110
r-alpha.cgcil.att.net	0	71	71	0	135	330	170
r-alpha.phlpa.att.net	0	71	71	50	135	220	160
r-beta.phlpa.att.net	0	71	71	50	136	330	110
12.119.53.118	0	71	71	0	137	330	170
r-alpha.pitb.telcove.net	0	71	71	0	146	330	160
r-alpha.atln.telcove.net	0	71	71	50	140	330	170
24.56.107.70	0	71	71	110	154	390	110
No response from host	100	71	0	0	0	0	0
r-alpha.cm1.peak-10.net	0	71	71	50	196	330	220
66.129.112.148	0	71	71	50	140	220	110
www.example.com	0	71	71	50	149	330	160

This is an actual WinMTR sent to us by a customer. WinMTR works similarly to tracert but keeps sending packets until stopped.

Now, notice the two marked lines. These are still within the network of the customer's ISP so the problem was not something we could handle—the customer has to call his ISP to complain.

We have another customer, on the other side of the world, that will complain about the site being down, or being slow (another indication of a possible network issue) and we always have to remind this customer to send in a WinMTR trace and the majority of the times, it's due to some trans-Atlantic connection that is slow, and not us.

So the next time you can't get to your site, you may want to run ping and tracert and see if it's our problem, your ISP's problem, or something going on between the two.

Congratulations!

2006-09-25T22:51:37-05:00

On the birth of Mistah Wheelus' first son this evening at 5:30 pm.

It's reported that all are doing well.

Welcome to the Saltmine Chronicles

2006-09-20T01:57:27-05:00

Welcome to the Official Launch of the Unofficial Official Blog™ of Pick // Internet Services—The Saltmine Chronicles!

I'll be your host and the primary author of this blog, which is bringing you a “behind the scenes” look at webhosting; an explaination of what exactly goes on around here as we sling bits across the Internet. The other intent is to present a more human face to the company and by explaining what goes on here, you can, well, not appreciate what goes on here, but at least understand what's happening and some of the difficulties that come up from time to time. I'll try to keep the technical discussions to a minimum, but if I feel I have to, I'll appropriately label the more technical bits (skip):

Technobabble

Like this. These bits go into more technical detail for those of you who are curious, but if you aren't you can simply gloss over this bit.

And please, don't feel you have to read the more technical bits.

I've also settled on a more-or-less Tuesday-Thursday publication schedule, but that's me. There are a few other folks around here that might be posting, like Mistah Wheelus (the General Manager) or Pavan (someone else in Technical Support), or even Carl (the head of Technical Support).

Hope you like it here.

A notification for the few readers (at this time) that are out there that we now have a new name and a new location for this blog.

2006-09-18T15:20:05-05:00

Surprisingly enough, Mistah Wheelus liked the name “Saltmine Chronicles” so that's what this shall be called then—The Saltmine Chronicles.

We also agreed upon the final location for the blog so expect that to happen in the next day or two.

A little secret

2006-09-22T16:29:04-05:00

Poor FrontPage. Nobody loves you. “Real” web designers wouldn't be caught dead using FrontPage to do client's work. Trouble is, Bill Gates has done such a good job of making it easy for the average Joe to create a reasonably acceptable website, that FrontPage websites are everywhere. As a result, if a hosting company wants to make any money, they have to offer the STUPID FrontPage extensions on the host server so the FrontPage web site will work correctly. But secretly, they hate FrontPage, and sometimes for good reasons; to be honest, FrontPage extensions are a pain in the backside to most hosting administrators. They are unstable, it takes NO effort on the part of the site owner to corrupt them, and the vast majority of people who use FrontPage not only know very little about web design, they know even less about the extensions and how to make everything come together as it should. SO, we end up having to chase after every little mistake a site owner makes, and often have to get on the phone and show site owners how to use their own software!! And this doesn't work, because NO hosting company knows the software, and only knows the extensions enough to install and re-install them. Is this what they call “The Blind Leading The Blind”?
<GRRRRRRRRRRRRRRRR>

MS FrontPage

I'm a bit hesitent to mention this, but yes, there aren't many web hosting companies that actually like Microsoft Frontpage. Or rather, while the web hosting company does support Microsoft Frontpage (because there are enough customers that do use it), it's the technical support representatives that ~~hate~~ dislike Microsoft Frontpage.

Oh, some of it might be due to anti-Microsoft feelings, but leaving those aside, there are technical reasons for the dislike, namely that Frontpage extentions tend to be fragile and it's all too easy for the customer to break them. And while the fix it actually rather easy for the technical support representative, it does take time.

Technobabble

The prescribed solution to a Frontpage problem is to first disable the Frontpage extentions from the site and then re-enable them, which only takes a few minutes at most. If that doesn't solve the problem, the second step is to disable the Frontpage extentions, then delete all the _vti* folders from the website, then re-enable Frontpage extentions (which will rebuild all the _vti* folders).

If that doesn't work (which is extremely rare), then the issue is escalated to the system administrators to look at.

This is standard procedure across many web hosting sites, not just Pick Internet.

Another aspect is that a lot of people (web designers especially) feel that most sites made with Frontpage are ugly. And while I may agree with that sentiment (and this is why this is the “Unofficial Official Blog”) I also don't think that's necessarily a bad thing (as explained in this episode of the show with zefrank) as long as I don't have to deal with it, but yes, that's another reason why Frontpage has the bad reputation that is has, deserved or not.

And one more aspect that you may or may not know—Microsoft is dropping support for Frontpage later this year. So if you rely upon Frontpage to maintain your site, you might want to start exploring alternative options as I expect Frontpage support at web hosting companies to dry up once Microsoft has officially “expired” Frontpage.

So, does that mean we'll drop Frontpage support come 2007? Not likely—we'll probably support it for a few more years yet to come. But I'm just saying it might be prudent to start looking for alternatives while you still have time.

(Oh, and do I like Frontpage? Um … I plead the Fifth)

A reminder

2006-09-12T18:09:44-05:00

Just a reminder (if only for myself)—be aware of when your domain will expire and be sure to renew prior to the expiration date so there will be no disruption in service.

Also, be wary of simply sending a check in response to anything that comes in the regular postal service. Some of the more shady registrars will send out what looks to be an offical expiration notice but really it's authorization to transfer the domain to them! I simply toss those in the garbage and check my account directly with my registrar.

Oh, and how can you tell if your site is expired? If you can't get to it but you can ping the IP address of the server (or bring up the server's webpage) and your hosting provider swears and and down they can see it fine, it's most likely an expired domain.

The Technical Details (aka Technobabble)

What's most likely happening here is the hosting company in question is using their primary name server (which translates names like www.pickint.net into an IP address like 204.29.162.248) which resolves the domain just fine. You, however, are using the name server of your ISP, which has to first query the root name servers to get the hosting company's name server.

When the domain expires, the root servers are instructed to remove this mapping, so your ISP doesn't know where to get the IP address, so that's why you can't bring your site up.

Oh, and for the record, it was this domain that expired today, hence this entry (and no, this isn't the permanent domain for this blog).

Potential Names for This Blog

2006-09-12T00:53:15-05:00

I've received some ideas for names (but not nearly as much as I expected alas), and so far, the list of names are:

PICK the Blog—the current name and one I'm fine with.
PICK I.S. the Blog—where the I.S. stands for “Internet Solutions,” the subtitle of the company.
Blogging for Bosses—which describes what I'm doing but it sounds rather cynical to me. I don't care for it, and I expect Charles won't go for this one either.
Saltmine Chronicles—which, for a work-themed blog is a wonderful name, but it has connotations that I doubt Charles wants associated with this blog (but personally, I love the name).
The Service Station—running with a 50s gas station theme, which appeals to me quite a bit. A whole retro-future look to it.
Uloborid—nobody I've run the name past likes the name, but I always felt the name fit the web.
The Official Unoffical Blog—heck, why not?
The Unofficial Official Blog—or this way?
Pick-ayune
Net Pick-ayune
eTimes Pick-ayune

Any ideas, please mail me.

Name This Blog

2006-09-08T17:16:46-05:00

Welcome to the “Officially Okay but Unofficially Sanctioned Blog” of Pick Internet, written by yours truely (although Mistah Wheelus (I can call you that, right Chuck?), Carl and Pavan might show up from time to time, but I'll make no guarentees at this point). Mistah Wheelus thought it might be a good idea for the company to have a blog to present a human face to the world and since I've been blogging since December of 1999 (hmmm … I'll have to ask Mistah Wheelus if I can link to The Other Place) I was the natural one to pick for the job.

Hi.

When I mentioned I might do this, my SO wrote back:

… though how on earth you're going to do the company blog is what I am wondering. I don't think you're the man for that job, not due to any inadequacy, but due to area of focus. I just don't see you getting that excited about the sorts of things that should be in a blog like that.

In discussing this with Mistah Wheelus, it became clear that about 70% of what I write in The Other Place (and I'm guessing he means 70% of what I write about Pick Internet) is fine for this blog (the rest being either way too technical, or grumbling about customers, even though I never name names) so that's nice to know. But I was thinking more of a “behind the scenes” type of blog, where I describe what goes on at a web hosting company, with the hope that it'll give people a realistic expectation of what they can expect from a web hosting company.

I showed Mistah Wheelus what I have so far of the blog and where exactly it fits into the organization. He wants the blog as an “Officially Okay With” type of blog, but not necessarily one that's served up under the Pick Internet domain. Which might necessarily mean a change of name (and venue, but that was expected from the start anyway). He's looking for maybe a new domain to put this (and maybe other blog related projects) under.

So that's where you come in (and if you're reading this, you are one of the few I selected for this)—any ideas for a name of this blog? The one up there is one I picked but neither I nor Mistah Wheelus are wedded to the name. Or the domain (it's my “play domain” and nothing permanently lives there). So … any ideas? Please mail me your ideas.

This is mock Latin

2006-09-08T02:42:31-05:00

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.

Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.