Google recently made far-reaching changes to the way it ranks search results, and the search marketing community has been abuzz with tales of woe ever since. Some have speculated that the key to understanding Google's latest is that they've applied some sort of test of "commerciality" to certain phrases, roiling the waters for sites ranked well on those phrases, and leaving non-commercial phrases more or less alone. The idea is that this would cause a stampede of site owners over to the paid AdWords program, or at least making the point that Google isn't up for providing a free lunch to clever "SEO-ized" sites indefinitely.
So is this just a theory? Or has Google composed a hit list of terms that are commercially valuable, and changed the way it ranks sites on those terms?

And if so, how does it determine which terms to go after? Some really obscure ones (yet with high bid prices in the pay-per-click program) have seemingly been affected, just as more popular terms have.

Consider this: they have plenty of data from the Adwords program to help them decide which terms are "valuable." All the data they need, in fact, about the number of advertisers competing over certain terms, and how much they're willing to pay to be shown on them.

But isn't this just wild speculation? How can we know for sure?

One way is to try a few queries on the new Scroogle tool that lists how many sites on a given query have dropped out of the top 100 since the last index (I won't go into how this comparison is possible... in fact it may not be possible for more than a few days as Google is very likely to change the playing field again).

On a query for "japanese maples," 32 of the former top 100 links have dropped out of the top 100, including poor #23:

http://www.regannursery.com/also_in_stock/japanese_maples_list.htm

Meanwhile, looking for the latin name, acer palmatum, gives a very different result. NONE of the links, according to Scroogle, have changed. Google has done nothing to the existing rankings on this term.

So you might ask yourself: how the heck do they do this? How do they decide what's commercial, and what isn't?

That's easy. They can look at how many advertisers are advertising on those terms, and how much they're bidding. (Unfortunately most of those Japanese Maple sellers weren't smart enough to bid on "japanese maple," let alone acer palmatum, something they would have done had they listened to our advice!)

There's a precedent for this - it might have actually been something that inspired Google, who knows. Infospace, publisher of the Metacrawler and Dogpile metasearch engines, came out with a method of determining the degree of commerciality first, and then assessing how many paid links to show in their mix on Metacrawler as against ordinary index (unpaid) results. I'm not sure how that filter worked, but it's likely that it's not particularly sophisticated but rather, largely tied to the number of advertisers in the space and the average bid on those words.

In the case of Japanese maple, there are at least five bidders on the term, whereas acer palmatum only has a couple of bidders, one of whom is selling Acer computers. If it's only a nickel term anyway, Google figures it's noncommercial (at least for all practical purposes), so doesn't interfere.

For the term "arboretum," only two links of 100 have dropped out. There are no advertisers.

For the phrase "pick your own," zero links have changed, according to Scroogle. There are also zero advertisers.

No listings have changed for "mango chutney." There are three advertisers. Not enough to create bidding wars, likely top bid less than a quarter.

The term "medical supply," by contrast, has at least twelve advertisers, including some large multinational corporations. Listings on this phrase have been shaken up significantly, with 89 of 100 listings dropping out of the top 100. Although not a huge cash cow on the ad side, it's about $1.25 to get into second spot... just around 0.75 for third position. A bid of 25 cents will get you into 5th or 6th place.

Now -- who's the new top dog on this term? Moore Medical. Not a household name, and for heaven's sake don't look at their PageRank, because it's a 2. But they do seem to be something: relatively deep-pocketed.

"As a $130 million, publicly-held company with more than 50 years of experience, we have both the infrastructure and distribution network to serve our customers' needs efficiently and cost-effectively."

The former #1, MedSupplyCo.com, certainly seems like a decent outfit. Good site, plenty of customers, PageRank of 6. However, not as impressive in other areas:

"The Medical Supply Company, Inc. was founded in 1998. Our goal is to provide quality medical supplies at discounted prices. We work hard to keep our customers happy, and it shows. If you have any comments or suggestions for us, please feel free to drop us a line."

The new #2 site, Allegro Medical, isn't publicly traded like Moore, and indeed doesn't stand out in particular. One thing you do notice, along with their PageRank of 5, is that they're a Google advertiser.

So maybe Google's process is essentially a two-stage one:

1. Target "commercial" queries for a "roiling of the waters" - re-rank all sites falling under certain "commercial" queries, depending on the perceived value of that term as measured by its value within the AdWords program;

2. Attempt to make a global judgment as to which of the following "types" a site or page falls into:

• informational/discussion/educational/governmental
• navigational/search/portal
• company
• store
• affiliate seller
• other

It has to be something like that. It's certainly no mere "speculation" that Google is up to something along these lines. There is enough evidence that points to certain patterns over and over again. The shortest path to the truth seems to be the following rule: "where there is a critical mass of advertisers, Google has chosen to re-rank the index."

Admittedly, this is an oversimplification at best. Certainly there are other characteristics of the re-index that look more like past spam blitzes: sites which have aggressively pursued link swaps or keyword-rich domains, for example, have apparently been big losers in the latest blitz. Armchair sleuths are busy at work trying to unravel what it all means. I suspect there is more continuity than we would like to admit. Google is doing what search engines have been doing for years: studying common SEO techniques and trying to ensure that clever marketers don't get the upper hand in the "free" index. What's different is how much is at stake if Google can indeed dislodge the best-laid plans of free-riders at this crucial stage. Next time, maybe they'll think twice about shunning the advertising program at this time of year.

I think we should, at least, put to rest the idea that there is any overt or covert "ranking reward" for being an advertiser. The re-ranking is based on principles that may have been espoused within Google and all search engines for years. We've seen trends towards giving more top-ten listings to sites that involve discussion, comparison, content, resources, etc.. It's just been accelerated now, and made more aggressive.

Another revenue angle to consider: Google now actually stands to make money from quality content sites which get ranked well. That's because many of these sites now show AdSense ads, and Google gets a little revenue share from them, too.

It's ingenious, really. Google has figured out how to get paid much more than the zilch they used to get paid for running a search engine, whether users click through to commercial listings or quality content, and yet without assaulting users with irrelevant, commercialized search index listings. Quite the opposite, actually. Arguably, relevancy is just as much in evidence as it always was in the main index. The listings for "acer palmatum" and "icthyology" haven't budged.

So does this mean SEO is dead? Far from it. Responsible search marketers have always been able to generate a mix of stable top 20 rankings on popular terms (by doing PR the hard way and generating real, not manufactured, online word-of-mouth), along with a healthy mix of top five rankings on less popular terms, by highlighting rich, varied, related content that intersects with the highly specialized search queries users type in all the time. The techniques for doing that won't be "caught," because there is nothing wrong with having rich, unique, varied content. (Or products that hardly anyone else sells!) Advertisers who have always struck the right balance between paid and unpaid listings, between interesting content and company information, and commercial pages, won't have been caught flat-footed by this latest diabolical Google reshuffle.

But plenty of cocky search-marketing cowboys -- those who felt a growing sense of entitlement to rankings they "earned" for their sites on the strength of little more than a few cheap parlor tricks -- have been knocked to the dirt hard in this latest dustup. Most will pick themselves up, knock back a couple of shots of the hard stuff, and prepare themselves to fight again. But some won't. Some of Google's most notorious freeloaders may finally gather their things, put their crumpled black hats on, and ride unsteadily off into the sunset, never to be heard from again.

That's the hope, anyway.