Every week, I receive a few questions from people who have dynamically generated sites and are having trouble getting them spidered and ranked in the major search engines. I've written about this subject before, as it's been a long-standing problem in the SEO world. Instead of simply reiterating my thoughts on the subject, I thought it would be smart to provide you with some information from someone with a more technical background than I have. Who better than my colleague, Alan Perkins?

Alan has been working with search engines since 1995. He holds patents in search engine technology and was lead developer of Search Mechanics, a product to help webmasters make their sites more search-engine-friendly. Alan is also the co-founder of e-Brand Management <http://www.ebrandmanagement.com/>, a company dedicated to helping people build and maintain a successful online presence.

Alan and I are currently working together to optimize a dynamically generated Web site for one of my clients. It's been quite a learning experience for me (and the client), and there's a whole lot to it. The good news is that dynamic sites *can* be optimized to be found in the search engines, but you do have to know what you're doing in order to make it happen.

So without further ado, here's my interview with Alan: --- Jill: Can you explain what dynamic content and dynamic URLs are?

Alan: The terms dynamic URL and dynamic content are frequently used interchangeably. However, this can lead to confusion because they are two separate, but related, terms. A URL is not content - a URL is the address of some content.

Dynamic content is information that is delivered to the Web browser in a different form than it exists on the server. It is usually pulled from a database and created on the fly at the server level through CGI programming, ASP, PHP, or by a content management system such as BroadVision(tm) or ATG Dynamo(tm).

Dynamic URLs, on the other hand, are simply Web site addresses that contain a question mark (?).

In contrast, static content is stored on the Web server in the same format that is delivered to the Web browser. And static URLs do not contain question marks.

In general, dynamic URLs are addresses of dynamic content, and static URLs are addresses of static content. However, this need not be the case, as we shall see later.

Jill: We often hear that search engines have a problem indexing dynamic content; why is this?

Alan: It boils down to two issues -- the same core content seen at different URLs, and different core content seen at the same URL.

When the same core content is at different URLs, a small site can appear to be very large because an unlimited number of URLs can be used to provide essentially the same content. Spiders can fall into "dynamic spider traps," crawling through thousands of URLs when only a few really needed to be crawled. Since a dynamic URL usually indicates dynamic content, the simplest way for a search engine to avoid these spider traps is to avoid dynamic URLs altogether. Remember, search engines want to index any given core content just once.

Now let's consider different core content at the same URL. There are a number of ways in which this might happen. For example, a site may have content that may be viewed at the same URL in multiple languages depending on the browser settings. Another example would be content that gets updated every few minutes or so.

Whatever the means, search engines typically index only one copy of a specific URL once every few weeks or so. Therefore, if a search engine indexes your English content at a given URL, the same search engine will not index your Spanish content at the same URL (during the same indexing period). And if your content is frequently updated, the search engine's copy of your content will not be fresh. A search engine prefers that the visitors to a particular URL see the same content its spider saw.

Jill: Sounds like sites with dynamic content have an uphill climb when it comes to the search engines. So what can we do to help them get indexed?

Alan: The general answer is to give each search engine what it wants: unique core content at a unique URL, plus the same core content seen by all visitors.

But I'm guessing you want specifics. So here they are!

1. Use static URLs to reference dynamic content.

If a search engine sees a static URL, it is more likely to index the content at that URL than if it found the same content under a dynamic URL. Therefore, you can turn your dynamic URLs into static URLs despite the fact that you are serving dynamic content. There are a number of ways of achieving this, and your method will vary depending upon your server and other factors. To go into all of these methods is beyond the scope of this interview; however, you can visit the following sites for two popular servers:

Apache: <http://httpd.apache.org/docs/mod/mod_rewrite.html> ASP: <http://www.asp101.com/articles/wayne/extendingnames/>

2. Link to dynamic URLs from static URL content.

With limited resources, it may prove difficult or impossible for you to implement a solution based on static URLs. Don't worry! There are other things you can do.

Over the years, the engines have tried to find ways of crawling dynamic content while avoiding dynamic spider traps. One technique they use is crawling dynamic URLs that are linked to from pages with static URLs. For example, if you give your site map page a static URL, but have links to dynamic URLs within its content, there's a good chance that the leading engines will crawl those links. If they like the content they find there, they will index that content. The search engines' reasoning here seems to be, "If you're prepared to link to this content, then so are we."

You can reinforce this reasoning by negotiating links to your dynamic URLs from pages on other sites (especially high-quality pages which are already indexed). Again, the search engines' reasoning here is "If other sites are prepared to link to your site, then so will we." If others won't link to your dynamic content, that might give you some idea why search engines won't either! If it proves impossible to get links to your dynamic content from other sites, then you can't expect a search engine to link to your site either.

3. Pay for inclusion whenever possible.

AltaVista, Ask Jeeves/TEOMA, FAST and Inktomi offer one or more means of paying for individual URLs to be spidered. You can use these paid-inclusion programs to get your dynamic URLs indexed. Paid-inclusion programs only affect inclusion and do not influence ranking, so it is still important to make sure your dynamic content is well optimized. For more details see the Add-URL pages of the respective search engines.

Conclusions:

1. Search engines have problems creating links to dynamic content.
2. If you can recognize these problems, you are halfway to getting your dynamic content indexed.
3. Where practical, use static URLs to reference dynamic content.
4. Otherwise, try to ensure your dynamic URL is linked to by content referenced by static URLs.
5. Consider using paid-inclusion programs.