Meet Daniel Brandt. He is a self-proclaimed public interest activist and the owner of Google-Watch.org Mr. Brandt founded Google-Watch.org after his own site, Namebase.org, did not get a good Google PageRank. Promoting his site or putting more work into it apparently never entered Mr. Brandt's mind, instead he decided that the fault wasn't with his site, but rather with Google. Thus he has started a campaign that badmouths the search engine and complains that its ranking algorithms are unfair. Unfair to whom? Well to Mr. Brandt's site that's who.
Unfortunately for Google, Mr. Brandt has found an audience. Increasingly journalists that are looking for a sensational story are using him as a source. They rarely bother to check the validity of his claims, and in a few cases they don't even mention the selfish motives that started this whole campaign, instead they paint him as some kind of benevolent consumer activist who isn't out there to make a buck. Google is just a shining star, and it's fun for some people to try to tarnish them. It gets readers anyways, just like tabloids.
The first such article I read was published by Salon.com and can be found here (http://www.salon.com/tech/feature/2002/08/29/google_watch/print.html). This article isn't so bad as the others, the author does expose Brandt's motives:
Brandt is not a disinterested party; the dispute between Daniel Brandt and Google is personal. He has spent thousands of hours building a Web site that he believes is both useful and important, and Google, in its algorithmic blindness, has given Brandt a lower page rank than he thinks he's entitled to. Brandt finds it genuinely hard to believe -- and even personally insulting -- that Google won't give him more credit.
However the article does paint Mr. Brandt in a friendly light, and so it lends credibility to his cause - a cause which most if not all search engine professionals would regard as absurd.
The crux of Brandt's argument can be summed up with these paragraphs from the Salon.com article.
When you type "NameBase" into Google, Brandt's site comes up first, but Brandt is not satisfied with that. "My problem has been to get Google to go deep enough into my site," he says. In other words, Brandt wants Google to index the 100,000 names he has in his database, so that a Google search for "Donald Rumsfeld" will bring up NameBase's page for the secretary of defense. For some reason, though, all of NameBase's deep pages -- its pages with specific names and citations -- have a low Google page rank, which causes them to show up low in the search results. Search for "Donald Rumsfeld" in Google and in the first five pages you get a lot of .mil and .gov sites, some news stories, and some activist sites. Namebase's entry on Rumsfeld doesn't come up. (It is in Google's database, but to find it somebody would have to first wade through hundreds of results.)
Brandt sees this as Google's major flaw. "I'm not saying there aren't some sites that are more important that others, but in Google the sites that do well are the spammy sites, sites which have Google psyched out, and a lot of big sites, corporate headquarters' sites -- they show up before sites that criticize those companies."
In other words, Brandt recognizes that there has to be some order to Google's results, and that some sites might deserve to come up before others. He just disagrees with the way Google does it. In Brandt's ideal world, if you searched for "United Airlines," you would see untied.com -- a site critical of United -- before you see United's page. And if you searched for Rumsfeld, you'd see NameBase's dossier on him before the Defense Department's site on the "The Honorable Donald Rumsfeld."
In truth, Brandt's problem is not Google, it is that his views are not the views of the public at large.
You see PageRank is not given by Google, it is given by people - specifically people that run websites. In a way PageRank is both a direct democracy and a representative democracy. If you run a website every site you link to is getting a direct vote from you, thus representing a direct democracy. Additionally you are giving weight, through that link, to the site which can then pass that weight, in the form of a vote, on to a third site, thus representing a representative democracy. So when one webmaster links to a site they are both voting for that site and allowing that site to use weight received to vote for other sites.
Google also looks at the context of incoming links. So if someone links to Burgerking.com using the word "hamburgers" then they are voting for Burgerking.com as an authority on the word "hamburgers." Additionally they could do the same thing to McDonalds.com. If Burgerking.com gets more such "votes" than McDonalds.com then they will likely, though not always due to on page factors like page title, body text, and the rest, be ranked higher if you search for "hamburgers."
Before Google, search engines relied on the text of a page itself to rank it. This was a problem because it put too much control in the hands of a webmaster. If you wanted to rank highly you only needed to tweak your own site. PageRank changed all this, now how you rank depends on the opinions others have of you. Sure, particularly large sites can indeed influence the rankings of sites owned by the same company, but that is where the representative democracy comes in. The parent site may have gotten all of it's weight from third parties and then passed some of that on to the child site. The end result is that the origin of the weight is third parties and so the representative democracy is working.
Today being ranked highly in search engines is more about what people think of you than what you think of yourself. Mr. Brandt's problem is that people do not think very highly of him. It could be the fact that his leftist views are not popular. It could be that his site does not appear to be trustworthy (the rainbow color scheme doesn't present a feeling of accuracy or academia). Whatever the reason is when people think of Donald Rumsfeld they think of the Department of Defense, which he runs, they think of news stories on places like CNN.com. They do not think of Mr. Brandt's site - especially when to find information on Donald Rumsfeld you need to mine pretty deep.
The depth is another reason why Mr. Brandt's site performs poorly. PageRank is like a pizza, the more people that need to eat the smaller each slice will be. With the thousands and thousands of entries Mr. Brandt has his PageRank becomes so diluted that each individual page is poorly ranked. If he wanted to make a site that specifically focused on Donald Rumsfeld he would have more success, instead he is focusing on thousands of people. It shouldn't be any wonder that his individual content pages, if you can find them, rank poorly. The fact of the matter is that sites that focus on one topic should get ranked higher on that topic than sites that focus on a wider variety of topics.
The fact is Google is neither biased towards big business nor towards activists. Google is completely objective and how they rank is entirely dependent on the views of the public.
For instance search on Google for "International Library of Poetry" and while the first listing is for the company itself the far majority of listings, including the ones immediately below the first listing, are for sites that expose the company as a scam. Why can these activist sites be ranked so highly when others are not? It has to do entirely with public perception. If the majority of the public thinks something then it will be reflected in Google's results. In this case the majority of the public thinks that the company scams people and so the views of the public are reflected in Google's search results.
If you search for "Royal Prestige" you will find a news expose by ABC News that target's the company's questionable sales tactics. Likewise if you search for "Paypal" the first listing will be for the company's homepage but the next three listings are for sites that are "anti-paypal."
Bill Machrone(http://www.pcmag.com/category2/0,4148,2653,00.asp), a columnist for PC Magazine and obviously not a search engine expert has also parroted for Mr. Brandt. Only he didn't explain the real motives behind Mr. Brandt's crusade. (The next issue he did print something of a correction after an email from Danny Sullivan of SearchEngineWatch) Bill mostly follows the line about Google not giving new content good rankings. This is completely false. New content on high ranking sites is not only often indexed within days, or something hours or minutes, but it is also ranked highly. This is why if you search on Google for something related to late breaking news you can often find news stories that are from that same day listed in the search results. What Google does is they detect if a site is frequently updated and then they crawl that site at more frequent intervals. It has been said that Google crawls CNN.com almost constantly. Google will then list this "fresh" content in it's search results using a guessed PageRank based on the site the content is from. Google cannot give it an actual PageRank since PageRank is entirely relative and so they'd need to reprocess it for the entire web, which is very time consuming (weeks).
What Google doesn't do is give entirely new sites high rankings, and why should they? New restaurants don't automatically become popular, they need to advertise and even then it may take awhile. This is why franchises are so popular. It's much easier to buy a McDonald's franchise (if you have the cash) than to open a no-name burger joint. So if you're a new site and you want traffic you'll need to advertise, just like if you were opening a new business. You wouldn't expect your new restaurant to succeed instantly without any promotion so why would you expect that of a new website?
This is just the way the world works, and it's the right way. Sites with more money do have an advantage, but then again sites with more money also tend to have better content. Its not necessary businesses either, places like universities often have popular websites, but they also have million dollar budgets. Sure, this is not all encompassing, but in general the more money a site has the better it is.
This isn't "Field of Dreams" if you build it they won't come. You need to advertise, you need to promote. If you do not do these things, if you're unwilling to put the time, effort, and money into making your site successful it isn't the fault of the search engines.
The Bottom Line
PageRank isn't perfect, but it is the best thing we have. If there was a way for Google to pool the opinions of everyone in the world on every website on the Internet we'd likely get better results. If Google was telepathic and could read your thoughts when you searched we'd likely get better results, but imagine the privacy issues with that one.
Google discovered a way to efficiently gather an approximation of public opinion on every web page on the Internet. It is an approximation but at this point in time anything other than an approximation would not be technically possible. Other search engines have now employed this same concept so even if Google lost popularity, whatever search engine took their place would serve very similar results.
The bottom line is that Google is nothing but a mirror of public opinion. When people think of hamburgers do they think of Burger King or McDonalds? Google can tell you. If your content isn't ranking as high as you think it should it is probably because the public doesn't agree with it or doesn't think very highly of it. If you don't like what you see when you look in the mirror, don't blame the mirror.