Jonathan Richards background as an e-commerce entrepreneur and SEO consultant to advertising and sales agencies, eventually led him to become THAT Agency's primary internet marketing strategist in mid 2012.
When launching a site with a new internal structure and URL structure, we use 301 redirects to tell search engines that the page has permanently moved to a new page. This page should be the “new version of the page”. Failing to do so will result in the common 404 error. Sometimes, even if you have 301 redirects in place you will still get whats called a “Soft 404″ error. The best practice for redirects can be summed up simply as follows:
“A page about dogs should be redirected to a new page about dogs.”
For small sites it is not difficult to set manual redirects in the htacess. For large sites that need thousands of redirecting pages, this can usually be done fairly efficiently with “URL rewrite rules” which match the old URL structure to the new structure as efficiently as possible. Sometimes, these redirects may not match with 100% accuracy and very specific pages may be redirected to higher level broader categories. i.e. page about dogs gets redirected to new page about animals for lack of a better page.
In most cases this redirect is will work as intended although it may not pass 100% of rankings to the target page. Sometimes, Google will not recognize the redirect because they believe it to be not relevant enough. This triggers a soft 404 error. This means that the old URL is not live AND the redirected page is not returning a 404 error.
To fix you may want to improve upon the logic used in the “URL rewrite rules” or in some cases set manual redirects for individual pages where logic is unable to make the redirect.
Some common errors that cause of soft 404’s are redirecting hundreds of old pages to the homepage or similar high level category within the site. All of these old pages can’t possibly live on a single page. If there is no new target page on the same topic, then it is sufficient to let the page return a 404 error. A good overview of acceptable redirects can be found in Google Webmaster Guidelines.