Do you show Google Webmaster Tools with a blocked Robots.txt file? I had this problem this week when I was ready to index a couple of new blogs I've been working on.
I set “site visibility in the privacy section of my WordPress settings to “Ask search engines not to index this site” and then switched to “allow search engines to index this site” after I built out my first content. I then went to Google Webmaster Tools and set up an account. Immediately, the Robots.txt file showed as blocked. I began to try all kinds of fixes to get it to show up, including doing a complete reinstall of one of my blogs, while leaving the other alone for comparison.
I then searched the web, including the forums at WordPress.org and found answers all over the map, from sources that we're trying to help, but didn't seem to give a solid answer. Several people did have the proper solution, but in every case I found, none of the people with the issue reported results back.
Many of the threads were a year, sometimes three years out of date. I became concerned that the solution was no longer valid, as I had downloaded the latest version of WordPress in this month, July 2012. I also set my site to block search engines from the setup files from WordPress for a manual install, an option that is not available in automatic installs.
Many hosting companies provide automatic installs. Bloggers are able to set “site visibility” in the privacy section of the control panel in this case.
I spent much time on this project only to find out that Google doesn't look at your Robots.txt file on your website, they look at the one that is cached at Google. You have to wait for cache to refresh, which in my case took twenty-four hours. Now, I'm showing fine on both sites.
If you are having this problem, set your “site visibility” to “allow search engines to index this file” and then wait twenty-four hours to set up Google Webmaster Tools. You'll save yourself hours and hours of work and frustration.