Thread: Robots.txt Question

I have recently noticed that when I look at our indexed pages in Google urls are starting to show up with the https:// prefix. This is causing pages to show up twice now one for http:// and one for https://

I looked all over my site trying to figure out how Google got to a link with https:// to start with. I realized it was coming in from the ShoppingCart.aspx page. The default robots.txt file you put with the cart disallows /shoppingcart.aspx, but the robots.txt file is case sensitive. I changed it to the correct case and now Google can not get there.

Now I need to try and get those pages that were indexed with https:// de-listed. I did some research and this seems to be the best method to get this worked out:

Use url rewrites to 301 redirect the https pages to the http pages.
OR
1. Add NOINDEX, NOARCHIVE to all https pages.
2. When google crawls the pages they'll drop out
3. Create a robots.txt file for secure and non-secure pages. Disallow all bots on the ssl one.

I am not sure how to use the ISAPI rewrite to do this. Anyone know how to do it that way? The rewrites I use are setup in this fashion:
RewriteRule /old-page.aspx /new-page.aspx [I,RP]

If I put this it does not work:
RewriteRule https://www.site.com/old-page.aspx http://www.site.com/new-page.aspx

The second solution listed I am not sure how to add the meta tag NOINDEX, NOARCHIVE to just the https pages. I have this code, but don't know how to get it put into the templates:

if (Request.ServerVariables["HTTPS"] == "on") {
Response.Write("<meta name=\"robots\" content=\"nofollow,noindex,noarchive,nocache\" />");
}

add a Literal control to the head section of your template.ascx of your skin.

In App_Code\TemplateBase.cs add your check in the Page_Load method, setting the Text property of the Literal control if you need to supply the meta information.

Originally Posted by ASPDNSF Staff - Rex

add a Literal control to the head section of your template.ascx of your skin.

In App_Code\TemplateBase.cs add your check in the Page_Load method, setting the Text property of the Literal control if you need to supply the meta information.

Do I need the source code to do this?

The file template.ascx is located on your sites Skin folder while the TemplateBase.cs is located on the App_Code folder..

I have the literal control placed in my template file, but changing it in the TemplateBase.cs file is not working out for me.

I have just a basic if test I put in there:

HTML Code:

if (Request.ServerVariables["HTTPS"] == "on")
{
  Literal1.Text = "name='ROBOTS' content='NOINDEX, NOFOLLOW, NOARCHIVE'";
}

Doing that gives me this compilation error:

HTML Code:

Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately. 

Compiler Error Message: CS0103: The name 'Literal1' does not exist in the current context

Doesn't sound like you added the Literal to the template correctly. Try sharing that bit with us.

Originally Posted by ASPDNSF Staff - Rex

Doesn't sound like you added the Literal to the template correctly. Try sharing that bit with us.

Here is what I have in my template file:

HTML Code:

<meta <asp:Literal ID="Literal1" runat="Server" Text="name='ROBOTS' content='INDEX,FOLLOW'"></asp:Literal>>

that's not going to work like that.

You cannot hide a tag inside a tag.

You need

Code:

<asp:Literal ID="Literal1" runat="server"/>

And in your code:

Code:

Literal1.Text = "<meta blah blah />";

Originally Posted by ASPDNSF Staff - Rex

that's not going to work like that.

You cannot hide a tag inside a tag.

You need

Code:

<asp:Literal ID="Literal1" runat="server"/>

And in your code:

Code:

Literal1.Text = "<meta blah blah />";

Okay I set it up this way and I still get this error:

HTML Code:

Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately. 

Compiler Error Message: CS0103: The name 'Literal1' does not exist in the current context

Originally Posted by ASPDNSF Staff - Rex

add a Literal control to the head section of your template.ascx of your skin.

In App_Code\TemplateBase.cs add your check in the Page_Load method, setting the Text property of the Literal control if you need to supply the meta information.

This is how my code is currently:

TemplatBase.cs (Page_load method)

Code:

if (Request.ServerVariables["SERVER_PORT_SECURE"] == "1")
{
	Literal1.Text = "<meta name='ROBOTS' content='NOINDEX, NOFOLLOW, NOARCHIVE'>";
}
else
{
	Literal1.Text = "<meta name='ROBOTS' content='INDEX,FOLLOW'>";
}

home-template.ascx

Code:

<asp:Literal ID="Literal1" runat="Server" Text=""></asp:Literal>

I created a test page that had nothing on it except the code from above and it worked properly.

I am not sure what is causing it to mess up when I put it into the storefront code.

Rizzy,

We have the exact same problem with our ASPDNSF site (ShoppingCart.aspx instead of shoppingcart.aspx inside the case-sensitive robots.txt file, causing many pages to show up as https in the rankings and thus causing a security certificate warning/error when clicked).

Did you ever have success placing the "nofollow noindex no archive" meta tag on the secure pages?

Hi peps,

Like Bartman i'm also interested in a solution. Was this ever resolved? If so could the solution be posted on here pleaseeeeeeeee !

Kind regards,

Anyone figure this out?