cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Go to solution
New

Robots.txt file is not appearing in the public_html directory

Hi there,

 

I've got a quick question regarding the robots.txt file.

 

The Robots.txt file is available on the website. We can see it. However, when I login to CPanel, the robots.txt file isn't showing up in the public_html directory.

 

The settings are already set to "show hidden files". Also, I know for a fact that there's a robots.txt file in the CPanel, but for some reason, it just isn't showing up.

 

Please help me. Thank you in advance.

2 ACCEPTED SOLUTIONS

@mixsky 

 

1) Yes you are on a cPanel server then - that is fine

2) I just checked on my own site and I don't have a  robots.txt file either which means it is auto generated by wordpress. I believe if you create the file in Yoast it will create a file which will override the default one.

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

View solution in original post

@PL281 

 

Got it. Thank you so much. I appreciate your time. You've been really helpful. Thanks a lot.

View solution in original post

16 REPLIES 16
Super User IV

@mixsky 

 

The only thing I can think of is the website is in a subfolder or different folder than the public_html

 

If you can get to http://yourdomain.com/robots.txt then it will show in the file manager

I would look at your "domains" sections in cPanel and confirm the path for that domain to find the exact folder.

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

Thank you for your quick response @PL281. I really appreciate it.

 

The website is in the public_html directory itself. This is true for sure.

 

When we click on the website directory within the public_html directory, there are usual folders like "cgi-bin, wp-admin, wp-includes, wp-content" and just below them there should be the robots.txt file. However, it isn't there. I just don't get a hang of it.

 

Btw, how would you look at my "domains" sections in cPanel and confirm the path for that domain to find the exact folder? How would you get the access?

@mixsky 

 

When you are in cPanel - there is a section for Domains / Add-Ons / Redirects 

If you click on Domains or Add-ons, it then shows the path that the domain is mapped to

 

Can you post a screen capture of the file manager

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

Hi @PL281,

 

In the cPanel, I didn't find any section for domains / add-ons / redirects.

 

Below is what I can see in the cPanel left-hand side pane.

 

snip2.PNG

Does that help?

@mixsky 

 

1) Need to see inside the public_html folder

 

2) The domains are back in the main cPanel area - not the file manager

 

Screen Shot 2019-11-06 at 12.11.39 PM.png

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

@PL281

 

I just checked in the admin area and there is a section for Domains. And I've checked it the website directory is mapped in the public_html directory.

 

Here's a look at the inside of public_html directory.

 

11.PNG

 

For the website directory 1906lodge (marked with Red), the Robots.txt file isn't appearing.

@mixsky 

 

https://www.1906lodge.com/robots.txt so yes this loads

So if you click into that folder you are saying it isn't there - are you running a plugin like yoast or something else that might be generating the file dynamically?

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

@PL281 

 

Yes, the robots.txt file isn't available there. If you're on Skype, I can share my screen and show you that the file isn't showing up in the directory.

 

Yes, we use Yoast plugin. However, it isn't created using the plugin. Please see below.

 

11.PNG

 

Please help.

When we create using the Yoast plugin, the button "Create robots.txt file" doesn't show up anymore.

@mixsky 

 

Right now I'm not available to do a screen sharing session - 

Question for you....do you have SSH access to the account - One issue that maybe is that someone else created the file and it's in the directory but due to ownership issues you are not seeing it in the web based file manager

 

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

@PL281 

 

I'm not sure about the SSH access. It's a delegated access. How do I find out if I'm given the SSH access?

 

That could be a reason. However, is it really possible that I can see the whole website in the cPanel (including the .htaccess file and what not) and only the Robots.txt file is having the ownership issue?

@mixsky 

 

Sure it is possible as it depends how / who setup the file in the first place - delegated access shouldn't matter - and just to confirm this is a cPanel server and not just a cPanel account on a VPS in the account you are working in

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

@PL281 

 

Then, what should I do now?

 

I don't understand what you mean by "cPanel server and not just a cPanel account on a VPS in the account you are working in".

 

However, I'm not on a VPS.

 

I'm logged in to cPanel using Godaddy itself. There's a section "web hosting". And under that section, when we click on "Manage", we're redirected to cPanel (logged in automatically).

@mixsky 

 

1) Yes you are on a cPanel server then - that is fine

2) I just checked on my own site and I don't have a  robots.txt file either which means it is auto generated by wordpress. I believe if you create the file in Yoast it will create a file which will override the default one.

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

View solution in original post

@PL281 

 

#1. Ok, sounds good.

 

#2. So, if the robots.txt file created by Yoast can replace it then this should also work. What if I create the robots.txt file manually and upload it into the respective directory? This should also serve the purpose, right? Please correct me if I'm mistaken.

@mixsky 

 

You are correct 

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

@PL281 

 

Got it. Thank you so much. I appreciate your time. You've been really helpful. Thanks a lot.

View solution in original post