One of the problems of publish SharePoint in a public website is that the internal webpages with sensitive content will be indexed. To prevent this we can set an a IIS redirection rule, but if we want to make it simple, we can make a file config in the root file to prevent to the search robots to not to crawl certain content. To do this, we can create a robots.txt with the following:
You can get more info from a previous post: https://sharepointrescue.wordpress.com/2015/05/20/what-is-robots-txt/
Till next time!