# Search engines and robots.txt
**URL:** https://discuss.openedx.org/t/search-engines-and-robots-txt/11899
**Category:** Site Operations Help
**Tags:** how-to, tutor
**Created:** [December 11, 2023, 2:43am UTC](https://discuss.openedx.org/t/search-engines-and-robots-txt/11899 "2023-12-11T02:43:42Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Rick\_Sauer](https://discuss.openedx.org/u/Rick_Sauer)
#### Post date: [December 11, 2023, 2:43am UTC](https://discuss.openedx.org/t/search-engines-and-robots-txt/11899/1 "2023-12-11T02:43:42Z")
Hi all, I’m looking for a way to have a robots.txt implemented so google doesn’t scrape my dev sites. I have found some other short answers, but weren’t very helpful. How are others achieving this?
---
### Author:  [@jramnai](https://discuss.openedx.org/u/jramnai)
#### Post date: [December 12, 2023, 4:39pm UTC](https://discuss.openedx.org/t/search-engines-and-robots-txt/11899/2 "2023-12-12T16:39:01Z")
Hello @Rick_Sauer and welcome to Open edX community!
I’ve written a blog post to achieve the same, which you can find here and refer:
[How to add a robots.txt file in Open edX](https://jramnai.hashnode.dev/how-to-add-a-robotstxt-file-in-open-edx)
I hope this will help!
---
### Author:  [@Rick\_Sauer](https://discuss.openedx.org/u/Rick_Sauer)
#### Post date: [December 12, 2023, 11:27pm UTC](https://discuss.openedx.org/t/search-engines-and-robots-txt/11899/3 "2023-12-12T23:27:48Z")
Thanks for the welcome!
Article is great and can see where the file goes, the step I’m not 100% sure about is
Go to the `edx-platform/lms/urls.py` file and add the following code
Currently running Tutor version 16.1.4
---
### Author:  [@jramnai](https://discuss.openedx.org/u/jramnai)
#### Post date: [December 14, 2023, 6:23am UTC](https://discuss.openedx.org/t/search-engines-and-robots-txt/11899/4 "2023-12-14T06:23:30Z")
> the step I’m not 100% sure about is Go to the `edx-platform/lms/urls.py` file and add the following code
This blog was written pre-tutor era.
Either you can edit your `lms/urls.py` file directly or you can create plugin of Open edX and add your URL pattern in LMS. You can find plugin creation guide [here](https://blog.lawrencemcdaniel.com/getting-started-with-open-edx-plugin-architecture/).
The code is pretty simple and just adds your robots.txt file’s URL pattern, using this Django project can locate and render the file.
---
### Author:  [@system](https://discuss.openedx.org/u/system)
#### Post date: [March 13, 2024, 6:24am UTC](https://discuss.openedx.org/t/search-engines-and-robots-txt/11899/5 "2024-03-13T06:24:30Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.