I wonder if this topic would give you a starting point: Is there an easy way to add a robots.txt to a tutor-based OpenEdX deployment? - it discusses how to use a robots.txt file to avoid crawling the whole site, which I realize is not your goal, but it could be useful? I don’t know how to answer your question but figured I’d point you in a potentially helpful direction.