Breaking this into three questions:
Why doesn’t edx-search require the Discovery service?
Because the Discovery Service isn’t enabled by default, but the /courses page needs to work out-of-the-box on a new instance. So, edx-search can only depend on LMS/Studio by default.
Why doesn’t edx-search use the Discovery service when it IS enabled?
It would make sense if edx-search could be configured to use the Discovery service when the service is enabled. I believe the only reason this isn’t the case is because of history.
The /courses page was written in ~2013, and edx-search was created in ~2015. The Discovery service didn’t come around until ~2016 (I’ve realized that the “course_discovery” references in edx-search aren’t a reference to Discovery service; it’s just the general idea of discovering courses. The service didn’t exist yet).
By now, edx.org hasn’t used the LMS /courses page at all for a long time: instead, they use a closed-sourced marketing site, which uses the Discovery service as a backend. So, edX/2U doesn’t think about the edx-search library very much. The /courses page in particular is entirely in the hands of the community.
Can we replace the edx-search usage with course_discovery ?
Unless we require all Open edX instances to run Discovery, I don’t think we’d want to replace the usage. But providing the Discovery service as an alternative backend sounds like a great idea. I don’t think edX/2U would be interested in the change, but as long as it didn’t introduce any regressions, I can’t imagine they’d oppose the change either.