Elasticsearch Connection Refused

Chris_Gaber · November 3, 2021, 2:59pm

When trying to run a reindex I get the following error in studio:

2021-11-03 14:50:29,721 WARNING 34 [elasticsearch] [user 8] [ip 172.18.0.1] base.py:293 - HEAD http://localhost:9200/courseware_content [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    response = self.pool.urlopen(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fdb7be12310>: Failed to establish a new connection: [Errno 111] Connection refused        
2021-11-03 14:50:29,722 WARNING 34 [elasticsearch] [user 8] [ip 172.18.0.1] base.py:293 - HEAD http://localhost:9200/courseware_content [status:N/A request:0.000s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    response = self.pool.urlopen(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fdb7be12820>: Failed to establish a new connection: [Errno 111] Connection refused        
2021-11-03 14:50:29,723 WARNING 34 [elasticsearch] [user 8] [ip 172.18.0.1] base.py:293 - HEAD http://localhost:9200/courseware_content [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    response = self.pool.urlopen(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fdb7be128b0>: Failed to establish a new connection: [Errno 111] Connection refused        
2021-11-03 14:50:29,725 WARNING 34 [elasticsearch] [user 8] [ip 172.18.0.1] base.py:293 - HEAD http://localhost:9200/courseware_content [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

I run curl and get the following:

cgaber@Desktop-Chris:~/work/devstack$ curl "http://localhost:9200"
{
  "status" : 200,
  "name" : "Milan",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

I am unsure what is causing this, any ideas?

Zia_Fazal · November 4, 2021, 8:54am

Edx is upgrading to Elasticsearch 7 and as result of it name of ELASTIC_SEARCH_CONFIG setting has changed to ELASTIC_SEARCH_CONFIG_ES7 make sure you have set value of this setting in your /edx/etc/studio.yml something like this

ELASTIC_SEARCH_CONFIG_ES7:
-   host: edx.devstack.elasticsearch710
    port: 9200
    use_ssl: false

assuming your elasticsearch docker container name is edx.devstack.elasticsearch710

Neo · February 10, 2022, 5:18pm

Hi @Zia_Fazal , I’m having a similar issue trying to refresh the metadata for Discovery service(lilac version). I was wondering if may be I need to include some variable similar to mentioned in your response. Thanks in advance for your support

Zia_Fazal · February 11, 2022, 10:02am

Hi @Neo I think in case of course discovery you should make sure ELASTICSEARCH_DSL for dev and ELASTICSEARCH_CLUSTER_URL for production are pointing to Elasticsearch host. I do’nt think there is a separate setting for ES7 in course discovery.

Neo · February 11, 2022, 2:20pm

Thank you so much @Zia_Fazal!! , ELASTICSEARCH_CLUSTER_URL variable works as you mentioned!!. Do you know if I need to run some kind of reindex for Discovery after that, since I’m getting this error.

elasticsearch.exceptions.NotFoundError: NotFoundError(404, ‘index_not_found_exception’, ‘no such index [course_run]’, course_run, index_or_alias)

Thank you so much for your support!!

Zia_Fazal · February 11, 2022, 3:57pm

@Neo yeah you need to run update_index command to create or update Elasticsearch index with the latest data in course discovery

$ ./manage.py update_index --disable-change-limit

More details can be found here

Neo · February 11, 2022, 4:12pm

Great!! @Zia_Fazal I will do that, thanks a lot !!!

hijulien-gan · June 22, 2022, 2:46am

Where is the file you said? I successfully ran my tutor, version 13.2.0. I encountered the same problem, but I didn’t find the file you said

Ievgenii_Vasiuk · July 21, 2024, 10:38pm

Hi, I’m encouraging the same error with ecommerce, deployed using recent tutor installation. curl works fine at the server to the https://ecommerce.my.host.name, but from the web for same url the connection is refused (e.g. at https://ecommerce…/accounts/profile/ or /courses/new/ add). Any ideas would be highly appreciated

Topic		Replies	Views
Elastic search configuration issue 'https://****.privatelink.centralindia.azure.elastic-cloud.com' ([Errno -2] Name or service not known) Tutor Help tutor	0	16	December 2, 2024
An error occurred when searching for “Zen and the Art of Motorcycle Maintenance” Site Operations Help	12	422	September 22, 2022
Open edX elasticsearch.exceptions.RequestError: RequestError(400, Tutor Help how-to , tutor	1	362	July 23, 2024
Elasticsearch: maximum shards open Site Operations Help	3	4933	October 3, 2022
Reindex Button or Search boxes not found Site Operations Help	12	1497	May 29, 2020

Elasticsearch Connection Refused

Related topics