Installing INSIGHTS: Database Migrate Error

I´m getting this error while following the official guide for the OpenEdx Insights Installation

The process stops at the TASK [edx_django_service: migrate database]

fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: [“make”, “migrate”], “delta”: “0:00:00.822474”, “end”: “2020-06-20 02:51:24.188051”, “msg”: “non-zero return code”, “rc”: 2, “start”: “2020-06-20 02:51:23.365577”, “stderr”: “/edx/app/analytics_api/analytics_api/analyticsdataserver/settings/production.py:34: YAMLLoadWarning: calling yaml.load() without Loader=… is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n config_from_yaml = yaml.load(f)\nTraceback (most recent call last):\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection\n self.connect()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect\n self.connection = self.get_new_connection(conn_params)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py”, line 227, in get_new_connection\n return Database.connect(**conn_params)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py”, line 84, in Connect\n return Connection(*args, **kwargs)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py”, line 179, in init\n super(Connection, self).init(*args, **kwargs2)\nMySQLdb._exceptions.OperationalError: (1045, “Access denied for user ‘migrate’@‘localhost’ (using password: YES)”)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File “./manage.py”, line 10, in \n execute_from_command_line(sys.argv)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line\n utility.execute()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv\n self.execute(*args, **cmd_options)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py”, line 361, in execute\n self.check()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py”, line 390, in check\n include_deployment_checks=include_deployment_checks,\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/commands/migrate.py”, line 64, in _run_checks\n issues = run_checks(tags=[Tags.database])\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/registry.py”, line 72, in run_checks\n new_errors = check(app_configs=app_configs)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/database.py”, line 10, in check_database_backends\n issues.extend(conn.validation.check(**kwargs))\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py”, line 9, in check\n issues.extend(self._check_sql_mode(**kwargs))\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py”, line 13, in _check_sql_mode\n with self.connection.cursor() as cursor:\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor\n return self._cursor()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in _cursor\n self.ensure_connection()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection\n self.connect()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit\n raise dj_exc_value.with_traceback(traceback) from exc_value\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection\n self.connect()\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect\n self.connection = self.get_new_connection(conn_params)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py”, line 227, in get_new_connection\n return Database.connect(**conn_params)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py”, line 84, in Connect\n return Connection(*args, **kwargs)\n File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py”, line 179, in init\n super(Connection, self).init(*args, **kwargs2)\ndjango.db.utils.OperationalError: (1045, “Access denied for user ‘migrate’@‘localhost’ (using password: YES)”)\nmake: *** [migrate] Error 1”, “stderr_lines”: ["/edx/app/analytics_api/analytics_api/analyticsdataserver/settings/production.py:34: YAMLLoadWarning: calling yaml.load() without Loader=… is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.", " config_from_yaml = yaml.load(f)", “Traceback (most recent call last):”, " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection", " self.connect()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect", " self.connection = self.get_new_connection(conn_params)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py”, line 227, in get_new_connection", " return Database.connect(**conn_params)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py”, line 84, in Connect", " return Connection(*args, **kwargs)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py”, line 179, in init", " super(Connection, self).init(*args, **kwargs2)", “MySQLdb._exceptions.OperationalError: (1045, “Access denied for user ‘migrate’@‘localhost’ (using password: YES)”)”, “”, “The above exception was the direct cause of the following exception:”, “”, “Traceback (most recent call last):”, " File “./manage.py”, line 10, in “, " execute_from_command_line(sys.argv)”, " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line", " utility.execute()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute", " self.fetch_command(subcommand).run_from_argv(self.argv)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv", " self.execute(*args, **cmd_options)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py”, line 361, in execute", " self.check()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py”, line 390, in check", " include_deployment_checks=include_deployment_checks,", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/commands/migrate.py”, line 64, in _run_checks", " issues = run_checks(tags=[Tags.database])", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/registry.py”, line 72, in run_checks", " new_errors = check(app_configs=app_configs)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/database.py”, line 10, in check_database_backends", " issues.extend(conn.validation.check(**kwargs))", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py”, line 9, in check", " issues.extend(self._check_sql_mode(**kwargs))", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py”, line 13, in _check_sql_mode", " with self.connection.cursor() as cursor:", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor", " return self._cursor()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in _cursor", " self.ensure_connection()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection", " self.connect()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit", " raise dj_exc_value.with_traceback(traceback) from exc_value", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection", " self.connect()", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect", " self.connection = self.get_new_connection(conn_params)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py”, line 227, in get_new_connection", " return Database.connect(**conn_params)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py”, line 84, in Connect", " return Connection(*args, **kwargs)", " File “/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py”, line 179, in init", " super(Connection, self).init(*args, **kwargs2)", “django.db.utils.OperationalError: (1045, “Access denied for user ‘migrate’@‘localhost’ (using password: YES)”)”, “make: *** [migrate] Error 1”], “stdout”: “./manage.py migrate --noinput --run-syncdb --database=default\nMakefile:101: recipe for target ‘migrate’ failed”, “stdout_lines”: ["./manage.py migrate --noinput --run-syncdb --database=default", “Makefile:101: recipe for target ‘migrate’ failed”]}

The key seems to be in the sentence: “Access denied for user ‘migrate’@‘localhost’ (using password: YES)” I already created a new user migrate in my insights machine and checked if a user migrate exists in my Open Edx installation machine.

Can anybody please help me?

Thanks a lot :slight_smile:

That’s a pretty old guide, and doesn’t use python3, so you might run into many issues there. I don’t know if there’s an updated guide for production deployments, but you can get some clues from the analytics devstack: Getting Started on Analytics

Alternatively, you can consult OpenCraft’s AWS analytics deployments doc, but we haven’t updated it for Juniper/python3 yet either.

You’re correct, that’s the error that’s blocking that step.

So you’ve created the migrate user on your database… have you granted it permissions?

GRANT ALL PRIVILEGES ON `reports`.* TO 'migrate'@'%';
GRANT ALL PRIVILEGES ON `dashboard`.* TO 'migrate'@'%';
GRANT ALL PRIVILEGES ON `analytics-api`.* TO 'migrate'@'%';
FLUSH PRIVILEGES;

Hi Jill, thank you so much for the answer, unfortunately im getting the same error:

  • Im working on two separates machines, one for my Open Edx instance and one for my Insights instance.
  • I have a user “migrate”, with granted permissions on my Open Edx Machine as you pointed.
  • The command in which the process is stopping is:

ansible - playbook - i localhost, - c local analytics_single.yml - - extra - vars "INSIGHTS_LMS_BASE=$LMS_HOSTNAME INSIGHTS_BASE_URL=$INSIGHTS_HOSTNAME"

Being:

LMS_HOSTNAME=“http://<OPEN_EDX_IP>”

INSIGHTS_HOSTNAME=“http://<INSIGHTS_IP>:18110”

  • I already tested if a remote mysql connection is possible between my two machines, it is.

Why is my user being blocked if “migrate” has the privileges and i can remotely connect from the “insights machine”? Can you please guide me? thanks for your help :slight_smile:

PS: Im installing insights in an Azure enviroment using 2 virtual machines, not using AWS, but fo far i had no problems with AWS_GATHER_FACTS or the EC2 and AWS tasks.

Ok i will answer myself for future users:
I had to create migrate, api001 and reports001 users like this:

user@your-insights-machine-ip

I guess it will work changing the “your-insights-machine-ip” for a “%”.

I had to change ANALYTICS_API_DEFAULT_HOST variable to my Open Edx Machine`s IP.

Now i have an error in “hadoop_common : native lib built”:

“native build only supported on Mac or Unix”

But im working on an ubuntu 18…so i guess there should be no error right?

As i can see it has something to do with maven-enforcer-plugin

Do i need to edit any of the files related to hadoop? Where should i be able to see this installation process`s logs so i get more information about this last one?

UPDATE: i got this screenshot of the failure

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc failure -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc failure

Thanks a lot for your help.

Sorry… there’s a lot of things that will fail to work outside of AWS with the normal setup… the analytics pipeline is not really designed to be run on a single VM in a production environment, and requires a lot of customisations all over the place in order to work. For example, the out-of-the-box production deployments rely on configurations in AWS’s Elastic MapReduce (EMR) service, which you can emulate with your own VM, but you have to learn a lot about hadoop and hive in order to do that. It’s possible, but will require a lot of time.

Hadoop isn’t part of Open edX, so you’d need to read up on it to find out how to install it on your environment.

7 posts were split to a new topic: Issues installing Figures