Installing INSIGHTS: Database Migrate Error

Site Operators Site Operations Help
1

I´m getting this error while following the official guide for the OpenEdx Insights Installation

The process stops at the TASK [edx_django_service: migrate database]

fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: [“make”, “migrate”], “delta”: “0:00:00.822474”, “end”: “2020-06-20 02:51:24.188051”, “msg”: “non-zero return code”, “rc”: 2, “start”: “2020-06-20 02:51:23.365577”, “stderr”: “/edx/app/analytics_api/analytics_api/analyticsdataserver/settings/production.py:34: YAMLLoadWarning: calling yaml.load() without Loader=… is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n config_from_yaml = yaml.load(f)\nTraceback (most recent call last):\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection\n self.connect()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect\n self.connection = self.get_new_connection(conn_params)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection\n return Database.connect(**conn_params)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py", line 84, in Connect\n return Connection(*args, **kwargs)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in init\n super(Connection, self).init(*args, **kwargs2)\nMySQLdb._exceptions.OperationalError: (1045, "Access denied for user ‘migrate’@‘localhost’ (using password: YES)")\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "./manage.py", line 10, in \n execute_from_command_line(sys.argv)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line\n utility.execute()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv\n self.execute(*args, **cmd_options)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute\n self.check()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check\n include_deployment_checks=include_deployment_checks,\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 64, in _run_checks\n issues = run_checks(tags=[Tags.database])\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks\n new_errors = check(app_configs=app_configs)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/database.py", line 10, in check_database_backends\n issues.extend(conn.validation.check(**kwargs))\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 9, in check\n issues.extend(self._check_sql_mode(**kwargs))\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 13, in _check_sql_mode\n with self.connection.cursor() as cursor:\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor\n return self._cursor()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor\n self.ensure_connection()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection\n self.connect()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit\n raise dj_exc_value.with_traceback(traceback) from exc_value\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection\n self.connect()\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect\n self.connection = self.get_new_connection(conn_params)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection\n return Database.connect(**conn_params)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py", line 84, in Connect\n return Connection(*args, **kwargs)\n File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in init\n super(Connection, self).init(*args, **kwargs2)\ndjango.db.utils.OperationalError: (1045, "Access denied for user ‘migrate’@‘localhost’ (using password: YES)")\nmake: *** [migrate] Error 1”, “stderr_lines”: [“/edx/app/analytics_api/analytics_api/analyticsdataserver/settings/production.py:34: YAMLLoadWarning: calling yaml.load() without Loader=… is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.”, " config_from_yaml = yaml.load(f)“, “Traceback (most recent call last):”, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection”, " self.connect()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect”, " self.connection = self.get_new_connection(conn_params)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection”, " return Database.connect(**conn_params)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py", line 84, in Connect”, " return Connection(*args, **kwargs)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in init”, " super(Connection, self).init(*args, **kwargs2)“, “MySQLdb._exceptions.OperationalError: (1045, "Access denied for user ‘migrate’@‘localhost’ (using password: YES)")”, “”, “The above exception was the direct cause of the following exception:”, “”, “Traceback (most recent call last):”, " File "./manage.py", line 10, in ”, " execute_from_command_line(sys.argv)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line”, " utility.execute()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute”, " self.fetch_command(subcommand).run_from_argv(self.argv)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv”, " self.execute(*args, **cmd_options)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute”, " self.check()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check”, " include_deployment_checks=include_deployment_checks,“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 64, in _run_checks”, " issues = run_checks(tags=[Tags.database])“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks”, " new_errors = check(app_configs=app_configs)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/core/checks/database.py", line 10, in check_database_backends”, " issues.extend(conn.validation.check(**kwargs))“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 9, in check”, " issues.extend(self._check_sql_mode(**kwargs))“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/validation.py", line 13, in _check_sql_mode”, " with self.connection.cursor() as cursor:“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor”, " return self._cursor()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor”, " self.ensure_connection()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection”, " self.connect()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit”, " raise dj_exc_value.with_traceback(traceback) from exc_value", " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection", " self.connect()“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect”, " self.connection = self.get_new_connection(conn_params)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection”, " return Database.connect(**conn_params)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/init.py", line 84, in Connect”, " return Connection(*args, **kwargs)“, " File "/edx/app/analytics_api/venvs/analytics_api/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in init”, " super(Connection, self).init(*args, **kwargs2)“, “django.db.utils.OperationalError: (1045, "Access denied for user ‘migrate’@‘localhost’ (using password: YES)")”, “make: *** [migrate] Error 1”], “stdout”: “./manage.py migrate --noinput --run-syncdb --database=default\nMakefile:101: recipe for target ‘migrate’ failed”, “stdout_lines”: [”./manage.py migrate --noinput --run-syncdb --database=default", “Makefile:101: recipe for target ‘migrate’ failed”]}

The key seems to be in the sentence: “Access denied for user ‘migrate’@‘localhost’ (using password: YES)” I already created a new user migrate in my insights machine and checked if a user migrate exists in my Open Edx installation machine.

Can anybody please help me?

Thanks a lot :slight_smile:

Sin título
Sin título1304×869 713 KB

1 Like
2

That’s a pretty old guide, and doesn’t use python3, so you might run into many issues there. I don’t know if there’s an updated guide for production deployments, but you can get some clues from the analytics devstack: Getting Started on Analytics

Alternatively, you can consult OpenCraft’s AWS analytics deployments doc, but we haven’t updated it for Juniper/python3 yet either.

You’re correct, that’s the error that’s blocking that step.

So you’ve created the migrate user on your database… have you granted it permissions?

GRANT ALL PRIVILEGES ON `reports`.* TO 'migrate'@'%';
GRANT ALL PRIVILEGES ON `dashboard`.* TO 'migrate'@'%';
GRANT ALL PRIVILEGES ON `analytics-api`.* TO 'migrate'@'%';
FLUSH PRIVILEGES;
3

Hi Jill, thank you so much for the answer, unfortunately im getting the same error:

  • Im working on two separates machines, one for my Open Edx instance and one for my Insights instance.
  • I have a user “migrate”, with granted permissions on my Open Edx Machine as you pointed.
  • The command in which the process is stopping is:

ansible - playbook - i localhost, - c local analytics_single.yml - - extra - vars "INSIGHTS_LMS_BASE=$LMS_HOSTNAME INSIGHTS_BASE_URL=$INSIGHTS_HOSTNAME"

Being:

LMS_HOSTNAME=“http://<OPEN_EDX_IP>”

INSIGHTS_HOSTNAME=“http://<INSIGHTS_IP>:18110”

  • I already tested if a remote mysql connection is possible between my two machines, it is.

Why is my user being blocked if “migrate” has the privileges and i can remotely connect from the “insights machine”? Can you please guide me? thanks for your help :slight_smile:

PS: Im installing insights in an Azure enviroment using 2 virtual machines, not using AWS, but fo far i had no problems with AWS_GATHER_FACTS or the EC2 and AWS tasks.

4

Ok i will answer myself for future users:
I had to create migrate, api001 and reports001 users like this:

user@your-insights-machine-ip

I guess it will work changing the “your-insights-machine-ip” for a “%”.

I had to change ANALYTICS_API_DEFAULT_HOST variable to my Open Edx Machine`s IP.

5

Now i have an error in “hadoop_common : native lib built”:

“native build only supported on Mac or Unix”

But im working on an ubuntu 18…so i guess there should be no error right?

As i can see it has something to do with maven-enforcer-plugin

Do i need to edit any of the files related to hadoop? Where should i be able to see this installation process`s logs so i get more information about this last one?

UPDATE: i got this screenshot of the failure

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc failure → [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc failure

Captura de Pantalla 2020-06-24 a la(s) 20.59.12
Captura de Pantalla 2020-06-24 a la(s) 20.59.12886×381 55.7 KB

Thanks a lot for your help.

6

Sorry… there’s a lot of things that will fail to work outside of AWS with the normal setup… the analytics pipeline is not really designed to be run on a single VM in a production environment, and requires a lot of customisations all over the place in order to work. For example, the out-of-the-box production deployments rely on configurations in AWS’s Elastic MapReduce (EMR) service, which you can emulate with your own VM, but you have to learn a lot about hadoop and hive in order to do that. It’s possible, but will require a lot of time.

Hadoop isn’t part of Open edX, so you’d need to read up on it to find out how to install it on your environment.

8

7 posts were split to a new topic: Issues installing Figures

9

Where do you do this?

10

You should try installing Figures instead if you want Insights outside AWS: Check this post

And this post

I had no luck installing insights…i had better luck with Figures, hope this helps.

1 Like
11

Yes, I’ve been stuck with this for awhile. Thank you!

12

I completely understand you :slight_smile:

Try installing figures, its much easier, if you have any doubt please tell me.

13

Hi, we decided to use Figures instead of Insights. I followed the exact instructions given by taufanbudiman, including having the right version of yarn etc. Everything went smooth until the last step when migrating, and the same problem that had originally showed up appeared again with: Importerror: No module named safe_lxml.
Do you know anything about this problem?

14

To any brave soul who comes here wandering.

If you find this error, I suggest skip doing the installation. And use the docker containers available already and change then to production like.

This error is because of various issues.

  1. Wrong javah path in /var/tmp/hadoop-VERSION-src/hadoop-common-project/hadoop-common/pom.xml

After that is fixed

  1. Missing variables in cmake command in /var/tmp/hadoop-2.7.2-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
    (I found this website but it’s in chinese, google translate may help Hadoop 2.7.2编译)
    After that is fixed

  2. Openssl version, I worked with 1.0.2 and 1.1.0 and I am not sure which worked exactly.

After that is fixed

  1. Installation will fail few more times in some other tasks but they are mostly solvable.

Seriously, don’t do this :expressionless: