Edx/var/log/tracking/

Site Operators Site Operations Help
1

Why tracking.log, under /edx/var/log/tracking/ is not updated.

2

Hi @HsuMonKo - as mentioned in a previous thread, if you would like to get help, you need to provide more details. What you were trying to do, and what you tried to debug the issue.

3

Hi @antoviaque, thanks for trying to help me. I’m setting up insights in my openedx site. tracking.log file which is useful in running the tasks to update insights is not updated.
I register new acc at my openedx site, enroll some courses and doing activities yesterday. But when i check the log file today, these activities are not updated in that tracking.log under /edx/var/log/tracking.

4

Hi HsuMonKo,
OpenEdx logging maintained by rsyslog system service. And log file periodically rotated by logrotate cron job. May be something get wrong in rotation script or rsyslog does not works…
Try to issue systemctl status rsyslog command and if there are errors try systemctl start rsyslog or systemctl restart rsyslog.

Also make sure that file /etc/rsyslog.d/99-edx.conf exists with this content - https://github.com/edx/configuration/blob/master/playbooks/roles/common/templates/edx_rsyslog.j2

5

Alright, thanks a lot. I’ll try it.

6

Hello @andrey.kryachko, when i check rsyslog’s status, it’s active. Then /etc/rsyslog.d/99-edx.conf also exists with the content in https://github.com/edx/configuration/blob/master/playbooks/roles/common/templates/edx_rsyslog.j2. But log file still doesn’t update.

7

please try to login to openedx (to make sure that tracking event generated) and then run this command on instance sudo lsof -nn -p $(pidof rsyslogd)

the output must contain all logfiles currently open by rsyslog.

...
rsyslogd 836 syslog    6w   REG 253,1 707296    560 /var/log/syslog
rsyslogd 836 syslog    7w   REG 253,1 1730789  2089 /var/log/auth.log
rsyslogd 836 syslog    8w   REG 253,1 197280    3082260 /edx/var/log/lms/edx.log
rsyslogd 836 syslog    9w   REG 253,1 590317    2332092 /edx/var/log/tracking/tracking.log

if not, try systemctl restart rsyslog. or try to reboot the instance.

8

Really thanks for your help. I’ll try it.

9

Hi @andrey.kryachko, when i run sudo lsof -nn -p $(pidof rsyslogd), the output contains
/var/log/syslog,
/var/log/auth.log and
/edx/var/log/lms.edx.log. But /edx/var/log/tracking/tracking.log is missing. Then i tried systemctl restart rsyslog, but /edx/var/log/tracking/tracking.log is still missing. :cry:

10

Hi HsuMonKo,

you wrote:

/var/log/auth.log and
/edx/var/log/lms.edx.log. But /

/edx/var/log/lms.edx.log - is this really what you see in the output or just a mistyping?
The LMS log file, according to configuration must be stored as /edx/var/log/lms/edx.log. If this is true, then something is wrong in your /etc/rsyslog.d/99-edx.conf configuration, please replace it with original and restart rsyslog service.

If that was mistyping, then try to check the tracking.log file owner and attributes. If they are differs from

$ ls -la /edx/var/log/tracking/tracking.log
-rw-r----- 1 syslog adm 568941 Dec 27 08:22 /edx/var/log/tracking/tracking.log

try to fix it and then restart rsyslog service.

sudo chown syslog:adm /edx/var/log/tracking/tracking.log
sudo chmod 0640 /edx/var/log/tracking/tracking.log
sudo chown root:root /edx/var/log/tracking
sudo chmod 0755 /edx/var/log/tracking
1 Like
11

Sorry @andrey.kryachko.
It’s really is /edx/var/log/lms/edx.log.
/edx/var/log/lms.edx.log was mistyping. Sorry for my mistake. I’ll try what u said. Thanks.

12

Hi @andrey.kryachko, really thanks and srry for my late reply.
tracking.log is finally updated by following your suggestions.
I have another question :smiley:
tracking.log file under hadoop, (hdfs dfs -ls /data) also need to be updated (because i’m trying to get analytics on my openedx.)
But the one under hadoop is not updated.

13

Hi HsuMonKo,

in production environment the /edx/var/log/tracking/tracking.log-* files usually synced from all app instances to S3 bucket by some cron job (as described in this article).

According to step 5 of single-box installation notes tracking.log’s is copied to dedicated “analytics stack instance” where been processed by this cron job.

I see where we start with file permissions issue! You try to install analytics_pipeline on same machine where LMS is running and owner of tracking.log was changed to hadoop user. I can suggest to add hadoop user to adm system group to be able to access tracking.log file - sudo usermod -a -G adm hadoop

14

Thanks @andrey.kryachko. I’ll try it.

15

working fine for me. thanks man.

16

Hi andrey.kryachko,

I am facing this issue, let me describe my context as below, hope you can help me in this case

  • I have a separate LMS instance that tracking.log is located at [root_folder]/data/lms/logs/tracking.log because I used tutor to deploy it.

  • In hadoop machine, I also copied tracking.log from LMS to /edx/var/log/tracking/tracking.log

  • The content of file /etc/rsyslog.d/99-edx.conf looks like attachment.txt (2.6 KB)

    …
    auth,authpriv.* /var/log/auth.log
    .;auth,authpriv.none,local0.none,local1.none -/var/log/syslog
    $template DynaFile,"/edx/var/log/%syslogtag:R,ERE,1,BLANK:[service_variant=([a-zA-Z_-])].–end%/edx.log"

    local0.* -?DynaFile
    local1.* /edx/var/log/tracking/tracking.log;tracking
    …

  • I tried to restart rsyslog and restart Ubuntu as well. However, the output does not have
    this line: rsyslogd 836 syslog 9w REG 253,1 590317 2332092 /edx/var/log/tracking/tracking.log
    Here is the output

    rsyslogd 10201 syslog 0r CHR 1,3 0t0 6 /dev/null
    rsyslogd 10201 syslog 1w CHR 1,3 0t0 6 /dev/null
    rsyslogd 10201 syslog 2w CHR 1,3 0t0 6 /dev/null
    rsyslogd 10201 syslog 3u unix 0xffff91578d2d9c00 0t0 1450 /run/systemd/journal/syslog type=DGRAM
    rsyslogd 10201 syslog 4r CHR 1,9 0t0 11 /dev/urandom
    rsyslogd 10201 syslog 5r REG 0,4 0 4026532048 /proc/kmsg
    rsyslogd 10201 syslog 6u unix 0xffff91568382e400 0t0 20936725 type=DGRAM
    rsyslogd 10201 syslog 7w REG 8,1 354486 6553661 /var/log/syslog
    rsyslogd 10201 syslog 8w REG 8,1 927560 6554443 /var/log/auth.log

  • The tracking.log file contains much more log that looks like (each line is not fully JSON string because of prefix 2021-03-26 09:57:57,353 INFO 796 [tracking] logger.py:50 - )

    2021-03-26 09:57:57,353 INFO 796 [tracking] logger.py:50 - {“username”: “openedx”, “event_type”: “/courses/course-v1:TEST+00088+2021_00088/instructor/api/list_instructor_tasks”, “ip”: “113.161.78.65”, “agent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36”, “host”: “dev.sandbox.co”, “referer”: “https://dev.sandbox.co/courses/course-v1:TEST+00088+2021_00088/instructor”, “accept_language”: “en-US,en;q=0.9,vi-VN;q=0.8,vi;q=0.7,fr-FR;q=0.6,fr;q=0.5”, “event”: “{“POST”: {}, “GET”: {}}”, “event_source”: “server”, “context”: {“course_user_tags”: {}, “user_id”: 5, “org_id”: “TEST”, “course_id”: “course-v1:TEST+00088+2021_00088”, “path”: “/courses/course-v1:TEST+00088+2021_00088/instructor/api/list_instructor_tasks”}, “time”: “2021-03-26T09:57:57.353746+00:00”, “page”: null}

My questions are:

  1. Is the data in tracking.log correct?
  2. I also have /etc/logrotate.d/hourly/tracking.log that will rotate tracking.log to tracking.log-20210331-1617171421.gz, which one will be put in hadoop? Because there was a cronjob that will put tracking.log instead of tracking.log-20210331-1617171421.gz
  3. Does anyway that make tracking.log appears after executes commandline sudo lsof -nn -p $(pidof rsyslogd)

Can you help me in this case?

Thank you