Using S3 or other Persistent Storage Container Storage Interface (CSI) to store LMS Tracking Logs – K8S Install – Centralized Logs

Site Operators
1

At the moment we’re working with a Tutor local install in production but would like to move to a K8S setup.

We’re looking at moving the LMS tracking logs to a centralized location (e.g. S3). This will help with future upgrades but has a side effect when we use analytic systems like Cairn to process the logs. Do you know of any articles to push tracking logs to S3 and would you recommend it?

I wasn’t sure how K8S handle multiple LMS pods with the tracking logs file so that’s why I stood up K8S locally to see how this works and if I could write to S3 directly from the LMS pods.

2

Hi @Zachary_Trabookis

I am not sure about directly writing to s3, but I had to do something similar, where I used to write in a specific volume, and move them to s3 with cron jobs.

Mostly did it for insights.

3

@chintan Thanks for the response. We’ll were looking to see how this would work with Cairn analytics because it ingests the logs more in real-time approach.

4

@regis I just noticed that you mentioned that when going to k8s that Cairn reads from the Docker daemon stdout rather than a ReadWriteMany volume, because not all k8s providers have this type of volume available.

How does Vector listen to these Docker daemon logs on k8s? Is Vector setup in the same pod or node as the LMS or does it do something else for k8s configuration (sidecar) to listen in to Daemon stdout if the LMS pods? I’m trying to understand how Vector is configured to listen for LMS logs on both k8s and local tutor environments.

This article Logging Architecture | Kubernetes take about using a sidecar with logging agent to send logs to a logging backend which I assume at this point to be something like persistent storage or S3 per AWS. If you’ve done something like this let me know if this would be recommended or not.

We’re most likely going to use AWS to setup k8s in the future and we’re currently on a local environment of tutor in production. I titled this article to look into using S3 storage for persistent storage of the tracking log files but thinking that’s most likely not ideal for k8s, and thus I would need to look into a Container Storage Interface (CSI) like Amazon EBS, Amazon EFS , and Amazon FSx for Lustre most likely. Reading over this article to understand what AWS offers for persistent storage for k8s Persistent storage for Kubernetes  | AWS Storage Blog.

We’d like to keep the tracking/all logs in some sort of persistent storage on AWS when in a k8s or local tutor environment that will allow us to not have to worry about migrating tracking/all logs when upgrading between Tutor releases. We wouldn’t have to worry about moving these logs to another tutor environment during tutor upgrade releases.

I’m not so worried about Cairn using this persistent storage to ingest data into Clickhouse datalake, because it sounds like Vector reads from daemon stdout of the LMS and I would prefer to keep it configured that way moving forward for real-time performance and unnecessary customization on our end.

Really just trying to understand how to centralize the tracking/all logs for tutor using persistent storage and understand how Vector is configured to ingest these logs.

5

@lpm0073
Have you run into a situation where you’ve had to logrotate the tutor cms/lms all.log and tracking.log files? We’re looking to use logrotate plus s3cmd to rotate logs and send them to S3 object storage.

I’m looking into doing something like this.

Considering using this tutor plugin and extending it for this log rotation and backup to S3. Would it make sense to do that here or some other plugin?

We’re currently not on Kubernetes yet but I’m also curious how you would handle that in this environment.

6

hi @Zachary_Trabookis and thanks for adding me to this thread. i have not run into that particular situation. HOWEVER, logging is an ongoing subject of inquiry for me. Presently, the log data is ephemeral in all of the platforms i that i currently monitor. i’ve researched a couple of options for unifying and persisting log data, but i’ve yet to find anything that i like, so far. cost has been the main detractor to anything like say, CloudWatch, that otherwise seems to work really well.

it’s not obvious to me right this second how, if left to my own devices, i’d approach piping log data from a k8s pod to an S3 bucket. At a glance this seems like it might be a hacky solution, even if you get it to work.

Incidentally, i DO use the hastexo S3 tutor plugin on all of my platforms, and generally speaking it works great for it’s intended use case. it’s been out-of-sight / out-of-mind for me for the last 12 months or so, which is good news i suppose :sweat_smile:

7

@lpm0073
I’m going to reach out to wg-data group to see how he would handle this logrotate and store on S3 storage for the all.log and tracking.log files.

Local
I think for now I’ll create a separate tutor-contrib-utils that will do this logrotate and s3cmd accordingly.

Using these directions as a guide to build the parts of the plugin.
How To Use Logrotate and S3cmd to Archive Logs to Object Storage on Ubuntu 16.04 | DigitalOcean

Kubernetes
I get that Kubernetes pods for the applications servers are ephemeral but don’t they store the tracking.log somewhere or is it only written to Docker stdout, stderr?

It looks like Cairn and Aspects both use Vector to pull Docker container log output with Docker logs | Vector documentation and ignore the platform defined configuration files at tutor/tutor/templates/apps/openedx/settings/partials/common_all.py at master · overhangio/tutor (github.com) for processing.

Here is where that’s defined in Aspects tutor-contrib-aspects/tutoraspects/templates/aspects/apps/vector/local.toml at main · openedx/tutor-contrib-aspects (github.com)

I noticed that there is a AWS S3 sink to store these events AWS S3 | Vector documentation but I’m not sure what will get stored on S3. Looks like you can set max_bytes or max_events for a batch before it’s sent to S3 and it looks like you can specify compression gzip.

8

Vector read from pods stdout. A Vector configuration that will allow you to store the tracking logs to s3 without too much headache would be the following:

sinks:
  s3_tracking_logs:
    type: aws_s3
    inputs:
      - docker_logs
    bucket: [bucket_logs]
    filename_append_uuid: true
    filename_time_format: "tracking.log-%Y%m%d-%s"
    key_prefix: "my-folder/date=%F/"
    compression: gzip
    encoding:
      codec: text
    region: [aws_bucket]
    auth:
      access_key_id: "xxx"
      secret_access_key: "xxx"

Keep in mind that this is a Vector yaml config and Aspects is using.toml files. Also, there is no tutor patch or extension mechanism to apply this to tutor-contrib-aspects, but that will take just some minutes:

Simply add {{ patch('vector-local-config') }} here https://github.com/openedx/tutor-contrib-aspects/blob/main/tutoraspects/templates/aspects/apps/vector/local.toml#L12 and use it from a plugin.

You can still do the logrorate to the tracking.log file in your plugin to a different bucket (or folder)

1 Like
9

@Ian2012
I guess that it wouldn’t hurt to do the logrotate/s3cmd operations too should Vector fail on some logs. Yeah, I was thinking that this may be a good back solution but it would put double logs on S3.

Do you have anything else to say about logrotate/s3cmd? I guess that it would still be good to logrotate to keep the logs from growing too large and keep the app servers small in disk space usage.

Not sure how you would setup logrotate/s3cmd to work in a Kubernetes environment though since the CMS/LMS logs are written to stdout/stderr. So it makes sense in this environment to use the Vector Amazon S3 sink instead. Am I wrong about this?

Thanks for providing an example for the Vector Amazon S3 sink. I was considering using Vector for both local and Kubernetes installs. Can this Vector sink rotate logs too?

10

The current configuration will just store the logs in s3 but will not rotate them.

You are right about using the Vector sink for k8s and local, it’s the better approach.

1 Like
11

@Ian2012 For logrotate, would you setup this in the CMS/LMS and worker Docker containers because there is no host machine for k8s install? I was considering doing this logrotate on the host machine for local install rather than within each container cms, cms-worker, lms, lms-worker.

Or would you do this https://vector.dev/docs/reference/configuration/sources/file/#file-rotation with Vector? I’m not even sure how to do this yet.

I am currently considering putting logrotate within the LMS container for example. I don’t believe that we can logrotate without the current LMS container user app being a non-root sudo-enabled user. The sudo command is not found on the container. Maybe we’ll have to create a separate user for this logrotate and leave this app user permissions set as is.

I think that this will work as long as I create a user with non-root sudo-enabled user privileges before this gets called in the Dockerfile.
tutor/tutor/templates/build/openedx/Dockerfile at master · overhangio/tutor (github.com)

12

I would probably do it in the host machine, but not sure how to

13

@Ian2012
I was staring to build out a Dockerfile for performing logrotate but I came across this repo which creates a sidecar container and looks at all or a specified list of containers and directories to perform logrotate on. Let me know what you think about this sidecar.

blacklabelops/logrotate: Dockerized Logrotate. Attach, Compress, Rotate! (github.com)

This also works in k8s they claim blacklabelops/logrotate: Dockerized Logrotate. Attach, Compress, Rotate! (github.com)

14

@Zachary_Trabookis worth the try. There is not much I can contribute here, cause I’ve never configured logrotate for k8s or docker.

Regarding Aspects, you can completely disable Vector as long as you have a backup of your tracking logs and Ralph is sending the events to Clickhouse. But as you don’t have backups in S3, the best for you would be to enable Vector with a custom configuration to send rotated tracking logs to S3

1 Like
15

@Ian2012
In your configuration above with Vector you didn’t specify a buffer.max_size. How will Vector know when to make a new buffer of log information?

I don’t mind using Vector to perform sending events to an S3 sink, however, I thought that you mentioned above that the log wouldn’t be rotated. Does Vector do that logrotate or does it only send the event from stdout and stderr and gzip compress it? I think you mentioned above that Vector doesn’t do logrotate.

What will the Vector sink AWS S3 configuration above send to S3? A gzip with what in it? The output of the [sources.docker_logs] Docker logs | Vector documentation which is the stdout, stderr from all the running containers.

It looks here tutor-contrib-aspects/tutoraspects/templates/aspects/apps/vector/local.toml at main · openedx/tutor-contrib-aspects (github.com) that Vector in Aspects filters by a couple specific containers to get log out from. Just trying to make sense of the Vector configuration above.

16

it will use the information from the sources you configure for it. In this case, the docker logs which are not stored by Vector anywhere except I think in memory and will be sent based on the format defined above.

Vector will not do logrotate over your LMS logs as those are not used by Vector in any way.

Vector will send a gzip with a json file in it containing every tracking log.

1 Like
17

@Ian2012 @lpm0073
I setup tutor k8s on my local machine to see how the /openedx/data/logs get generated between two lms services. I noticed that loading a video page and playing a video that the tracking events went back and forth between the two LMS pods.

Image shows two LMS pods on the left and on the right loaded video component along with the lower right, where I issued a tutor k8s logs --follow --tail 20 lms command to see if the logs print out.

I noticed the following:

  • Events loaded in either LMS pod. The request came in to either load video, play video, or pause video and either pod handled it.
  • The /openedx/data/logs/{all|tracking}.log files output their own pods request handled event information and pods events look different from each other.
  • The lower right using the tutor log command I noticed the events came in as if you had just one LMS pod running.
  • I noticed there was no volume mapping like we have with local install back to tutor-env/data/{cms|lms}/logs directory.
    image
    image1920×784 255 KB

I get that the LMS pods should be ephemeral and this log data sends out through the Docker containers through stdout or stderr and that makes sense but if you didn’t have something like Vector listening on the other end and these pod went down then those logs in /openedx/data/logs might get lost.

Having said all the above, how does one backup the /openedx/data/logs{all|tracking}.log in a k8s environment without using something like Vector AWS S3 sink?

cc @sambapete

18

@regis provided a solution here for listening on a host machine or separate container for a signal termination of the LMS Docker service. I will try this solution out to see if I can perform the logrotate/s3cmd accordingly.