Bulk emails sending too fast causing gmail to block account. Is there a way to rate limit?

Site Operators Site Operations Help
1

I talked with google customer service the other day and they assured me that it’s possible for an account to send to up to 2000 emails per day. Today I tried to send a bulk email to about 1300 class participants. But it said it successfully sent to ~120 of them, before it errored out for the rest. The logs show an SMTP error 421 “try again later” (see below). I contacted google support and confirmed the sending account itself is not blocked for exceeding the daily limit, and they suggested it was probably sending too fast, and that you’re only allowed to connect once per second. They then showed me how to look up a log and it indeed seemed to be sending at a rate of about 2 per second.

So my question is: is there a way to slow down mail sending so that it doesn’t get blocked?

My secondary question is, when an email sending errors out like this, is there any way to pick it back up again, or do I just have to accept that ~120 people will get the same message again? (if I can find a way to resolve this issue.

[36;1mlms-worker_1     |[0m Traceback (most recent call last):
[36;1mlms-worker_1     |[0m   File "/openedx/venv/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
[36;1mlms-worker_1     |[0m     R = retval = fun(*args, **kwargs)
[36;1mlms-worker_1     |[0m   File "/openedx/venv/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__
[36;1mlms-worker_1     |[0m     return self.run(*args, **kwargs)
[36;1mlms-worker_1     |[0m   File "/openedx/venv/lib/python3.8/site-packages/edx_django_utils/monitoring/internal/code_owner/utils.py", line 179, in new_function
[36;1mlms-worker_1     |[0m     return wrapped_function(*args, **kwargs)
[36;1mlms-worker_1     |[0m   File "/openedx/edx-platform/lms/djangoapps/bulk_email/tasks.py", line 334, in send_course_email
[36;1mlms-worker_1     |[0m     raise send_exception  # pylint: disable=raising-bad-type
[36;1mlms-worker_1     |[0m   File "/openedx/edx-platform/lms/djangoapps/bulk_email/tasks.py", line 581, in _send_course_email
[36;1mlms-worker_1     |[0m     connection.send_messages([email_msg])
[36;1mlms-worker_1     |[0m   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 110, in send_messages
[36;1mlms-worker_1     |[0m     sent = self._send(message)
[36;1mlms-worker_1     |[0m   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 126, in _send
[36;1mlms-worker_1     |[0m     self.connection.sendmail(from_email, recipients, message.as_bytes(linesep='\r\n'))
[36;1mlms-worker_1     |[0m   File "/opt/pyenv/versions/3.8.6/lib/python3.8/smtplib.py", line 871, in sendmail
[36;1mlms-worker_1     |[0m     raise SMTPSenderRefused(code, resp, from_addr)
[36;1mlms-worker_1     |[0m smtplib.SMTPSenderRefused: (421, b'4.7.0 Try again later, closing connection. (MAIL) r186sm3113766qkf.128 - gsmtp',
2

@oedx, the platform already has a setting variable which is BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS 0.2 (second) by default, so I guess just by changing it to 2, it might work as you are expecting.

As for your other question, I am not sure about the exact implementation1, it seems that the function/task is by design tolerant for errors, but not sure, about the exact error above, since the error name is not handled but the error code is! 2.

But anyway, please try to set the setting, and let me know if the problem still occurs.

1 Like
3

This doesn’t seem to have worked…

I’m using tutor so I created a plugin to set it, and then I grepped the config to confirm it was set

grep -r BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS /home/tutor/.local/share/tutor/env/apps/openedx/config/
/home/tutor/.local/share/tutor/env/apps/openedx/config/lms.env.json:  "BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS": "2",

However I see it still sending emails at a rate > 1/s

Was it maybe that it needed to just be
“BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS”: 2
instead of
“BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS”: “2”
? (i.e. no quotes on the 2) (I want to make sure I get it right before I spam a bunch of people again…)

4

Since you are changing a json file, it would depends on the parser. somtimes it doesn’t matter.
However to be exact use it without qoutation, hence this how the deafult value is defined as shown here

Also there is another env variable that might intrest you, which is the number of emails per task, which is by defualt is 500 as defined BULK_EMAIL_EMAILS_PER_TASK here `
If you set for one, then if an email failed to sent, then I would assume you would be able to debug or in case of fauier it would be easier to recover?

5

I did
“BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS”: 2
but it still sent 94 emails in 86 seconds before getting blocked, so I think it’s still not working. (I can see other values have no quotes around numbers, so I do think that’s probably the right syntax).
I continue to see

smtplib.SMTPSenderRefused: (421, b'4.7.0 Try again later, closing connection.

in the logs.

I feel like maybe that BULK_EMAIL_RETRY_DELAY_BETWEEN_SENDS isn’t the right setting because maybe it’s only the RETRY delay, as opposed to just the normal sending delay?

6

@oedx,

Yeah you are right, the 2 second is a delay when an error or exaception is thrown, so the probelm here is that mostly that the type of error/exception is not handled SMTPSenderRefused. i.e. the error is not flagged as retryable though it should.
I have pushed a PR to fix it fix: make bulk_email send email task handles a retryable exception by ghassanmas · Pull Request #29080 · edx/edx-platform · GitHub

7

How could I test your fix on Tutor on my non-production platform? (Upgrading to Maple is viable if that makes things easier.)

8

@oedx
If you are using tutor, you can apply my changes in a simple one file tutor plugin, where we can use a git cherry-pick as a patched and then rebuild the openedx image to apply changes, the steps:

1- Go to tutor plugins directory:
cd $(tutor plugins printroot)
2- Create a plugin as a file:
touch myplugin.yml

3- Write the following in file: e.g. using nano, vim or any editor:

name: myplugin
version: 1.0.0
patches:
  openedx-dockerfile-git-patches-default: |
    RUN git fetch --depth=2 https://github.com/ghassanmas/edx-platform/ 91e7a0eb729f3e66587b154f5e4444dbe2414422 && git cherry-pick 91e7a0eb729f3e66587b154f5e4444dbe2414422

4- Activate the plugin tutor plugins enable myplugin
5- Save config tutor config save
6 Build the openedx image tutor images build openedx

Then you can start as you start normally and to check that my changes are applied, you run a command in the lms e.g. git log you should see this log on top

1 Like
9

I’m ready to try this now, but I also just upgraded my beta site to Maple. So do I still need to try this, or is it already baked into Maple?

10

No it’s not merged into maple, I was hoping t would. Though I have to wait for a review on the PR first against master, and still waiting.

11

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

13

I have tested this now by creating a plugin per the instructions. I tried to send emails in a class with ~130 students enrolled. The same error still occurs and the emails error out after 90+ sends:

lms-worker_1     | 2022-04-23 02:20:56,892 ERROR 107 [edx.celery.task] [user None] [ip None] tasks.py:699 - Task 1af6ac0c-d92e-421b-aab0-4b7893f84c74: email with id 30 caused send_course_email task to fail with u"fatal" exception.  38 emails unsent.
lms-worker_1     | Traceback (most recent call last):
lms-worker_1     |   File "/openedx/edx-platform/lms/djangoapps/bulk_email/tasks.py", line 581, in _send_course_email
lms-worker_1     |     connection.send_messages([email_msg])
lms-worker_1     |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 110, in send_messages
lms-worker_1     |     sent = self._send(message)
lms-worker_1     |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 126, in _send
lms-worker_1     |     self.connection.sendmail(from_email, recipients, message.as_bytes(linesep='\r\n'))
lms-worker_1     |   File "/opt/pyenv/versions/3.8.6/lib/python3.8/smtplib.py", line 871, in sendmail
lms-worker_1     |     raise SMTPSenderRefused(code, resp, from_addr)
lms-worker_1     | smtplib.SMTPSenderRefused: (421, b'4.7.0 Try again later, closing connection. (MAIL) o4-20020ae9f504000000b0069e75e7387fsm1669644qkg.43 - gsmtp', '"Foo Course\n Staff" <email@domain.com>')
lms-worker_1     | 2022-04-23 02:20:56,899 INFO 107 [edx.celery.task] [user None] [ip None] tasks.py:304 - BulkEmail ==> _send_course_email completed in : 79.53051471710205 for task : 1af6ac0c-d92e-421b-aab0-4b7893f84c74 with recipient count: 131
lms-worker_1     | 2022-04-23 02:20:56,899 ERROR 107 [edx.celery.task] [user None] [ip None] tasks.py:332 - Send-email task 1af6ac0c-d92e-421b-aab0-4b7893f84c74 for email 30: failed: (421, b'4.7.0 Try again later, closing connection. (MAIL) o4-20020ae9f504000000b0069e75e7387fsm1669644qkg.43 - gsmtp'

While I appreciate the attempted fix, I continue to assert that the given fix is not a fix for the root cause of there apparently being no way to rate limit the core sending (which is the root design error), as opposed to rate limiting the error handling and retry (which is what the merged change is addressing).

14

I have also created a test to check for the 421 SMTPSenderRefused exception, so theortically it suppose to catch this exception, it has been also merged in master.

Can you check for the following:

  1. Have you built the openedx image? and then after building make sure this new openedx image, is being used by lms and lms_worker… doing tutor local start after building the image should be suffient for that.

  2. Have you checked that, the plugin which cherry pick my commit, is applied?. you could check for that by running git log and check for the hash inside any conainter e.g lms which suppose to run the new built image. And again you are suppose to see that the commit hash on top.

15

I just was rebuilding the image for other reasons, and it turned out it was erroring out, so you’re correct that I hadn’t actually built the image. However, right now I’m getting the following error:

Step 21/85 : RUN git fetch --depth=2 https://github.com/ghassanmas/edx-platform/ 91e7a0eb729f3e66587b154f5e4444dbe2414422 && git cherry-pick 91e7a0eb729f3e66587b154f5e4444dbe2414422
 ---> Running in 790bbd2f72fa
From https://github.com/ghassanmas/edx-platform
 * branch            91e7a0eb729f3e66587b154f5e4444dbe2414422 -> FETCH_HEAD
Auto-merging lms/djangoapps/bulk_email/tasks.py

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'root@790bbd2f72fa.(none)')

I ran the specified git config commands and then built again but it still errored out. Any idea how to get around that?

16

This error is mostly related to tutor, if when you run tutor --version you get a version that is < 12.1 and >= 12.0. then chances are this error is expected hence this[1] commit which fix it (applies from >=12.1.*). One way to resolve pip install tutor==12.2.0[2] be followed by tutor config save then build the image

Other way to fix without upgrading tutor, is just add this line in which default git setting is configured, here is same tutor file shared above where I added a new line to set default email/name of git same as tutor commit. Also you would need to tutor config save after editing the plugin.

name: myplugin
version: 1.0.0
patches:
  openedx-dockerfile-git-patches-default: |
    RUN git config --global user.email "tutor@overhang.io" && git config --global user.name "Tutor" 
    RUN git fetch --depth=2 https://github.com/ghassanmas/edx-platform/ 91e7a0eb729f3e66587b154f5e4444dbe2414422 && git cherry-pick 91e7a0eb729f3e66587b154f5e4444dbe2414422

Lastly Regarding this

Hum I am not sure… How/where you run that git command, I am just confused since you would need to run inside docker build env, and I don’t know a way of doing without changing the Dockerfile, hence each time we tutor config save we are changing/overwriting the Dockerfile of the openedx image.

Hope that would work

  1. tutor commit to add git default ↩︎

  2. Only if you are already on 12.* a.k.a lilac don’t do that if you are 11, 10 …etc, I guess you already know but just in case ↩︎

17

It looks like that extra RUN command to set the git config did the trick for build. And it looks like the patch successfully fixes the error, because I sent to about 300 people for the first time ever through open edx! Thanks a million!!!

1 Like
18

Great!, we can confidently I guess mark this as a bugfix for nutmeg hecnce fix: make bulk_email send email task handles a retryable exception by ghassanmas · Pull Request #29080 · openedx/edx-platform · GitHub