At some point I used the mysql8utf8mb4 plugin from here: Upgrading MySQL charset to utf8mb4 - #2 by Abdess in order to get around the Open edX emoji regression that occurred from Maple → Nutmeg. During upgrades I normally disable all my plugins, since 1) I don’t know if they will work with the new version and 2) I’ve had errors in the past due to incompatible plugins.
It turns out that the root cause of these mysql errors was not having this plugin enabled across upgrades! So if anyone ever knows they changed over to the MySQL 4-byte support, you need to make sure the mysql8utf8mb4 plugin is enabled across upgrades!. After doing that, I could upgrade from Nutmeg → Olive → Palm → Quince with no problems!