Juniper Alpha and MongoDB issue

Site Operators
1

I tried migrating a copy of our test environment from ironwood.2 to juniper.alpha1.

Since I can create a course in Studio and access course contents in the LMS, I am assuming that the migration went well for MongoDB. That does not seem the case for the forums.

It looked like a typical 500 error in /edx/var/log/lms/edx.log but when looking deeper in /edx/var/log/supervisor/forum-stderr.log I found the following error (see below).

I do not run a replica set. The mongo configuration that was in /edx/app/edxapp/lms.auth.json under ironwood has been replicated in /edx/etc/lms.yml under juniper but that doesnā€™t seem to work for the forumsā€¦ Any ideas how I can solve that issue? Where do I need to define the select_server method?

2020-01-01 05:12:46 - NoMethodError - undefined method select_server' for {"mode"=>:primary}:BSON::Document: /edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:212:in eachā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/contextual/mongo.rb:252:in first' /edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/contextual/mongo.rb:252:in find_firstā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/contextual.rb:20:in find_first' /edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/findable.rb:113:in find_byā€™
/edx/app/forum/cs_comments_service/lib/helpers.rb:13:in user' /edx/app/forum/cs_comments_service/api/users.rb:16:in block in <top (required)>ā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1610:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1610:in block in compile!ā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in block (3 levels) in route!' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in route_evalā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in block (2 levels) in route!' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in block in process_routeā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in catch' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in process_routeā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in block in route!' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in eachā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in route!' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in block in dispatch!ā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in block in invoke' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in catchā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in invoke' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in dispatch!ā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in block in call!' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in block in invokeā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in catch' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in invokeā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in call!' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/bundler/gems/rack-contrib-6ff3ca2b2d98/lib/rack/contrib/locale.rb:15:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-1.6.8/lib/rack/logger.rb:15:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/rack-1.6.8/lib/rack/commonlogger.rb:33:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:219:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:212:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-1.6.8/lib/rack/head.rb:13:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/rack-1.6.8/lib/rack/methodoverride.rb:22:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in block in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in synchronize' /edx/app/forum/.gem/ruby/2.4.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:104:in block in call' /edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:69:in cacheā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:104:in call' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-timeout-0.4.2/lib/rack/timeout/core.rb:122:in block in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/rack-timeout-0.4.2/lib/rack/timeout/support/timeout.rb:19:in timeout' /edx/app/forum/.gem/ruby/2.4.0/gems/rack-timeout-0.4.2/lib/rack/timeout/core.rb:121:in callā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:606:in process_client' /edx/app/forum/.gem/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:702:in worker_loopā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:549:in spawn_missing_workers' /edx/app/forum/.gem/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:142:in startā€™
/edx/app/forum/.gem/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in <top (required)>' /edx/app/forum/cs_comments_service/bin/unicorn:16:in loadā€™
/edx/app/forum/cs_comments_service/bin/unicorn:16:in `ā€™
I, [2020-01-01T05:12:46.603279 #4548] INFO ā€“ : source=rack-timeout id=30ec3372b9301f46ff1ec9eb0c3a50bf timeout=20000ms service=2ms state=completed
127.0.0.1 - - [01/Jan/2020:05:12:46 -0500] ā€œGET /api/v1/users/30?complete=True&request_id=2460e9f6-7a09-48df-838c-456b5cc31893 HTTP/1.0ā€ 500 30 0.0016

2

@nedbat I had a hunch.

I reinstalled a fresh copy of juniper.alpha1 straight from the edX repos, no modifications, nothing, following instructions from https://openedx.atlassian.net/wiki/spaces/OpenOPS/pages/146440579/Native+Open+edX+platform+Ubuntu+16.04+64+bit+Installation on a single server.

That way I am sure it will not be a problem with my fork or with migration from ironwood.

Guess what? Same errorā€¦

There might be an issue with the forums and MongoDB in juniper.alpha1

3

There is a slight difference between ironwood.master and juniper.alpha1 in cs_comments_service.

ironwood.master uses the ruby gem bson-3.2.6.
juniper.alpha1 uses the ruby gem bson-4.7.0.

I tried replacing the gem in juniper.

The forum would not even start after the change.

I then reinstalled the previously installed gem. The forum would now start but I still have the ā€˜select_serverā€™ error mentioned in my previous post.

I still canā€™t access the forum in the Demo course.

Has anyone tried installing juniper.alpha1? Are the forums working for you?

Thanks for any information.

6

I dug a little bit deeper last evening.

I discovered that under /edx/app/forum/.gem/ruby/2.4.0/gems we had mongo-2.11.2 for Juniper and mongo-2.1.2 on Ironwood.

I looked at the https://github.com/edx/cs_comments_service repository just to look when and if there has been a change.

There was a commit made for ā€œUpgraded mongo ruby driverā€ in https://github.com/edx/cs_comments_service/commit/9256710c8fe1f4e544a29af4d741d15a60d0c6f3 just as open-release/juniper.alpha1 was cut, but it has since been reverted by https://github.com/edx/cs_comments_service/commit/ce5ba1e90eb61e33b8dc651ed28cfaf03ac857b4 after open-release/juniper.alpha1 was cut.

Interesting.

Since I cannot update open-release/juniper.alpha1 at edX and that we do have our own fork at EDUlib, I decided to revert the change in our edulib-juniper.alpha1 branch in order to revert the commit.

I then proceeded to reinstall a new juniper.alpha1 server with our fork.

As expected, the forums were now working again :grin:

I am just a sysadmin and I know development is hard. Unfortunately, it happens from time to time that commits may not have been properly tested before being introduced. The revert came on December 31st or January 1st (8 days ago as of January 8th), almost a week after open-release/juniper.alpha1 was cut. Therefore, if you were an early adopter / tester of open-release/juniper.alpha1, your forums may not be working correctly. Or maybe I was the only one testing it when @nedbat made it availableā€¦ Who knowsā€¦

@nedbat I guess we should now expect a juniper.alpha2 release soon? Or does the ā€œcurrentā€ juniper.alpha1 does include the revert?

7

@sambapete Thanks so much for the digging! Iā€™ve cherry-picked the revert commit onto open-release/juniper.alpha1. Iā€™m not planning to make another branch soon, but we will see.

1 Like
8

@sambapete Did the revert above resolve the issue you noted here?

9

@jill Yes it did. And as @nedbat noted, it was then cherry-picked onto open-release/juniper.alpha1 in order to fix the issue. If you installed recently, your forums should work out of the box.

1 Like
10

Excellent, thank you!