# Urgent: ECommerce in Lilac: Custom Payment Processors Broken **URL:** \ **Category:** Build-Test-Release\ **Tags:** ecommerce, lilac\ **Created:** [May 29, 2021, 9:23pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055 "2021-05-29T21:23:17Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![sarina](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/sarina/32/2308_2.png) [@sarina](https://discuss.openedx.org/u/sarina)\ **Post date:** [May 29, 2021, 9:23pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/1 "2021-05-29T21:23:17Z")
Hi all, In the course of building out Lilac, we’ve run into a stumbling block around Ecommerce. @sambapete writes in Slack [here](https://openedx.slack.com/archives/C01AGTSB1LL/p1622051473031800) and [in this thread](https://openedx.slack.com/archives/C01AGTSB1LL/p1622057390040400) that changes edX has made to the ecomm service has blocked custom payment processors from being used in the new Payment MFE (that is, only Cybersource works in the Payment MFE). Further complicating matters, the old ecommerce workflow is currently broken, and no longer PCI compliant - which is why we were defaulting to the Payment MFE in Lilac. To quote Pierre: > It may still come as a surprise to those with a custom payment processor when they will start upgrading / deploying Lilac and discover their ecommerce flow doesn’t work anymore. There is no easy way backwards if they are not warned not to upgrade. **BIG QUESTION: Who in the community is using payment processors aside from Cybersource?** Are there any people who are well-versed in ecommerce and able to help out? We see four remediation paths, 2 long term and 2 short term: 1. (Long term) Allow the payment MFE to be configured to use other payment processors, and add those payment processors to the codebase alongside cybersource, paypal, and apple pay. 2. (Long term) Fix the issues in ecommerce that prevent the old payment flow from working. 3. (short term) Figure out what commits in the ecommerce repo can be reverted to make the old ecommerce flow work again. [We’ve identified one](https://github.com/edx/ecommerce/pull/3238), but there may be others. 4. (short term) Run the old ecommerce flow from Koa, but apply library upgrades. This may be dangerous as API contracts could be broken. Who does this affect? Who is willing to jump in here and help? @djoy and I are happy to provide some assistance, but both of us are unfamiliar with the ecommerce service. We can help expedite your work/pull requests however we can. We need community experience and expertise to move this issue forward. Finally, I’d like to acknowledge that communication of the limitations of the new Payment MFE, as well as deprecation of the old ecommerce workflow, was not done well (see [this Discuss thread](https://discuss.openedx.org/t/deprecation-removal-legacy-frontend-implementations-depr-17-depr-42-depr-109/3193)). We needed to be more transparent and louder about breaking changes. To that end, I am running an internal RCA (root cause analysis) in the coming weeks for involved internal teams to capture learnings from this incident and inform us how we can do better moving forward. Best, Sarina ---
**Author:** ![sambapete](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/sambapete/32/4100_2.png) [@sambapete](https://discuss.openedx.org/u/sambapete)\ **Post date:** [May 29, 2021, 9:40pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/2 "2021-05-29T21:40:24Z")
Thanks for putting this out @sarina and @djoy. I will definitely try to help to the best of my knowledge. ---
**Author:** ![gabrieldamours](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/gabrieldamours/32/5569_2.png) [@gabrieldamours](https://discuss.openedx.org/u/gabrieldamours)\ **Post date:** [May 30, 2021, 2:20pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/3 "2021-05-30T14:20:52Z")
@giovannicimolin I think we use other payment processors for some of our clients in the Middle East, no? ---
**Author:** ![Maksim\_Sokolskiy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/maksim_sokolskiy/32/8303_2.png) [@Maksim\_Sokolskiy](https://discuss.openedx.org/u/Maksim_Sokolskiy)\ **Post date:** [May 31, 2021, 8:23am UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/4 "2021-05-31T08:23:19Z")
@sarina We also use non default payment processors and will try to help investigating and fixing the situation. ---
**Author:** ![Felipe](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/felipe/32/60_2.png) [@Felipe](https://discuss.openedx.org/u/Felipe)\ **Post date:** [May 31, 2021, 5:44pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/5 "2021-05-31T17:44:58Z")
We also run custom payment processors at eduNEXT. Some of our team members are in the BTR and are aware and discussing the options there. ---
**Author:** ![giovannicimolin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/giovannicimolin/32/159_2.png) [@giovannicimolin](https://discuss.openedx.org/u/giovannicimolin)\ **Post date:** [June 1, 2021, 1:27pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/6 "2021-06-01T13:27:35Z")
@gabrieldamours Thanks for the ping! > [@sarina](#): > > **BIG QUESTION: Who in the community is using payment processors aside from Cybersource?** Are there any people who are well-versed in ecommerce and able to help out? We have a few clients clients using custom payment processors ([PayTabs](https://github.com/open-craft/ecommerce-paytabs) and HyperPay). > [@sarina](#): > > - (Long term) Allow the payment MFE to be configured to use other payment processors, and add those payment processors to the codebase alongside cybersource, paypal, and apple pay. I think this is the most viable alternative and the one that is aligned with the platform’s long-term goals. > [@sarina](#): > > Who does this affect? Who is willing to jump in here and help? @djoy and I are happy to provide some assistance, but both of us are unfamiliar with the ecommerce service. We can help expedite your work/pull requests however we can. We need community experience and expertise to move this issue forward. We already took the first steps in making e-commerce payment providers pluggable through these PRs: > > > This adds configuration points to utilize installable django applications to pro…vide payment processors within ecommerce. > We also have an \[example pluggable application for PayTabs\](https://github.com/open-craft/ecommerce-paytabs) which utilizes these extension points. > > \*\*JIRA tickets\*\*: Implements BB-3315 > > \*\*Dependencies\*\*: None > > \*\*Screenshots\*\*: > > \*\*Sandbox URL\*\*: TBD - sandbox is being provisioned. > > \*\*Merge deadline\*\*: None > > \*\*Testing instructions\*\*: > > A test account needs to be created on paytabs, and the fields filled in the ecommerce repo in \`ecommerce/settings/devstack.py\` under the \`PAYMENT\_PROCESSOR\_CONFIG\`. Also paytabs needs an externally accessible address to finish the payment. We're using ngrok to expose the devstack ecommerce port for paytabs to callback to. Here is an example config stripped of credentials: > \`\`\` > 'paytabs': { > 'merchant\_email': 'test@example.com', > 'secret\_key': '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*' > 'default\_phone\_number\_country\_code': '00971', > 'return\_base\_url': 'https://f544d7fd8e8d.ngrok.io' > }, > > \`\`\` > > 1. Install the PayTabs E-Commerce pluggable backend in ecommerce virtualenv: \`pip install git+https://github.com/open-craft/ecommerce-paytabs.git\` > 2. Configure ecommerce to use the newly installed payment processor in \`/edx/etc/ecommerce.yml\`: > \`\`\` > ADDL\_PAYMENT\_PROCESSORS: > - paytabs.processors.PayTabs > \`\`\` > 3. Configure ecommerce on devstack to use paytabs as the default payment processor from \[the SiteConfiguration options\](http://localhost:18130/admin/core/siteconfiguration/1/change/) > 4. Add a \[waffle switch in ecommerce\](http://localhost:18130/admin/waffle/switch/) with the name \`payment\_processor\_active\_paytabs\` to enable the processor > 5. Disable, or add and disable, the \[waffle flag\](http://localhost:18130/admin/waffle/flag/) named \`enable\_client\_side\_checkout\` to force checkouts to direct to the PayTabs site > 6. Setup paytabs configuration in \`ecommerce/settings/devstack.py\`. > 7. Update the \[LMS SiteConfiguration\](http://localhost:18000/admin/site\_configuration/siteconfiguration/1/change/) with the necessary extended profile fields: \`"extended\_profile\_fields":\["first\_name","last\_name","mailing\_address","city","state"\]\` > 8. Go to the \[account settings page\](http://localhost:18000/account/settings) to fill in the necessary fields configured above. > 9. Login as a test student user and \[upgrade your demonstration course enrollment to verified\](http://localhost:18130/basket/add/?sku=8CF08E5) using ecommerce. This will add the upgrade to the basket, and prompt to "Checkout with PayTabs". > > \*\*Author notes and concerns\*\*: > > 1. There isn't a simple way to bubble error messages back to learner's browsers from within the basket page. If all fields are not set in their account profile, they get a generic "transaction failed" when PayTabs refuses if they are missing even one required field, which would make it difficult for them to determine on their own what they need to set in the account profile editor. > > \*\*Reviewers\*\* > \- \[\] @lgp171188 > \- \[\] edX reviewer\[s\] TBD > > > Currently, there is no way to specify extra python requirements for any role tha…t depends on \`edx\_django\_service\`, \`ecommerce\` and \`discovery\` in particular. > > This PR adds Ansible task to \`edx\_django\_service\` role, which installs extra requirements specified in \`edx\_django\_service\_extra\_requirements\` variable. > > Also, it modifies \`ecommerce\` and \`discovery\` roles metadata to make use of the new variable. > > \*\*JIRA tickets\*\*: > \- \[OSPR-5773\](https://openedx.atlassian.net/browse/OSPR-5773) > \- \[BB-3669\](https://tasks.opencraft.com/browse/BB-3669) > > \*\*Discussions\*\*: > \- https://github.com/edx/course-discovery/pull/2882#issuecomment-761019948 — initially, we created a pull request that was adding new dependency, but reviewers pointed out that it's better to allow instance operators to specify dependencies via configuration. > \- \[discuss.openedx.org forum post\](https://discuss.openedx.org/t/how-to-install-private-dependencies-while-deploying-idas/4177) > > \*\*Sandbox URL\*\*: > \- https://stage.manage.opencraft.com/instance/7900/ > > \*\*Testing instructions\*\*: > > 1. Connect to sandbox app server: > \`\`\`bash > ssh pomegranited@54.37.150.161 > \`\`\` > 1. Activate ecommerce virtual environment: > \`\`\`bash > source /edx/app/ecommerce/venvs/ecommerce/bin/activate > \`\`\` > 1. Ensure that \`lolcat\` is installed and works: > \`\`\`bash > lolcat /edx/etc/lms.yml > \`\`\` > 1. Deactivate ecommerce virtual environment: > \`\`\`bash > deactivate > \`\`\` > 1. Activate discovery virtual environment: > \`\`\`bash > source /edx/app/discovery/venvs/discovery/bin/activate > \`\`\` > 1. Ensure that \`lolcat\` is installed and works: > \`\`\`bash > lolcat /edx/etc/lms.yml > \`\`\` > > \*\*Reviewers\*\* > \- \[\] @pomegranited > > \*\*Settings\*\* > \`\`\`yaml > ECOMMERCE\_EXTRA\_REQUIREMENTS: > - name: lolcat > version: 1.4 > > DISCOVERY\_EXTRA\_REQUIREMENTS: > - name: lolcat > version: 1.4 > \`\`\` > > > This PR adds a new \`ECOMMERCE\_EXTRA\_CONFIG\_OVERRIDES\` configuration, which allow…s overriding any e-commerce \`\`settings.py\`\` variable. > > With this PR any e-commerce settings can be overriden. For example, if we want to override \`\`LANGUAGES\`\` - > \`\`\`yml > ECOMMERCE\_EXTRA\_CONFIG\_OVERRIDES: > LANGUAGES: > - - en > - English > - - ar > - Arabic > \`\`\` > > \*\*Dependencies\*\*: None > > \*\*Merge deadline\*\*: None > > \*\*Author notes and concerns\*\*: > N/A > > \*\*Reviewers\*\* > \- \[\] @kaizoku > \- \[\] @giovannicimolin > \- \[\] edX reviewer\[s\] TBD These take care of adding pluggability to the old views (the ones that were deprecated). The next step here would be to merge the e-commerce PR and then add support on the MFEs to use this pluggable mechanism (and maybe some extra endpoints in the service to let them retrieve payment processor metadata). We have not started using the MFE’s on our deployments yet, so there’s no work done for that. Does anyone have an idea of the effort to implement this mechanism in the MFEs? (CC @djoy). Maybe we can join efforts in the community and implement this last piece of work together? CC @Felipe @Maksim_Sokolskiy ---
**Author:** ![djoy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/djoy/32/542_2.png) [@djoy](https://discuss.openedx.org/u/djoy)\ **Post date:** [June 1, 2021, 4:25pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/7 "2021-06-01T16:25:47Z")
Thanks for all the detail @giovannicimolin - the MFE has already been structured in a fairly modular way for the three payment processors that exist today (Cybersource, Apple Pay, and Paypal). They’re not fully pluggable, but they have a fairly consistent interface which we can work with. If we want to work on this in the short term, broadly I think we’d err on the side of adding more sibling directories here: [frontend-app-payment/src/payment/payment-methods at master · edx/frontend-app-payment · GitHub](https://github.com/edx/frontend-app-payment/tree/master/src/payment/payment-methods) In the future if we’re able to make this more fully pluggable, we can go back and clean it up and firm up the abstraction layer. One wrinkle may be the way the checkout UI works w/r/t credit card fields. With Cybersource, it uses an iframe to embed code from Cybersource that handles CC processing so that the ecommerce service never even sees user credit card numbers. To keep the ecommerce service and frontend-app-payment from needing to be PCI compliant, we might need to find similar mechanisms for other processors. I’m not 100% up the details there or about what we’d want, but it’s something we’ll have to consider carefully, as some operators may care more about it than others. ---
**Author:** ![arbrandes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/arbrandes/32/158_2.png) [@arbrandes](https://discuss.openedx.org/u/arbrandes)\ **Post date:** [June 1, 2021, 6:02pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/8 "2021-06-01T18:02:25Z")
For the record, during [the contributor’s meeting](https://drive.google.com/file/d/1jLFPCcLSth1EWGsOu30ncXumiozc8s5F/view?usp=sharing) today we decided that at launch Lilac would release as is: with no support for the old ecommerce flow, and no support for other payment processors. Only the MFEs as they stand. If, however, by Lilac.2 there is a tested patch to resurrect old ecommerce, we agreed we’d merge it in. We’ll target support for other payment processors on the ecommerce MFE(s) for Maple. ---
**Author:** ![giovannicimolin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/giovannicimolin/32/159_2.png) [@giovannicimolin](https://discuss.openedx.org/u/giovannicimolin)\ **Post date:** [June 2, 2021, 1:17pm UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/9 "2021-06-02T13:17:26Z")
@djoy > [@djoy](#): > > If we want to work on this in the short term, broadly I think we’d err on the side of adding more sibling directories here: [frontend-app-payment/src/payment/payment-methods at master · edx/frontend-app-payment · GitHub](https://github.com/edx/frontend-app-payment/tree/master/src/payment/payment-methods) > > In the future if we’re able to make this more fully pluggable, we can go back and clean it up and firm up the abstraction layer. I think that we can avoid complex implementations (pluggable frontend modules) and go for generic implementations of payment providers for the long run as well (let’s leave the complexity of the payment provider to be handled by the backend, while the frontend only renders pages using a common interface). We worked with HyperPay and Paytabs so far, and both used different payment flows that don’t require PCI certification. Hyperpay: needs some backend requests to prepare the payment flow, then renders an iframe with the payment implementation (so card data never touches e-commerce). PayTabs works similarly: after a request to set up the payment, it’ll return a URL you can redirect a user to fill in the payment information on its own page. These are two different but very common types of payment flow among payment providers and should cover most use cases. ---
**Author:** ![Abdulmajeed\_Alameer](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/abdulmajeed_alameer/32/2981_2.png) [@Abdulmajeed\_Alameer](https://discuss.openedx.org/u/Abdulmajeed_Alameer)\ **Post date:** [October 26, 2021, 12:34am UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/10 "2021-10-26T00:34:06Z")
Most common payment processors in the Middle East are HyperPay and PayTabs. We are using HyperPay in our implementation. Are we still forced to use the new MFE mechanism? is there a way to use the old ecommerce flow on Lilac? ---
**Author:** ![Yago](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/yago/32/367_2.png) [@Yago](https://discuss.openedx.org/u/Yago)\ **Post date:** [September 25, 2023, 10:58am UTC](https://discuss.openedx.org/t/urgent-ecommerce-in-lilac-custom-payment-processors-broken/5055/11 "2023-09-25T10:58:29Z")
Hello, is there any solution to this issue? We would like to integrate a new custom payment processor but we are not sure on how to do it or even possible, as [tutor-ecommerce’s documentation says its not supported](https://github.com/overhangio/tutor-ecommerce#custom-payment-processors). Thanks in advance Regards