Hello @Yagnesh!
As far as I know, there is no REST API implemented for content management in Open edX yet, you can find the proposal for MVP in this thread, but it will take much time until it’s implemented and released in a named Open edX release.
However, I think there’s a way to automize process of importing 4000+ courses into Open edX, but it still pretty complex.
In Open edX courses can be built programmatically with open learning XML OLX.
To solve your problem of converting text data from Excel to OLX you can use this script open-sourced by @ndespujol, and presented in the thread: Script to create complete Open edX courses from a template in an Excel file.
To use this script, you would need to transform data for every course into the special XNF template.
Probably it would be reasonable to create a script to transform your Excel data to follow XNF template, so you won’t need to fill in all the data manually.
Using this script, you will be able to generate OLX files for your courses, which then can be uploaded into Open edX using ./manage.py cms import command.
I hope this helps.