@victorleungtw, the install_prereqs sub-command is built into paver and should run without any issues. When following the steps in the edx/devstack repository, did you notice any errors in any of the steps before? Also, did you follow all the steps specific to macOS documented in the README file of this repository?
To investigate your issue further, it will be helpful if you can provide more details about the Open edX release that you are trying to run in the devstack - is it the master branch or is it a named release like Juniper, Ironwood etc. and if there are any quirks in your setup?