Ok, I figured it out
In BuddyPress installation guide, it says I needed to delete "Forums" WordPress page so I did.
Turn out that "Forums" page held on to mywebsite.com/forums permalink although it was already trashed.
So I went into phpMyAdmin and change the slug of the page to forums1 and guid to mywebsite.com/forums1 (I'm not sure if changing guid is necessary) and then created a new page with name "Forums". Now this new page can have permalink mywebsite.com/forums I can insert shortcode
[bbp-forum-index] as Sarah Gooding pointed out here
Alternatively, I believe you can bring the "Forums" page back to life (Hi BuddyPress, why should we delete the "Forums" page in the first place?) and choose default template (just don't wanna mess with BuddyPress templates) and insert the same shortcode.