Is it really that simple?
No.
Also can a small forum run on my own PC?
Technically speaking, yes.
(As long as your ISP isn't blocking port 80 or it's running on a local network.)
To do that you'll need to install a web server (software) on your PC.
Unless it's only running on a local network you probably don't want to do this. Buy hosting.
My understanding is that I just need to download the free forum software onto my computer and then set it up.
No, you download it to your PC then upload it to your server & run the installation.
Some hosting providers might have an option of installing it directly from the Control Panel.