Do prisoners have to pay for mail?

The only state I knew that would let you send stamps in was Alabama but I am sure there are others. Check the relevant doc websites.
Getting out messaging system will let you add money for them to reply to messages if you so choose to do this.
JPAY allows you to transfer stamps for messages or attach a pre-reply stamp to your message for them to respond.
Inmate penpals should have the means to write and send letters etc without looking for hand outs from thier penpals so please bare this in mind.
If you are still in doubt phone the facility. There are some helpful CO's.

Some states do allow sending stamps, but many don't. But you don't need to do it.

Would it be possible to send them prestamped addressed envelopes so they can reply to letters we send them?

They pay for all the supplies and postage.