What bill pay feature?
When I log in to my account, it gives me an option in the list on the right of the screen to "Pay Bills with this Card". It links to a site that lets you search for eligible payees. I haven't done a thorough search yet to see how many payees are applicable to me, and before I waste my time, I want to know if I earn points for the money spent using this service.
EDIT: Didn't notice the FAQ button on the bill pay site until now. It says "5. Will I earn rewards for bills I pay with my Chase card? Yes, you may as long as you are enrolled in a reward program and are using an eligible Chase card. See your reward program Rules and Regulations for details."
ANOTHER EDIT: Facepalm! I thought this was a service, but apparently it is just a listing of companies. When you click "Get started" all it does is take you to their site to log in and try to pay with your Chase card. Not nearly as helpful as I thought. Pretty useless actually...