Replying to myself as a follow up:

Make sure the S3 bucket is set to Static Website Hosting mode, and make sure your Index and Error documents are set appropriately. Then make sure your CloudFront distribution is pointed at the Website's endpoint and NOT the normal S3 endpoint.

That's exactly my configuration and works like a charm.

I initially had issues because I was pointing CloudFront at the normal S3 endpoint. Once that was solved, everything started working smoothly.