You could look at the referrer for "Vote this up/submit this" URLs. If they didn't come from the story itself or from within ycombinator.com, you can simply ignore them for voting purposes and route the traffic to the comment page or to the story itself.
Let the user actually read the story, then come back to vote on it if they like.
It's probably also worth making a note when a user clicks through to a story so that later on you can verify that he's read it before counting his vote for it. (Naturally, you'd also need to note referrals from the story itself to make this work.)
Jason, just to be clear, all my tweets link to the original post on my blog, so any votes or submits from my twitter followers who clicked on the link it my tweet would come from the story itself. I'm definitely not sending out tweets with links to the HN story or to the HN submit link.
Let the user actually read the story, then come back to vote on it if they like.
It's probably also worth making a note when a user clicks through to a story so that later on you can verify that he's read it before counting his vote for it. (Naturally, you'd also need to note referrals from the story itself to make this work.)