Biblical giants? They were half fallen angel, half human. Not many Christians (and maybe even Jews) believe this, but if you study the context of Genesis 6 and the rest of the Bible anytime it speaks of Noah's flood and judging of angels, they coincide (especially in the New Testament in the book of Jude) then there's the Book of Enoch, which isn't in protestant or Catholic Bibles, but the Ethiopians have it in theirs, and it speaks about how the angels took human wives to themselves. The Liger hybrid species explains scientifically the nephelim (or Giants) hybrid species to some degree, how it can be so massive is explained by the DNA. These are the views of some Christians, but again, assuming Biblical giants, there's a lot of different factors into play.
Man, it would be so amazing if we could somehow attribute ancient stories of dwarfs, trolls, giants, to humans meeting other hominid species that we coexisted with.
From what I've heard, dinosaur bones tended to be taken for the bones of giants, at least in the Middle Ages. Dragons were much older than any awareness of dinosaurs; most medieval myths about dragons trace back to the draco standard carried by late-Roman cavalry. (See https://en.wikipedia.org/wiki/Draco_%28military_standard%29 .)
This, by the way, means that there really were dragons in sub-Roman parts of the Dark Ages world: Wales and Cornwall, for example, where draco-bearing cavalry (including King Arthur's, if there was a King Arthur) were fielded probably until the 600s or 700s.
Dragon mythology as an explanation for dinosaur bones seems pretty straight-forward. I'd even say that "dragon" was just the early word for "dinosaur", and the mythology is based on an earlier extrapolation of characteristics from fossil remains than what we use today. A somewhat less strict extrapolation, but even today a lot of what we 'know' about dinosaurs is pure guesswork.
But dinosaur bones driving myths of biblical humanoid giants? That's a much bigger stretch. Most dinosaur bones are pretty non-humanoid, and people who are generally a lot more familiar with human and animal skeletons than we are today wouldn't mistake one for the other. Also, while dragon mythology is mostly reasonable based on fossil evidence (except for the flying and fire-breathing... and gold hoarding) giant mythology is completely different. The biblical texts and non-biblical books like Enoch give them names and actions that are much more historical sounding than mythological. You don't get details like that from big leg bone fossils.
So, I don't buy dinosaur bones as the source of giant mythology.
I have no idea where dragon myths or giant myths originate; giants are easy to guess, but dragons are a strangely common belief for creatures with so little basis in reality. (Plasma cosmology has a very interesting explanation for dragons, and for a lot of other strange things in human history, but the physics of that theory are pretty bad.) I'm just mentioning what it is that dinosaur bones were identified with in the Middle Ages -- centuries before anyone knew that there had ever been such things as dinosaurs.
(To an extent, dragons fall into the same category as unicorns, griffins, and phoenixes: not mythology so much as really bad zoology. All four of these species were believed to exist in the real world as contemporaries of medieval humanity, sometimes surprisingly nearby -- like griffins in the Caucasus.)
Well height variation still occasionally occurs within all populaces everywhere. "Giants" and "dwarves" as a distinct race was probably a drunken glimpse of different looking people from other villages, in a dark wood or the like.
A related thought is that if there ever was any other intelligent species on Earth, noticeably different than humans, we wiped them out long ago.
But given that in fantasy, all the populations seem to be technically the same species (hence able to breed half-elves, half-dwarves, quarter-veelas, etc), the another likely outcome is that the variation would go away over time
Following this thought experiment, it could be that half-breeds are sterile hybrids and can't reproduce, kind of like mules (horse + donkey). Then you wouldn't get genetic normalization.
If they have a different number of chromosomes it's likely.
I thought that as well, but for instance in harry potter one of the characters is quarter-veela, implying that one of the character's parents is a half-breed, and was able to reproduce. I'm unsure if there are other quarter-breeds in other fantasy universes.