Assuming we end-up with reasonably efficient fusion power and an even mildly space based economy we are eventually going to start mining the ort cloud. Which set's up a natural path to vary slowly expand into the stars.

Basically, if it's profitable you get exponential growth till it's not and even slow exponential growth ends up with ridiculous growth numbers fairly quickly on a geological timescale let alone a galactic one.