XeLaTeX and LuaHBLaTeX are both HarfBuzz-based, so if HarfBuzz handles Khmer properly, then you'll likely be able to get it to work with LaTeX, but it will probably need a few tweaks.