When testing locally using XAMPP, the site is noticeably slower (even in backend) than the live server. I am wondering how this could be. Pages just load faster when working on the live site.

Could it be a browser caching issue? i.e. less caching when working locally

Probably because your computer to work for both client and server. Localhost only for test, it will be different on hosting.