Build your own ARM system: Can it be done?

I've been wondering recently what would be involved in building your own ARM-based computer, in a similar fashion to how you build desktops. Is it possible? As in, can you buy a motherboard, SoC, appropriate RAM, etc. From say Qualcomm or a retailer. And if so, how difficult would it be? I've built two normal desktops and done a few minor laptop repairs but nothing beyond that, though I'm willing to learn more skills.

Basically, I just like the idea of making my own little HTPC based on Android or the ARM version of Ubuntu.