Q: Why is it so hard to build android for different devices?

I've had this question for quite a long time now and I don't understand the process of porting very well.

Why can't there be a process where I can specify the drivers to be used and build android source code with it to create a ROM?