Dream of a Chromecast-supported in-home network

I recently bought a Chromecast. It's a really cool device. Dead simple, and very effective. But it got me thinking about something I'd like in my future, which would be a lot more complex.

I want to be able to have a Chromecast-like HDMI dongle, but with two functions:

  1. A receiver, much like the Chromecast functions as it is. It plays what you cast to it, or
  2. An emitter, streaming the video from my devices.

So I would have a Receiver plugged into one HDMI port on every TV in my home. And into my DirecTV receiver, Wii U, PS3, and PS4, I plug in an emitter. Using my Nexus, I can tell which emitter to cast to which receiver, or based on which are active, maybe they could operate intelligently but with possibility of manual control. And of course, any properly equipped phone, tablet, or PC could be an emitter OR receiver.

While this would be pointless for a one-room setup, I'd be able to watch DirecTV in any room in my home with one receiver and an RF remote, play music simultaneously to several televisions while I'm hosting a party or working around the house, or switch rooms right in the middle of a level (when the bigger TV gets freed up).

This is certainly a pipe dream, and would weigh really heavily on my wireless network, but I could definitely see this being in the future of the internet connected home.

Feel free to tell me why this idea is or isn't horrible :D