In public places, people have no expectation of privacy. It doesn’t matter whether the device is a dslr or glass.
You don’t have to ask for permission to take someone else’s pictures or videos.
Is it rude/creepy? Possibly. Is it illegal? Hell no. Otherwise the jail would be full with paparazzi.
