The other answers have covered that most objects scatter light (diffuse reflection) and therefore are visible similarly from any angle. But also consider a perfectly shiny object, with only specular reflection, as you assumed in your original question. It would not be invisible! Consider a small sphere, such as a ball-bearing ball. It looks “shiny”. But if you examine it carefully, you will see that its appearance is a distorted image of the room around it, whose walls are probably diffuse-reflective.
A useful tool for thinking about this is to recognize that it's possible to follow the path of a hypothetical light-ray backwards from your eye through many reflections. If the ray ends at a light source, then you must be seeing light in that direction; if the ray ends at something absorbing then you aren't. In computer graphics, this is known as ray tracing.
Let's eliminate all diffuse reflection. Suppose you (and an ideal lamp) were in a room with perfectly mirrored (i.e. specular) walls and a mirrored sphere. Furthermore, suppose that you absorb all light — you are perfectly black. What will you see?
The space around you will seem to be an infinite number of images of everything in the room (the "house of mirrors" effect). What you see will will be either perfectly white (for backwards rays which reach the lamp) or black (for those which reach your body).
However, you will still be able to recognize the mirrored sphere, because of its characteristic distortion of its reflected image! The edges will be especially visible, because the angle of reflection varies especially rapidly there.
On the other hand, if the walls were a single color and perfectly uniform brightness, you would only see a single spot on the sphere, the reflection of the lamp, and all else would be the same black as the walls. And if it was a box instead of a sphere, you would only see the reflection if you held it at the right angle — the box would be invisible by way of being the same color as the walls it is reflecting.
Here's a ray-traced simulation of the first scene I described. The light is a sphere and "your body" is a box. All reflections in this image have the angle of incidence equal to the angle of reflection.
For the sake of editability, here is the POV-Ray scene file which produced the above image:
#include "colors.inc"
global_settings { max_trace_level 30 }
#declare Specular = finish {
ambient 0
reflection 1.0
diffuse 0
brilliance 0
}
camera {
location <-1, 1, -7.0>
look_at <0, 0, 0>
}
box {
<-1,-4,-1>, <1,1,1>
finish { reflection 0 diffuse 0 brilliance 0 }
translate <-1, 1, 7.0>
}
sphere {
<0,4,0>, 2
pigment { color White }
finish { ambient White }
}
plane { +y, -10 texture {Specular} }
plane { -y, -10 texture {Specular} }
plane { +x, -10 texture {Specular} }
plane { -x, -10 texture {Specular} }
plane { +z, -10 texture {Specular} }
plane { -z, -10 texture {Specular} }
sphere { <2,0,0>,1 texture {Specular} }