Why do ceramics have a yield strength? From what I've learned so far, I look at yield strength as the beginning of plastic deformation in an object.
If ceramics don't (well usually don't) undergo plastic deformation, how can it be said that ceramics have a higher yield strength then metals?
 A: You can also think of yield strength as the end of the elastic region.  For ceramics this is convenient because they do indeed have an elastic region.  Alternatively, instead of saying they have "no" plastic region, say that ceramics have zero plastic region, which fits with the fact that the breaking point is at the point where yield strength is measured.
And remember that, in practice, all materials are... well... real.  No real material follows the simple 2-part stress-strain curve.  Most follow it pretty darn well, but there's always complications due to the real life nonhomogenaity of materials.  So when there's funny corner cases, that's probably okay.  The reality of physics will round them out!
A: In General
There is a big ol' IN GENERAL appended to everything we say about material classes. There will be overlap, exceptions, and special cases. The stuff here is simply in general: you would not be crazy for expecting this kind of performance.
Why Are Ceramics so ... Not Plastic?
Ceramics generally do not have the ability to transfer dislocations/defects nicely. (Dislocations, for the purposes of this discussion, is an error in the structure, like a missing atom.) Many ceramics are a random jumble of atoms, so moving some atoms here doesn't guarantee they will rebond with atoms over there. There are no hinges or chains to orient, as you would see with polymers. Atoms are simply so well bound and in a non-repeating structure, that moving one bit nessisarily means breaking the bonds. Broken bonds usually means material failure!
On the other hand, metals can happily move any dislocations from one part of the crystal to another. This allows them to be ductile and stretch. The missing atom, or the plane of atoms, can rebond with atoms their new positions. Since you are breaking the bonds of only a few atoms at a time and they can reform in their new position, you get a low energy requirement and thus a lower yield strength.
Polymers, on the anomalous third hand, can be imagined as a net or tangled up headphones. You can stretch these a bit, as the random bits of the polymer chain line up with the force applied. This is relatively easy to do, as no bonds need breaking, so polymers generally have low strengths. (Also, less dense connections between chains makes for a lower ultimate and breaking strength: you have less bonds per volume to break!)
