Shadow
which arrow casts the shadow? it doesn't matter.
drag two arrows and watch each cast its shadow along the other -- the two products never differ
The dot product of two arrows is one arrow's shadow along the other, times the length it fell on. It doesn't matter which one casts the shadow: the number is the same either way, zero when square, negative past that.
Concepts: linear-algebra, geometry, projection.
An interactive you play with one thumb, on Wundr.