Physiologically they are very similar. Both races have their inferior canines sticking out of their mouths and have pointy ears. Their color of the skin of orcs and high elves is a similar light green. As well, both races have purple eyes.
Also (and this could be completely a coincidence) in Lord of The Rings, orcs are “mutated” elves. Perhaps Matthew wanted to make a reference?