Mendis isn't accurate enough, doesn't plan well enough, and doesn't get enough work behind the ball to be effective in Tests.
In limited overs it's a different thing; batsmen looking to score find his subtle variations hard to line up, and he seems reasonably intelligent in containing them. Plus a lack of accuracy doesn't hurt so much when you are bowling 6 balls in 6 different spots anyways, with 4 of 5 men back on the rope to cover for you if you miss your mark. Plus you don't really need to spin the ball a lot, just get enough movement off the pitch or in the air that you miss the middle of the bat and get the mis-hit.
In Tests he just doesn't seem as good at setting batsmen up or working them over, lacks the accuracy to bowl in the right areas for long spells (hence the inevitable pressure releasing ball), and doesn't put enough revs behind the ball most of the time to get serious purchase off the wicket, so doesn't trouble batsmen who are content to play late, see him off, and just put away the bad balls.