Point is what you consider a "decent sample size" is entirely subjective. It really doesn't make a lot of sense to break up a player's career stats to even 5 test chunks and rate them through that unless the difference is very significant (Ponting in India or Warne in India for example are prime examples of cases that tell you something about the player).
In Sanga's case, he has done well against every team at least at home or away (especially since he became a pure batsman), and his exploits in NZ/Aus and recently in Eng show that he doesn't have some technical deficiency that stops him from getting runs in unfamiliar conditions.
Dravid in a recent cricinfo piece literally said Sanga had no apparent weakness - I think he knows a thing or two about batsmenship than most.
If 4 tests is not a decent sample size to compare for home or away against an opponent, then you're going to have trouble comparing players across most eras. Again, it's not great, but it's something. Pretending that 1 or 2 innings are enough however is absurd. There's a clear difference there.
Even Ponting's India record is weird if you logically break it down. Was he poor against spin in general? Not at all. IIRC he averages 50 in matches against Murali and his record against spinners not named Harbhajan is generally very good. Is it because he has a problem playing in the sub-continent? No, because India is his only poor record. Is it just about India in general even? No, as he generally walloped India in Australia.
However, one still has to make the concession that whatever the factor is - recognisable or not - it still exists. And to have 6 instances of <40, 3 home and 3 away, is just not good enough for that tier. That's in the Hayden/Sehwag tier*. One can be technically fantastic, no weaknesses, but performance matters. You could have only one reliable shot but if the result is you averaging 50 home and away against everyone, then the argument against your technical deficiency is moot IMO.
*Just checked to see what they have exactly, coincidentally both also have 6 instances of <40 averages home or away like Sangakkara. For interest's sake, I checked Jayawardene's and he has 7 instances. It doesn't seem to be a coincidence that these batsmen, while being lauded, have been questioned about their ability to play in all conditions.