I think that's a bit of a red herring there. I have no issue with removing them as they don't deserve Test status based on their performances but frankly, aside from the fact that it goes into the record book as Tests, it's not a huge deal. They play two test series that people just ignore anyway, and don't pay attention to. With the amount of other cricket that's constantly being played, it's irrelevant except for statistical purposes (e.g, Bangladesh games go into the Test record).
I don't see how people in India, for example, who never watch Tests anyway, including India-Australia ones, will care one way or another whether Bangladesh play a Test or not. The Test-loving population might care, but most of us ignore those Tests anyway, unless it happens to be our side playing them, and even then, with so much other cricket around, it's pretty much a short interruption.
If you have international sport, there'll be mismatches. I don't know an international sport where you don't have mismatches. The only way to avoid mismatches, or one country outproducing another for whatever reason, is to have a controlled league based system: e.g, the IPL, where all teams are on an even footing. Test cricket will never be on an even footing considering the startling differences between the Test nations and their infrastructure, interest, and a bunch of other factors. Sometimes countries will be lucky and be on top, but the idea of seven-eight countries competing for any large length of time on any sort of an even footing is fantasy. Unless, of course, you introduced things like the IPL and standardized everything. But obviously, that can't happen with countries.