There's definitely a grain of truth in that; a fair few "gentleman" amateurs played at a level of cricket their talent didn't necessarily merit.
However there's more than enough evidence of test cricketers achieving success in other sports to suggest they weren't all effete arseholes who couldn't cut it in "proper" sports.
Herbie Collins won an NSWRL premiership with Easts, Warwick Armstrong played in the VFL for South Melbourne, ditto Keith Miller for St Kilda, Wally Hammond played professional football for Bristol Rovers, Dennis Compton ditto for the Arsenal (winning the league in 48 and FA Cup in 50) and Les Ames for Gillingham.
- W.G. Grace was the national champion of 440 yard hurdles and represented England at bowls.
- C.B. Fry tied the world long jump record, played for England and football and reached the FA Cup final with Southampton, and played for the Barbarians at Rugby.
- Brian Close played football for Leeds United and Bradford City.
- Patsy Hendren played football for Coventry City, Queens Park Rangers, Manchester City and Brentford.
- Bill Edrich played football for Tottenham Hotspur.
- Reg Foster captained England at football.
- A.E. Stoddart captained England at rugby union.
- Ted MacDonald played football and rugby for Victoria.
- S.M.J. Woods played rugby for England
- Gregor McGregor played rugby for Scotland
- Jimmy Sinclair played rugby for South Africa and England
- Maurice Turnbull played hockey and rugby for Wales and was South Wales squash champion.
- Victor Richardson played baseball for Australia and represented South Australia at golf and tennis
- Johnny Douglas won the gold medal in the middleweight boxing in 1908 Olympics in London.
Bradman was a brilliant tennis player and had the opportunity to turn professional if he had not opted to focus on cricket. At 16, he had to choose between tennis, at which he excelled as a country champion, and cricket. He agonised over his decision and never ceased to test himself against the best, taking great satisfaction in beating every Australian Wimbledon player in his era in "friendly" matches.
He was a brilliant squash player and won the 1939 South Australia squash championship having not played squash for nearly two years until three months before the tournament.
He was the best billiards player in the Australian squad during the Ashes tours of 1930, 1934 and 1938. In 1934 he was beaten by the world's No. 1, Walter Lindrum. Bradman's competitive juices flowed. He had a billiards room built in his new Adelaide home. "He practised every day for a year," said Lady (Jessie) Bradman proudly, "and then challenged Lindrum to a return encounter and matched him."
Then in his early 30s, he won multiple sprint races at the Army's School of Physical & Recreational Training at Frankston, Victoria in 1940.
He was a superb golfer who won the Mount Osmond Golf Club championship in 1935 and 1949, and shot under par at every major golf course in South Australia and Victoria. He and continued to play until he was well into his eighties when he regularly ‘broke his age’.