There's a bit of comparing apples and onions going on here.
One, tennis or golf in their usual forms are purely individual sports, whereas a lot of the skill in soccer or rugby is in team coordination and knowing where the other guy will be intuitively. There's no point in being able to send a pinpoint pass if no one's there to receive it.
Two, the time span involved doesn't compare. A footballer may play one or two matches in a week -- say three hours of competitive play in seven days; in a Grand Slam, a tennis player has up to seven matches, maybe three hours each, over two weeks; golfers four rounds of what, five hours each, over four days... How can you compare the stamina required? How can you compare running 100 meters to running a marathon?
Cricket crosses these barriers; it's about individual performances within a team context, played over long sessions; so all the different components of skill, team work and endurance are required.
I imagine if you were to define 'difficult' as the difficulty per individual per minute, then winning an Olympic Gold in the 100 meters would be the most difficult. But Roger Federer needs to do more to be No. 1 in the world over a year...
If you discount individual sports and only include team sports, using 'difficulty per individual per minute' I think soccer wins.
But if you think of difficulty per individual per week, cricket is probably only challenged by baseball, and given that baseballers are rotated between games, cricket wins.