He certainly descended into hard times, though they didn't quite so quickly as you say there Sir Nev, to my knowledge. To say he scored that one Test century and did little else is somewhat unfair. For several years Bannerman was widely recognised as the best batsman in Australia, and at least one English scribe even called him the second best in the world after WG. While it's true that his legendary Test ton was his only one in FC cricket as well, he hit several more centuries on tour for Australia I believe in matches which, while not acknowledged as FC at the time, possibly would be today - he made a century against Leicestershire in a tour match for example, IIRC.
It's true though that within a year or two of that 165 his decline began, and it was written that "Bannerman's light went out at the end of 1878." He ceased being a great batsman and became merely a reasonable one - indeed, Phillip Derriman wrote (and I'm trying to remember the quote off the top of my head so forgive me if it's out a little) that "Bannerman descended not into obscurity but, which is perhaps worse, into mediocrity. For the second half of his career he was something of a hack, scoring well enough to maintain his place in the New South Wales side, but never well enough to be selected for anything more."
He came upon difficult times later in his career, though it seems that this was more a case of hard living and burning the candle at both ends, as it were, rather than any particular misfortune. His friend, the great Australian bush poet A.B. "Banjo" Patterson, famously referred to him in prose as "Poor old Charlie Bannerman." Surprisingly enough, and to his credit, he didn't let it defeat him and managed to forge a second career as an umpire and eventually living well into his 70s, I believe.