Quite true that a large majority of men (and women generally as they mature faster) stop growing in height around 20 years of age, except for the late bloomers.
In badminton, height is a significant advantage but you still need the skills to take advantage of it and, as in everything in life, there are pros and cons, nothing is all one way.
For a very tall person, it's a definite advantage to reach a high shot but, on the other hand, it can be a bit of an effort or strain to reach a very low shot and get up immediately as quickly as a much shorter person, I suppose.
Even so, as can be gathered from some of the height statistics of players and their tournament results, height does matter a little but it's not the main determinant, skill is the ultimate factor of success.
Notable 'short' players at or among the top include Lee CW (1.70m or 1.72m?) as you've mentioned, Wang Shixian (1.68m), Susi Susanti (1.61m).
I'd say,even Lin Dan at 1.76m or 1.78m according to certain sources is not exactly tall on average.
I remember watching the match between Ivanov/Sozonov and Ahsan/Setiawan. The height difference between the pairs was almost comical, especially when Ahsan and Ivanov were standing face to face (or face to chest, I suppose) at the net.