I'm impatient - there's no denying that fact.
When I ssh into this one CentOS box, I type my username, hit enter, and then it delays for perhaps 5-10 seconds. I never knew why it was, and it only bugged me enough to annoy, but today, it pushed me over the edge, and a quick Google search on "centos ssh password delay" found a nice solution:
In the /etc/ssh/sshd_config file, uncomment the line:
GSSAPIAuthentication no
and add the line:
UseDNS no
then: /etc/init.d/ssh restart
And BANG, that annoying delay is gone from then on!
[EDIT] If that is not the main problem, you may find that adding the IP address and a hostname for the client machine to the host's /etc/hosts file might also fix this problem.