Not really. Sigs are the last thing on the page to load, are cached so you never have to redownload em, and for anyone on a remotely decent connection, downloads quickly.
I think Chris D's sig is on this order of magnitude, so I figured it was fine. ;-)