Slow certificate

I have created my own Certificate to install SSL for OWA 2003 on my Windows 2003 Server.  When I used my own Certificate to create my SSL site, accessing the site is very slow.  It takes approximately 15-20 seconds for the authentication screen to pop up and then the logon screen for OWA loads.  Once I'm logged on to OWA, it's screens load instantly and it runs fine.

If I install the certificate on, say my home machine, the SSL site loads very fast (just like if I had bought a verisign certificate or a similar trusted certificate). Is there a way to make my self-signed SSL site load quickly from a PC that has not added that certificate to it's trusted root authorities? I don't care if they have to click yes at the security warning page, as long as they don't have to wait 20 seconds.

Again, I'm running IIS 6 on a Windows 2003 server (using OWA 2003, but that shouldn't matter, as it is slow for any web site I set up with my self-signed certificate)