Hotmail email deliverability problems and how to fix it!

July 23, 2008

For our latest project – I’m in TV, we used VPSLand to host the website. We were using VPSLand for the first time, and after a few initial hiccups, we thought we had everything in place. Or did we?

About a week from launching the first private beta version of the website, we realised that our emails to hotmail accounts were vanishing into thin air! This was extremely weird – because hotmail would accept the delivery of our emails and then just eat them up.

Emails would not get delivered to the members – nor would we get a bounce email back.
Hotmail would just quietly eat up all the emails that we threw at it – and not even give us any indication. No wonder as none of the users who signed up with a hotmail account were able to verify their email addresses.

We were extremely concerned with this issue and were at our wits’ end as to what was going wrong. This had not happened before. We had to finally, temporarily suspended hotmail signups for a week until we figured out what is wrong …

We spent an entire week on researching this issue. We tried permutations and combinations of email headers, what encoding to use, sending purely text and purely html emails, validating or SPF records and revalidating them again. Trying various permutations of our SPF records — but nothing worked. We were back to square one.

Then finally, frustruated – I decide to write an email to the hotmail support staff and see what happens.
I was hardly expecting a reply back – when they actually did. I was pleasantly surprised.
A few more email exchanges later – they told us that they were blocking our IP for no apparent reason at all! Hotmail does that.
Any new IP that tries to send it email – it blocks it assuming its spam.

Finally, after letting them know of our intentions, they agreed to open up a small window to allow our emails through. They would then monitor these emails for reports of spam – and if they found nothing suspicious, they would open up access permanently.

Phew! It finally worked!
After a painful week of researching, all it took was to ask them to do it manually — add us to a temporary white list.

However, for those of you who face the same problems, here is what you need to do first before hotmail will give you that small window of opportunity. Also, the application matters.
We were actually not spamming our users. All emails sent were either verification emails, password reminders or friend notifications.

So, your mileage might vary.

Make sure your SPF records are in place and are accurate.
If you don’t know what SPF records are, check out the following links to get you started:

  1. Information about SPF and why it is important:
    http://en.wikipedia.org/wiki/Sender_Policy_Framework
  2. SPF Record generator tool by Microsoft
    http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/
  3. Validate your SPF records:
    http://www.kitterman.com/spf/validate.html
  4. Sender ID validator from ReturnPath
    http://senderid.returnpath.net/how.php

Make sure your server is actually sending mail without problems to other providers.
In our tests, GMail was a peach – and accepted everything we threw at it. Mostly in the Inbox and sometimes in the SPAM. But it did accept it and never bounced any of our emails.
Yahoo too accepted all our mail – but put it in the Spam folder. The easy way to get out of this mess is to get as many people as you can to mark your email as NOT SPAM and also add your sender email ID to their contacts list.

A cool mail relay testing tool that we used was:
http://www.abuse.net/relay.html

Make sure you are following the Microsoft Postmaster Best Practices.
These can be found here:
http://postmaster.live.com/Guidelines.aspx

Finally, if you are doing more or less everything mentioned above, and your email still refuses to go – write to the postmaster.
https://support.msn.com/eform.aspx?productKey=edfsmsbl&ct=eformts&scrx=1

This is a long form but needs to be filled.
Once you have done this, a hotmail support person will get back to you generally within 24 hours and help you from then on.

In our case, we had to make some corrections to our SPF records according to their policies and when they were satisfied, they let us through.

I hope this helps and provides some insight for all those stuck with a similar problem.
Please do note however, that this is not always guaranteed to work. It worked for us and I am posting this from my experience.


The Bigger Picture …

July 6, 2007

As you can see, we have successfully setup our blog and it seems to be working fine :)

If you look closer, you will notice that this blog is hosted on wordpress.com and not on our own server.

Pah! You’ll exclaim -“Whats the big deal in that?”
You just log on to wordpress.com, fill out a couple of details, choose a theme, edit the custom image and you’re done. Anybody could do that – so what is infact the big deal?

Before I go on, let me explain our dilemma …
The host from which we have obtained our server space, for the ThinkingSpace Technologies website, gives us a lot of options in the .Net package – but extremely poor permissions to run PHP code.

Almost all the major (and cool) blogging engines (including wordpress) are in PHP – and they failed to run on our servers due to the permission settings set by the host.

We then turned our attention to .Net blogs – and came across a few good ones – dasBlog and SubText.
However, this time, the Asp.Net permission issues kicked in and we had trouble setting these up too …

We spent quite some valuable time getting squares to fit through round holes — code (which we had not written) needed to be edited and the entire application recompiled to run on our servers …

And even after this, the results were not what we had in mind …

We pushed on – wasted more time to try and get applications to run on our servers which clearly – the server was not happily letting us run.
Thats when it hit us – we were missing the bigger picture …

The bigger picture in setting up the blog was to provide a channel for us to express our thoughts and views …
We needed a blog – not a blog that ran on our server specifically (though that would be awesome).

Hence, we kept our (geeky) egos aside and decided to go with a blog hosted on WordPress instead of waste more time in trying to set one on our own server. (There was even a lot of talk about writing an entire engine ourselves to host on our servers … but we soon realised that that would be trying to kill a fly with a bazooka).

Hence here we are, with a blog on WordPress – customised exactly to our needs …
With the bigger picture – back in mind…

- Saurabh