Hotmail email deliverability problems and how to fix it!

July 23, 2008

For our latest project – I’m in TV, we used VPSLand to host the website. We were using VPSLand for the first time, and after a few initial hiccups, we thought we had everything in place. Or did we?

About a week from launching the first private beta version of the website, we realised that our emails to hotmail accounts were vanishing into thin air! This was extremely weird – because hotmail would accept the delivery of our emails and then just eat them up.

Emails would not get delivered to the members – nor would we get a bounce email back.
Hotmail would just quietly eat up all the emails that we threw at it – and not even give us any indication. No wonder as none of the users who signed up with a hotmail account were able to verify their email addresses.

We were extremely concerned with this issue and were at our wits’ end as to what was going wrong. This had not happened before. We had to finally, temporarily suspended hotmail signups for a week until we figured out what is wrong …

We spent an entire week on researching this issue. We tried permutations and combinations of email headers, what encoding to use, sending purely text and purely html emails, validating or SPF records and revalidating them again. Trying various permutations of our SPF records — but nothing worked. We were back to square one.

Then finally, frustruated – I decide to write an email to the hotmail support staff and see what happens.
I was hardly expecting a reply back – when they actually did. I was pleasantly surprised.
A few more email exchanges later – they told us that they were blocking our IP for no apparent reason at all! Hotmail does that.
Any new IP that tries to send it email – it blocks it assuming its spam.

Finally, after letting them know of our intentions, they agreed to open up a small window to allow our emails through. They would then monitor these emails for reports of spam – and if they found nothing suspicious, they would open up access permanently.

Phew! It finally worked!
After a painful week of researching, all it took was to ask them to do it manually — add us to a temporary white list.

However, for those of you who face the same problems, here is what you need to do first before hotmail will give you that small window of opportunity. Also, the application matters.
We were actually not spamming our users. All emails sent were either verification emails, password reminders or friend notifications.

So, your mileage might vary.

Make sure your SPF records are in place and are accurate.
If you don’t know what SPF records are, check out the following links to get you started:

  1. Information about SPF and why it is important:
    http://en.wikipedia.org/wiki/Sender_Policy_Framework
  2. SPF Record generator tool by Microsoft
    http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/
  3. Validate your SPF records:
    http://www.kitterman.com/spf/validate.html
  4. Sender ID validator from ReturnPath
    http://senderid.returnpath.net/how.php

Make sure your server is actually sending mail without problems to other providers.
In our tests, GMail was a peach – and accepted everything we threw at it. Mostly in the Inbox and sometimes in the SPAM. But it did accept it and never bounced any of our emails.
Yahoo too accepted all our mail – but put it in the Spam folder. The easy way to get out of this mess is to get as many people as you can to mark your email as NOT SPAM and also add your sender email ID to their contacts list.

A cool mail relay testing tool that we used was:
http://www.abuse.net/relay.html

Make sure you are following the Microsoft Postmaster Best Practices.
These can be found here:
http://postmaster.live.com/Guidelines.aspx

Finally, if you are doing more or less everything mentioned above, and your email still refuses to go – write to the postmaster.
https://support.msn.com/eform.aspx?productKey=edfsmsbl&ct=eformts&scrx=1

This is a long form but needs to be filled.
Once you have done this, a hotmail support person will get back to you generally within 24 hours and help you from then on.

In our case, we had to make some corrections to our SPF records according to their policies and when they were satisfied, they let us through.

I hope this helps and provides some insight for all those stuck with a similar problem.
Please do note however, that this is not always guaranteed to work. It worked for us and I am posting this from my experience.


Can’t have your cake and eat it too!

July 5, 2008

If you have gone through any of these project bidding and freelancing sites like RentACoder, Guru, etc., you will definitely come across hundreds of projects in which the clients have the following requirement:

The application must be of very good quality and very cheap also.

This is generally followed by a line which states that if “you deliver timely, good quality software which is also cheap, we will give you more projects in the future”.

I thought to myself – cheap and good quality… How does this possibly work?
(In my opinion, quality never comes cheap!)

There can possibly be two reasons for people posting this:

  1. People posting on such sites have extremely low expectations of the end product.
    If their definition of good quality is something which plain works, then I guess this point holds true. There are lots of companies out there doing really shoddy work – believe me – I have seen (20+ developers) software sweatshops which produce such low quality applications that some kids learning a new technology, hashing out their first application over a weekend would do a far better job.
  2. People posting have truly no clue …
    In this case, the people really want good quality software – but they have no clue on how much it actually costs to develop it. These people would be the most disappointed of the lot because more often than not, some desperate company or freelancers choose to do the project in the measly amount they’re paying and what they get in the end is really shoddy software (which eventually drives them to the category 1 people above).

Writing excellent software is a decently intensive process with thousands of things going into it.
At a birds-eye level, you need to do your research well, work up a good architecture, code well and finally test it thoroughly.

Moral of the story is that you cannot have your cake and eat it too!
If you want to get something done cheaply, you either have to cut down on the features (a highly recommended option) or settle for something which you need to cross your fingers hoping that it does not crash and burn.


On Copyrights in India…

June 24, 2008

If you have been following our blog, you will know that sometime ago, we had filed a copyright infringement case against another Indian company from Nasik for blatantly copying the entire user interface and functionality of ActiveCiti.
I agree that imitation is the best form of flattery – but blatantly downloading pages from someone else’s web property and then just doing a search-replace to put your name in — is just wrong!

Anyways, even though many people have told us that we’ve gone overboard with actually filing a criminal case against the accused company, I beg to differ. We have spent time, money and effort in developing something and we are going to defend it to the best we can.

During the entire case – which is already 8 months old, we have learnt quite a few things about Copyrights – and more importantly, “Copyrights in India”. In this post, I want to share some of this information with you…

The Berne Convention
The Berne Convention was an international agreement governing copyright issues and India is a signatory in this. What this means is that India will respect copyrights given by other signatory countries and other countries will respect the copyrights on works that India issues.

Copyrights in India
Copyright protects the artistic expression of an idea.
To put it more simply, ideas can be patented and expressions can be copyrighted.

As software cannot be (thankfully) patented in India, the best you can do is to get your work copyrighted.

Technically, all you need to do to copyright a website is put the words – “Copyright [Company Name]. [Years for which the copyright is valid]“
However, this does not work at all in the court of law – for obvious reason.

Hence, even though it is an expensive process, we went ahead and filed an application for obtaining the copyright – the legal way. To do this, we had to send all the material that we wanted to copyright (i.e. the entire website) in triplicate to the Copyright office in Delhi.

The Copyright office is located at:
The Ministry of Human Resource Development
Department of Secondary Education & Higher Education
Copyright Office

B2/W3, Curzon Road Barracks,
K.G. Marg, New Delhi – 110001
Tel: 3384387, -2549, -2458 / Extn. 31

Once they received this, they immediately issued us a receipt for the same (stating that our application is in process). The process generally takes 3 to 4 months to complete after which you will receive your copyright certificate.

Once done, you own the copyright for your work for 60 years.

Major changes made to the work warrant another copyright.
However, we are not very sure whether making subtle changes to the original work requires this.

Anyways, the process for obtaining a copyright is slightly complicated with quite a few forms to fill.
And the only copyright office being in Delhi, it makes sense obtaining the help of a lawyer or a firm which specializes in this.


We’re in *some* news …

July 31, 2007

Just a quick update …
We’ve been getting profiled and making the news some places ….

Though it isn’t really front page stuff, it feels nice to know that people are writing about you …
Just wanted to share the articles with you folks …

One article has been written in the mint this Sunday and can be read at:
http://www.livemint.com/2007/07/29190002/Three-startups-trying-somethi.html

While the other one is a blog and can be found out here:
http://go2indiaweb20.com/2007/07/20/activeciticom-the-evitecom-of-india/

(Though we’re quite different from evites – which is an online invitation and RSVP web application, we’re not complaining)

In the coming months, you will definitely see a lot of positive changes which will definitely set us apart from the rest of the websites you see in this domain …

- Saurabh


The Bigger Picture …

July 6, 2007

As you can see, we have successfully setup our blog and it seems to be working fine :)

If you look closer, you will notice that this blog is hosted on wordpress.com and not on our own server.

Pah! You’ll exclaim -“Whats the big deal in that?”
You just log on to wordpress.com, fill out a couple of details, choose a theme, edit the custom image and you’re done. Anybody could do that – so what is infact the big deal?

Before I go on, let me explain our dilemma …
The host from which we have obtained our server space, for the ThinkingSpace Technologies website, gives us a lot of options in the .Net package – but extremely poor permissions to run PHP code.

Almost all the major (and cool) blogging engines (including wordpress) are in PHP – and they failed to run on our servers due to the permission settings set by the host.

We then turned our attention to .Net blogs – and came across a few good ones – dasBlog and SubText.
However, this time, the Asp.Net permission issues kicked in and we had trouble setting these up too …

We spent quite some valuable time getting squares to fit through round holes — code (which we had not written) needed to be edited and the entire application recompiled to run on our servers …

And even after this, the results were not what we had in mind …

We pushed on – wasted more time to try and get applications to run on our servers which clearly – the server was not happily letting us run.
Thats when it hit us – we were missing the bigger picture …

The bigger picture in setting up the blog was to provide a channel for us to express our thoughts and views …
We needed a blog – not a blog that ran on our server specifically (though that would be awesome).

Hence, we kept our (geeky) egos aside and decided to go with a blog hosted on WordPress instead of waste more time in trying to set one on our own server. (There was even a lot of talk about writing an entire engine ourselves to host on our servers … but we soon realised that that would be trying to kill a fly with a bazooka).

Hence here we are, with a blog on WordPress – customised exactly to our needs …
With the bigger picture – back in mind…

- Saurabh


Hello World :)

July 6, 2007

What can I say? I am a geek…
Infact, even though some people will vehemently oppose it, we all are – at ThinkingSpace Technologies …

Which is definitely not a bad thing now is it?
When you know you’re a geek – you know you can get things done (technologically speaking).
Nothing is impossible and no mountain is too high to climb …

So, as all good (geeky) things in the world start, so does this blog with a simple:

Hello World!

Another thing mandatory to do with the first post is to write about what the blog is going to be about …

Well, ThinkingSpace Technologies is a small startup founded by 3 friends from the same class, same college – who quit their day jobs to follow a dream and chase a vision.

Part of the vision is writing extremely simple (shamelessly simple ;)) software which gets the job done and can be used by anyone without much computing experience – thus enabling a greater majority of people to benefit from technology.

What we plan to share with you through this blog is our journey …
The situations we face, our thoughts and perspectives on current trends, why we do somethings the way we do, etc. etc.

So there! :)

Welcome to the ThinkingSpace Technologies’ Blog.

- Saurabh