Lessons learned after Google nearly killed my site
I am the creator of Uploader window, a website that helps users to add an upload widget to their own apps or websites.
We’ve been running this website for almost 3 years without any issues (since 2017). But something happened yesterday that almost shutdown my entire business at a moment’s notice.
My Whatsapp started ringing. In addition to email support I give many paying users my personal WA number for emergencies. It was a message from a client using our services for over 2+ years now. He said that they are getting a warning on their site because of our script. Anybody visiting any site in which my clients integrated my app were also showing this red screen of death.
I opened my Inbox and there was another email from a client about the same issue.
Now we run automated tests to monitor server uptime and check server for problems every 30 seconds. Unfortunately automated test scripts were happily getting HTTP/200 replies while people using the Chrome browser were being told this is a scam business trying to steal their bank account information.
Not knowing the first thing about what had happened I started looking for ways I could fix it quickly. And this is when I realized that while it was something new to me, it has been happening for a very long time and many businesses are affected by it. All because this whole process for getting blacklisted is pretty automatic and you don’t really know about it until you’re actually blacklisted.
Lesson #1: The first place to look is Google webmaster tools
I never got any email from Google but as many articles suggested the best place to start is inside the GWT (renamed Search Central). As soon as I added my site to GWT, I got a big red security issue that my site has been categorized as a deceptive phishing site which harms customers. It also listed my homepage as the deceptive page.
It was issued a few hours ago and I was already on 5 blacklists in a matter of a few hours. Soon firefox users started seeing a warning too.
I checked the source code of my homepage, checked all Javascripts by hand, manually logged in to my server and checked all running processes on my servers to see if in fact we had been hacked. Nothing, absolutely nothing was changed.
There was a box to submit your site for a review but it didn’t say how much time it would take. Some said it may take upto 4 days, other sites said it could be even longer! My business would most certainly be dead if it took that long.
Nevertheless I submitted for a review and that was it.
Lesson #2: Hacker news is the next best place to ask for help
When searching for why this has happened and ways to fix it I stumbled upon similar threads from Hacker news. It seems a lot of other people have been victims of Google's trigger-happy blacklisting and it’s not an easy fix.
Once you’re listed, you get bumped off Google search results and all users are banned from visiting your site (some users said they couldn’t even access it after accepting the advanced warning). Basically it’s `poweroff -h` for your site.
But there are plenty of Google engineers and good helpful people on Hacker news. People who go out of their way to help others
I was super tempted to contact this person directly but I decided it isn't very polite to bother people like this. Instead I posted my own thread mainly looking for advice on how I can expedite the review process and maybe learn what the hell had happened yesterday.
I listed all the reasons I could think off and people were quick to help. My call for help quickly reached HN’s homepage and I finally figured out what had happened
Lesson #3: Never link to any user content from your main domain (or any subdomain or even redirects)
So after a lot of brainstorming and ideas from HNers I finally figured out the culprit(s).
We have a live demo on our home where people can upload a test file. The demo is a way for users to actually see how the uploader will look in their own apps. It's a way for users to test drive the site without actually signing up.
Visitors love the demo. It greatly helps our conversions. I did think the demo could be used for file piracy so any files uploaded to the demo are automatically deleted after 24 hours. We never allow anything other than video and image files either. And we host the files on a cdn, so they are linked using a subdomain (cdn.uploader.win)
We also give all users a 20MB test storage. This is to help them integrate the Uploader in their own site during development.
I believe that somebody signed up for our service (it’s free to sign up) and then uploaded a malicious file on our test storage and abused this feature.
We cannot unfortunately disable the live demo or remove the test storage because they both serve a purpose. But there are few things we can do to mitigate it in future:
- Link to any user uploaded content using a separate domain altogether (never main domain or subdomain).
- Expire test files quickly. Every few minutes instead of 24 hours.
- If you are giving users any scripts to integrate on their site make sure to host them on a separate domain.
Lesson #4: Don’t use base-64 images (or inline images)
I’m not 100% sure about this but while I was frantically trying to scan my website using various online services a lot of them warned about using base64 content in the homepage.
Also if you remember Google did not say it was the subdomain rather it linked directly to my homepage as hosting the deceptive content.
The reason for adding inline images is to reduce the number of HTTP requests (which speeds up the page load). This got us a good score on Pagespeed insights but at the same time a lot of online virus scanners do frown upon pages that do it.
Since I will never know what I did wrong I just removed all inline content (as suggested by some HNers).
Again this is just hypothesis and it may or may not have been the case... but why take chances?
Lesson #5: Communicating with your customers can be super helpful
My biggest worry in all this was how it would affect my customers who depend on my site for uploads. I was feeling so helpless and anxious.
So I decided to instead communicate directly with some customers while I was waiting for resolution. Not only were most of my customers super supportive but I also realized that most of my customers are just developers like me and understood that things like this happen and cannot be controlled.
Such words brought down my stress level from 100 to 60 :)
Finally Good News!
The whole ordeal was sorted in only 6 to 7 hours and things are great again! I believe the HN thread getting on the homepage tremendously helped me and somebody from Google saw it and expedited the review after all (or maybe I was just lucky who knows). In just 4 hours after posting the thread I heard from Google (got an email this time) that my site was reviewed and removed from blacklist.
It was a great learning experience for me but also a little scary how fragile online businesses are. You can write all the cron jobs to monitor every single part of your business but you can never be prepared for everything.
And when shtf, the best solution is to always be open about what happened and ask for help. There are so many good people who will go out of their way to help you out.
So special thanks to all the people at Hacker news who were not only super helpful with their suggestions and comments, but also helped me get this sorted quickly!
P.S. In case you learned something from my experience or liked reading this, please take a minute to check out Uploader window by clicking here. It’s 100% Free to try and it can reduce your development times by a lot!
Share this post
