Thursday, May 24, 2012Partly Cloudy 26°C
Announcements

An Update to the Virus Situation

Posted by Tim / February 26, 2010

A couple of days ago we began receiving notifications that some of you were getting warnings that the site was infected with malware or other viruses. As many of you know, this is not the first time we have been infected. At first, our own inspections didn't reveal any issues and after some initial testing with some of you who contacted us we believed that it was just a cache issue, where virus software like Avast was producing false positive reports by looking at cached versions of the site.

After posting about this issue on the site yesterday, we learned that the issue was potentially more widespread that we initially thought so we decided to take the site offline this morning to complete additional analysis - one internally and the other by a third party security specialist.

What we found is that the site had in fact been compromised again. Obviously, this disappoints us greatly. We do not want the site to be responsible for inflicting harm on the computers of those who visit it; and we are also mindful that this second breach suggests the site is not as secure as it should be and that you might rightfully question our ability to provide a safe, virus-free browsing experience.

The situation is not acceptable. It's not acceptable by us and we know it's not something that any of you should accept.

We apologize for this security breach and even though we believe the site is now clean we know that words of contrition and explanation are not enough. Obviously our future actions - both in responding quickly to any warnings AND in avoiding any subsequent security breaches are what's most important here.

Unfortunately we don't have much more information to share beyond what was discussed the previous time but I'll try my best to answer any question in the comments below.

Discussion

48 Comments

DS / February 26, 2010 at 07:26 pm
user-pic
Maybe its time to take this site off Dreamhost and pay for some managed hosting services.
Calling All Tech Staff / February 26, 2010 at 07:52 pm
user-pic
Your technical staff needs to read things like this:

http://www.communities.hp.com/securitysoftware/blogs/rafal/archive/2010/02/25/a-big-case-of-oops.aspx

You not only need to scrutinize your production code itself to make sure it is virus-free, you need to scrutinize it for SQL injection vulnerabilities, and you need to scrutinize your database contents - every comment ever written, every article ever written, every thing in your site databases - to make sure that none of the data has been infected, as described in that link.
JM / February 26, 2010 at 07:53 pm
user-pic
Why did blogTO delete the post asking your readers if they have had any virus warnings? Are you trying to sweep the fact you dismissed your readers concerns for at least two days under the rug?
Tim / February 26, 2010 at 07:59 pm
user-pic
We're just trying to centralize the discussion here. I mentioned the previous post above so not sure exactly what we'd be trying to hide.
Grant / February 26, 2010 at 08:06 pm
user-pic
I'm not sure of the specifics of how this site is hosted, but if you're using a shared host like Dreamhost or Mediatemple (both of which are notoriously bad) you should really look at getting some real hosting so you have more control over things like operating system (hopefully something Unix-based) and scripting system (PHP, Ruby, Java, etc.) updates. This would also give you access to knowledgeable, accountable support staff that will be able to help you out with these kind of situations. Also, if you're using an off-the-shelf blogging platform like Wordpress or Movable Type, you really need to aggressively keep up with updates to it (this is especially true of Wordpress).

This stuff sound be very elementary to most programmers, but clearly you guys aren't getting it because you're being infected over and over again. Luckily I'm on a Mac so this doesn't directly affect me, but it's beyond unacceptable that you can't seem to figure out basic systems administration, and even worse seem to feel that it's acceptable to continue to run a compromised server that is presumably continuing to infect your readers' systems.

Again, I hate to jump to these kind of conclusions without the whole story, but you guys aren't leaving me any other choice.
JM replying to a comment from Tim / February 26, 2010 at 08:08 pm
user-pic
Any discussion would probably lead to this post naturally. I'm not exactly sure what you'd be trying to hide either which is why it is odd to have that specific post deleted while the other post on the virus topic remains. Why not leave the post up for your readers to learn the story on their own rather than leave it to your own synopsis?
Angie / February 26, 2010 at 08:14 pm
user-pic
>> so not sure exactly what we'd be trying to hide.

This kind of defensiveness, while natural - I'm sure it's tough to not take these comments personally - isn't going to help you get back on track.

I think the point your readers are trying to make is that taking something OUT of the discussion is not helpful. Leaving up the old post, closing comments and then redirecting users to THIS conversation allows for continuity without the perceived "sneakiness".

More comments about this on my site as well:
http://www.angiemckaig.com/archives/2010/02/26/followups_crisis_management/
mac user / February 26, 2010 at 08:50 pm
user-pic
for a couple of weeks every time i go to this site, my safari quits on its own within seconds of viewing the home page. now, it works fine. i'm assuming its the "virus" that was causing that....
Frank / February 26, 2010 at 09:05 pm
user-pic
I suggest subscribing to something like "McAfee Secure" (Used to be called Hacker Safe).

http://www.mcafeesecure.com/us/

You subscribe, and every night it penetrate-tests your site, giving you specific advice on how to lock things down. It should point to cures for your SQL-injection problem and cross-site scripting vulnerabilities.

I do not work for McAfee--I am just a satisfied customer.
Xavier / February 26, 2010 at 09:12 pm
user-pic
I just got another notification.
Snickers / February 26, 2010 at 09:24 pm
user-pic
Still getting the same 4 virus notifications.
Tim / February 26, 2010 at 09:30 pm
user-pic
What do the notifications say.
MaSTer / February 26, 2010 at 10:40 pm
user-pic
Get a better hosting site! like what the hell man
rhonda / February 26, 2010 at 11:20 pm
user-pic
Here are the details of the threat report I get on this site:

Threat Report

Total threats found: 4

Small-whitebg-red Drive-By Downloads (what's this?)

Threats found: 4
Here is a complete list:
Threat Name: MSIE ADODB.Stream Object File Installation Weakness
Location: http://www.blogto.com/arts


Threat Name: MSIE ADODB.Stream Object File Installation Weakness
Location: http://www.blogto.com/fashion/publicbutter


Threat Name: Direct link to MSIE ADODB.Stream Object File Installation Weakness
Location: http://www.blogto.com/arts/2008/10/war_and_peace_conquers_the_canadian_opera_company/


Direct link to: http://www.blogto.com/fashion/preloved
Location: http://www.blogto.com/toronto/the_best_vintage_clothing_stores_in_toronto/

selfimportantmuch / February 26, 2010 at 11:29 pm
user-pic
post was overdramatic re: "this is not acceptable, etc". well, ok, get on it!
Mike W / February 27, 2010 at 01:59 am
user-pic
Is the deleted post the one in which users were specifying what system/browser/plugins they had and if they got warnings?

It seemed to me it could've helped ID any patterns. It's too bad I never got a chance to follow up on those posts..
E C / February 27, 2010 at 06:59 am
user-pic
`Hindsight is 20/20 as they say and if we knew then what we know now we certainly would have handled things differently.`

Those are the words of blogTO the last time this happened. Well, apparently hindsight is less 20/20 and more akin to sticking one`s head in the sand, as it took a couple of days for the proper reaction *again* on round two. I got less than lucky and my machine`s infected. Thanks, guys.

Not coming back to this site again -- and aggressively educating friends, etc. about the consistent security issues.
Andre / February 27, 2010 at 08:30 am
user-pic
No criticism here for the way you folks have handled things. I keep coming back because of your content. You're working hard and taking things seriously-- good enough for me. Hang in there!
James / February 27, 2010 at 09:18 am
user-pic
Look, I love BlogTO and this site really does put together some great coverage. But a Microsoft web platform? Its a bit of a joke. Time to change this.
Jonathan Lepos / February 27, 2010 at 11:05 am
user-pic
I'm not sure if my recent computer problems are from your site, but the situation sucks. If I decide to limit my blogTO reading strictly through my google reader, am I at risk?
The Situation / February 27, 2010 at 11:28 am
user-pic
The first time you guys were hit with a virus I sent you an email describing what happened and you said it was all taken care of. The next day i visited I got hit with the same viruses. I stayed off for a week and when I returned got hit AGAIN. Now a new round of viruses.
Luckily I haven't had to reformat but all these virus warnings I'm getting from this site is causing me to lose sleep.
My virus scans tell me I'm clean but how can I be so sure. Maybe some guy in Russia has all my personal info and stealing my identity. I don't even do online banking anymore in case I have some hidden keylogger or some shit.

I hope I don't end up dead from all of this.
JJ / February 27, 2010 at 11:37 am
user-pic
Appreciate the forthrightness, but come on guys... you gotta fix this.
Tim replying to a comment from The Situation / February 27, 2010 at 11:43 am
user-pic
Sorry to hear about this. We know many people have been adversely affected. We had hoped that the security fixes and previous server move would have solved things. We're working on both immediate and long-term fixes to try to rid the site of these infections.
The Situation replying to a comment from Tim / February 27, 2010 at 12:13 pm
user-pic
Its good to hear Tim. I know that you're now aware of the backlash you'll get from people when these matters do not get immediate full attention.
You can make The Situation happy with a new mac..HA! lol j/k :P
JM replying to a comment from Tim / February 27, 2010 at 01:20 pm
user-pic
When can we expect another announcement detailing these immediate and long-term fixes? The reason I ask is that this update was on the first page for all of one night - Friday night. It is possible many people who may not have visited last night (or on the weekend at all) do not know of their machines being possibly infected.

Also, the 'Update to the Virus Situation' box at the side of the site should be larger and a different colour than the rest of the boxes beneath the update.



Chris / February 27, 2010 at 02:36 pm
user-pic
I really like blogTO and have been visiting the site a couple times a week for over a year now. I usually view it on my Mac with Firefox, and noticed about six weeks ago that my browser was shutting down immediately upon navigating to the site. I realize now that this was a symptom of the problems discussed above, but I'm wondering if blogTO is also the reason I woke up three weeks ago to find my hard drive was fried so thoroughly that a reputable data-recovery service was unable to salvage anything from it. I guess I've always been a bit cavalier about web-security, thinking that as a Mac user I was less vulnerable to malware and other threats.
o_O / February 28, 2010 at 12:57 am
user-pic
Hope you guys get this sorted out but unless I'm on a Mac I'm not coming near your site ever again.
Matt / February 28, 2010 at 01:39 pm
user-pic
Tim, you really only have _one_ option if you're site has been compromised, flatten all effected servers and start from scratch. By flatten I mean format the drives, reinstall the OS, rebuild the application stack, and so on and so forth. Any hacker with a bit of experience will open up a dozen other inconspicuous backdoors to the site after a successful compromise. If you don't choose to flatten (ie. format and start from scratch) the problem will likely persist until such time as you do. You can apply fixes until you're blue in the face and you'll never close the holes. Google has decent overview of what should have been your plan of attack. Ignore their advice at your own peril.

http://googlewebmastercentral.blogspot.com/2008/04/my-sites-been-hacked-now-what.html
grrr. / March 1, 2010 at 11:12 am
user-pic
TAKE CARE OF IT.
My computer got totally &^$*(#$% up from your site (my fault, I didn't beleive the warnings). I love the site, but am scared to visit.
JM replying to a comment from Matt / March 1, 2010 at 01:32 pm
user-pic
What about option #2: Write a post on the weekend on how disapointed they are that this happened again and they're trying really, really hard to make sure it doesn't happen a third time. Then once the post is bumped off the first page 12 hours later, return to business as usual and pat themselves on the back thinking a tiny box on the front page is enough notice to their readers that if they have inadequate anti-virus, it's all but guaranteed they have a virus thanks to this websites security.
IT Guy / March 1, 2010 at 04:10 pm
user-pic
While I think there very may have been a risk for many people, I think a lot of people are blaming their own random computer problems on this site. Ie. Your hard drive was not fried by a drive by malware infection from blogto.

What would be a huge help, and also a way of stemming some backlash against blogto, would be to COMPLETELY DISCLOSE what infections/malware/spyware you were serving up while infected. That would allow people to more effectively scan/search their computers, and stop people from blaming you for infections that were not caused by blogto.

Moreover, rhonda, thank you for posting your infection notification results, but unfortunately, we need something official from blogto in regards to what infections were being served by the site.

Also, blogto should move to a linux/unix platform, something like centos that supports selinux would be a good choice. Moreover, make sure the 'security specialist' you hire does a full vulnerability assessment and then follows it up with a pen test.

Moreover, move to a new physical server, with new HDs for your setup, start from scratch and do it right from the ground up. Use one of your backups to import the new content and make sure every single file is scanned and checked before being moved to the new server.

I am assuming the reason you are hesitant to do this is that your content is perhaps locked into some kind of propriety setup as other had indicated (wordpress etc..)... what can I say, that sucks, time to hire some data entry clerks?
tim / March 1, 2010 at 09:58 pm
user-pic
Thanks for the continued feedback and concern everyone. Here are some updates since I made this post:

1. We continue to monitor the site for infections and it appears to have been clean since we made the fix on Friday.

2. We are speaking with a security speciality about engaging them on an ongoing basis. First, to do a complete audit and make recommendations. And, then, to monitor the site for any instances of infections on an ongoing basis. Note: Prior to the weekend we already have had the site audited by three external security experts and we've already acted on previous recommendations from them.

3. We are examining options for a new server environment. We are looking for a managed dedicated server. We are open to recommendations if anyone has any.

4. We are evaluating other internal policies/practices to help ensure the site is as secure as we can make it.
Eric S. Smith / March 2, 2010 at 04:48 pm
user-pic
It would really help your users and you if you were to list exactly what bad stuff your site was serving up, and how it got in the door. It will help us know if we were vulnerable, and it will help you to fend off (a little) comments like, "Great, my favourite mouse doesn't click any more. Thanks, BlogTO."

That you don't list the what and the how suggests that you still don't know -- or that you know that you're still vulnerable.
Chris / March 2, 2010 at 11:19 pm
user-pic
What exactly is happening to those infected?? Because my laptop crashed a couple weeks ago...
Halfmydadsage / March 4, 2010 at 05:48 am
user-pic
I too have malware after seeing the warnings.
I have had trojan after trojan. For the less IT savvy, we are probably not able to give you precise details but 3 weeks ago after seeing the warning... and then not realizing how serious the risk was...
I love the site but I too wonder what can be done. I am about to wipe my computer due to these malware issues that have plagued me for few weeks .... I have never encountered this problem...
it's not a mouse issues. It's trojan and redirecting problems that continue dispite cccleaner, AVG and loss of personal time and resources.
HUK / March 4, 2010 at 08:33 am
user-pic
I THINK WE SHOULD ARRANGE FOR A CLASS ACTION SUIT AGAINST BLOGTO BECAUSE OF ITS NEGLIGENCE. SUE SUE SUE!!!!!!!!!! WE ARE ENTITLED TO SUE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
JM replying to a comment from tim / March 4, 2010 at 10:45 pm
user-pic
Tim you wrote this comment on another blog on February 25:

"I appreciate the suggestions. I can assure you that updating readers who visit the site with more prominent messages of this nature is a top priority. In fact, if all goes according to plan, we will be unveiling some design changes next week that help facilitate this."

By this, do you mean you will be deleting the notice at the top of your main page that everyone complained was too small for its purpose only DAYS after it was posted? Or are you waiting for Friday at 8PM to mention to thousands of readers your site may have infected them with a virus?
Ryan L. / March 10, 2010 at 07:11 am
user-pic
I've tried contact Tim and other BlogTO staff to no avail. My work computer had been infected since the first round of virus problems. And while I had some initial success in removing the infection (after several hours of lost time), the damn thing came back, and I now see why.

I am greatly disappointed in the reactions of BlogTO staff in regards to this matter. They seem more concerned with sweeping this under the rug than addressing the fact that there are likely thousands of infected computers out there due to their lack of action.

Tim, if you want to be 'open' with people, why take the announcement link off the main page? I had absolutely no idea the site was infected again because I happened to miss the 3 hour window when this article was on the main page (mostly because I have been avoiding the site due to being scared about being reinfected from other machines).

If you want to ensure the site is 'secure', then why dismiss the initial comments where people stated that there was still virus warnings coming from the site as a user-end problem (inadequate cache clearing)?

You already admitted you were wrong in not telling people sooner the first time around, so why delay the second time?

And why were multiple emails ignored where I tried to get more information about the virus so I could have better luck trying to target it and improve chances of its removal? My work PC now has to be wiped due to all those things mentioned above. Time and most importantly Data has been lost. Should I be encouraging management to calculate financial losses as a result and have them send you the bill?

Suffice to say, my trust in BlogTO has been lost and unlikely to return.

"Hindsight is 20/20 as they say and if we knew then what we know now we certainly would have handled things differently."

So by handled differently, you mean do things exactly the same as before?

Tim replying to a comment from Ryan L. / March 10, 2010 at 10:38 am
user-pic
Ryan - I think it's a bit unfair to say you've tried contacting us and we haven't responded. I think at least 24 hours is a fair amount of time to get back to you given the number of emails we get. Anyway, as you know, we're now in touch with you and trying to help you out.
Ryan L replying to a comment from Tim / March 10, 2010 at 11:02 am
user-pic
The initial email was sent -two weeks- ago after trying unsuccessfully to acquire more info from the comment section of the first post regarding the virus. It's clearly quoted in the most recent email sent.
Tim replying to a comment from Ryan L / March 10, 2010 at 11:29 am
user-pic
Ok. I'm not sure what happened to that email and why you didn't get a response. Both the editorial team and I try our best to respond to all emails within 24 hours.
alan / June 9, 2010 at 10:09 am
user-pic
yesterday my computer security app picked up a virus just as i logged into blogTO....today i got the "red screen of death" basically telling me to navigate away from this web site....just saying...
Ed the Shlock / June 9, 2010 at 10:49 am
user-pic
Yep. Same.

What's going on, BlogTO?
Tim / June 9, 2010 at 11:09 am
user-pic
Thanks for your comments. We took the site temporarily offline and scanned it. We did remove a virus from a file and the file is now clean. Please let us know if you continue to experience any virus-related issues.
Dude / June 9, 2010 at 02:56 pm
user-pic
I got a warning screen at five minutes ago.
Tim replying to a comment from Dude / June 9, 2010 at 03:20 pm
user-pic
Do you have a copy of the warning? What did it say? What O/S and browser are you using? What page were you on?

Thanks for your help.
Bye / June 9, 2010 at 03:57 pm
user-pic
Too late, good-bye BlogTo, hello Torontoist. If you make your living off the Web you should know what the (&#% you're doing with it....
Eric S. Smith replying to a comment from Tim / June 9, 2010 at 04:12 pm
user-pic
"We did remove a virus from a file and the file is now clean."

<i>Jesus.</i> And here I was thinking that the new-look site was, if not as pretty, at least secure. My reluctance to send people links to BlogTO is vindicated, I guess.

Add a Comment

Other Cities: VancouverMontreal