Annoying cyborgs attack, distort analytics [UPDATED][SOLVED-ish]

Over the last couple of weeks, I’ve been dealing with a strange phenomenon: a substantial (but not crippling) amount of traffic suddenly came our way.  The characteristics of this traffic are:

  • it’s direct (i.e. — no referrer and not search traffic)
  • it’s all from IE browsers
  • it’s nearly all to the homepage
  • it’s widely distributed in terms of geography, network etc.
  • it’s of very poor quality — low time on site, very high bounce, very low engagement
  • its real — confirmed in multiple analytics packages
  • it flies under DDos radar because it is less intense than a DDos burst, and rather indistinguishable from real traffic.

This traffic just simply started one day, and has gone up or down a little bit since.  Here’s what I’ve been able to conclude:

  • it’s likely not bot-traffic in the traditional sense.  Assets such as the javascript and ads for the page are getting loaded along with the DOM.
  • It’s likely not human either — the pattern is too uniform and the quality universally crappy.

This traffic has characteristics consistent with both bot and human behavior — I think we should call it cyborg traffic!  The pattern is consistent with a voluntary browser-net of some sort (people whoring out their OS’s to a central service — see Roger Dooley’s proposition below) or some kind of malware that is involuntarily opening windows in users’ browsers (less likely.)  If this behavior did not seem to include older IE browsers, I’d also speculate that it could be related to prerendering, but that seems unlikely given the facts.

Others have noticed it too, some positing causes:

  • This thread on webmasterworld contains lots of people reporting and reflecting on the problem
  • Roger Dooley (the fellow who started that thread) has proposed with some good evidence that the whole thing is due to a shady entity called Gomez from a company called Compuware.  Roger currently seems to be waiting to hear back from these guys — I hope he does soon, and posts the results of any conversations.
  • A post appeared on the google analytics product forums reporting the same behavior
  • A response to the webmasterworld thread by @incredibill seems to indicate that he’s found a way, via the request headers, to distinguish this sort of traffic from human traffic.  Any chance you could share Bill?

For updates on this situation, see Roger’s Post, or check back here — I’ll update when more info comes to light.

[UPDATE March 5th, 2012]

More consensus that this is a botnet, but little specific additional clarity about the nature of the traffic involved.  Good additional discussion appears here.

Someone affected soul even posted a rollup of their logs, with user agents:
https://analytics-a-googleproductforums-com.googlegroups.com/attach/5aade66b7c1d07b6/user_agents.csv?pli=1&view=1&part=4

 

[UPDATE March 7, 2012]
Here’s the first potentially reasonable mitigation I’ve come across, (from the google product group thread, above.)

“BB_CCIT” Says:

We have been getting the same kind of traffic to our homepage now for 17 days. Slow enough that it doesn’t do anything but ruin our analytics and advertising impressions.

One way that we started filtering things out was…

1) If it is an internet explorer user
2) It has no referrer (direct traffic)

If so we mark the IP on our blacklist at the bottom of our fully loaded page. If we detect a mouse movement or click event using javascript, we then update our database and mark their IP address as a verified user via an ajax call. This filtering system basically allows the bot to visit our site once and after we blacklist them any re-visits to our site will receive a 404 page for them.

Even if a blacklist were not used, one could conditionally load analytics packages in this way … I think.

Additional update:  Google seems to be investigating.  A google staffer posted:

We’re still investigating this issue and I’ll keep you posted when there are further updates. We appreciate your patience.

[UPDATE April 27, 2012]  We’ve found a workable way to exclude this stuff from Analytics. Check it out here.


40 Comments on “Annoying cyborgs attack, distort analytics [UPDATED][SOLVED-ish]”

  1. Phil says:

    Read this post:
    https://groups.google.com/a/googleproductforums.com/d/msg/analytics/BsZ41iF2iFM/qW9nBG6M80oJ

    Looks like a DoS botnet (that has also installed FunWebProducts toolbar).

    I am waiting for ISP disable this, and AV to update.

    Note: this is not site monitoring software (gomez/siteconfidence) as it is comming from a very diverse set of IP and user-agents.

    Thanks

    Phil.

  2. Matt says:

    Thanks for this Phil … have you identified specific ways to distinguish this traffic from normal traffic? If so please let me know — our site is still affected. What’s more, I’d like to pass any specific info of this kind on to wordpress.com engineers who may want to know.

  3. [...] month I posted about a surge of illegitimate traffic we’ve experienced on Grist.  This traffic was [...]

  4. Matt says:

    I just updated this post to reflect the fact that we’ve found a pretty good way to mitigate this in the case of Google Analytics — in other words we now keep the bad impressions out of GA. Check it out here:

    http://stkywll.com/2012/04/27/annoying-robots-a-solution-for-google-analytics/

  5. [...]  The characteristics of the traffic of this botnet are suspciously similar to the sort of traffic I wrote about over a year ago when I worked at Grist.  In particular, the traffic instantiates JavaScript, identifies itself [...]

  6. These are really impressive ideas in about blogging.
    You have touched some fastidious points here. Any way keep up wrinting.

  7. Ahaa, its fastidious conversation on the topic of this post here at this weblog, I have read
    all that, so at this time me also commenting at this place.

  8. I’m extremely impressed together with your writing abilities and also with the layout for your blog. Is that this a paid subject or did you customize it your self? Anyway stay up the nice high quality writing, it is rare to peer a nice weblog like this one nowadays..

  9. kik login says:

    Hi, I do believe this is a great site. I stumbledupon it ;) I
    will revisit yet again since I book-marked it. Money and
    freedom is the best way to change, may you be rich and continue to guide other people.

  10. Thank you a bunch for sharing this with all people you really understand what you’re talking approximately! Bookmarked. Please also seek advice from my website =). We will have a hyperlink change agreement between us

  11. ktmnecro says:

    Have been experiencing this for a year to one of our URLs. We get about 15k requests per day from 1000s of IP addresses from mostly end user ISPs all over the world ( mostly us ). User agent is a mix of real windows agents. This really does seem like end user PCs being used for some automated task. Why are they hitting one of our URLs so often and for a year now beats me.
    I checked out if gomez was behind this by first calling them and seeing if they have our domain in their system. They said they did not and also told me that their “last mile” service would have a “gomez” identifier in the agent. I wanted to verify this so i downloaded and installed their client program which is called gomezpeerzone, activated it and then tcpdumped the traffic. Indeed it was used by their control to hit lots of sites, but at the tcp level all their requests did have “gomez” in user agent.
    So I’m still stumped.

    • Matt says:

      Yup that sounds exactly like what was happening to us … aside from the mix of user-agents. Do you know anything about the traffic? Does it initiate any DOM events?

      • ktmnecro says:

        It loads and executes google analytics javascript so I imagine it’s a fully functional browser.
        In our case it’s isolated to a single URL ( some older article/page ), but there were more pages experiencing this when i detect it it initially last year.
        Last year I decided to just 404 that page for everyone in the hopes that their system would go away and also i didn’t want the analytics bloat, but as I started to revisit this issue the other day, to my surprise, the traffic is still there.. about 10 requests per minute all day long.
        The gomez trail seems cold, but as i researched these gomez type “make money at home free” clients, there are 100s of them that are used to buy/trade traffic.
        I also though that maybe this is some botnet control mechanism. Your botnet nodes hit some public website URL constantly and if you need to “communicate” with your botnet, maybe reset a password or something, you leave some cryptic message in the comments of the page that your bots are scrapping. But then why the js execution.
        In anycase I started logging everything now and I’ll do some analysis in the next week.

      • Matt says:

        Hey –

        Very interesting stuff. In our case we were able to distinguish the traffic by its complete lack of DOM activity — ie: no mouse events, no key clicks etc … the solution on how to do that is here: http://stkywll.com/2012/04/27/annoying-robots-a-solution-for-google-analytics/

        My thought was this was always something to do with advertising. Do you run ads? The Chameleon botNet had a very similar character to what we experienced, and I wrote a bit about that here: http://stkywll.com/2013/03/20/annoying-robots-and-the-chameleon-botnet/

  12. Nesting boxes may be lined with wood shavings, sawdust or even shredded
    paper. One of the machine’s best features is its motor which is thermally protected to prevent overheating

  13. Hi there, I’m a fresh developer just starting and I need a portfolio.
    Do you need a webpage design at no charge?

  14. That seems to be like my own shower my family and i
    decided to buy only recently, extremely pleased about it for any person found
    on the fence with regards to buying one, get it done, you wont be sorry

  15. Rodrigo says:

    A large amount of excellent guidance on this web site, really need a steam shower unit within my bathroom

  16. this one is visually like my own enclosure i actually got a hold of just lately, very pleased with it for
    individuals found on the fence about getting one, get it done, you will not be sorry

  17. ebay.com says:

    Aw, this was an extremely nice post. Spending some time and actual effort
    to make a great article… but what can I say… I put things off a whole lot and
    don’t manage to gett anything done.

  18. Jasmine says:

    His final staged opera performance was in March 2004 at New York City’s Metropolitan Opera.
    The steps are very, very long and wide and pure marble.
    These hotels are full of safety and providing with
    all your needs.

  19. Hello, Neat post. There’s an issue along with your site in web explorer, would check this?
    IE still is the market chief and a large element of other people will omit your excellent writing because of this problem.

  20. If some one desires expert view about blogging after
    that i propose him/her to visit this website, Keep up the pleasant work.

    تشک بادی , تخت بادی , استخر بادی , قایق بادی , محصولات بادی اینتکس ,
    کاناپه بادی, مبل بادی,استخر پیش ساخته, استخر فریمی و استخر ایزی ست

  21. I almost never comment, but i did some searching and wound up here Annoying cyborgs
    attack, distort analytics [UPDATED][SOLVED-ish] | StkyWll.

    And I actually do have some questions for
    you if you usually do not mind. Is it simply me or does it appear like a
    few of the responses come across as if they are written by
    brain dead people? :-P And, if you are writing at other sites, I’d
    like to follow anything fresh you have to post. Could you make
    a list of every one of your social pages like your twitter feed, Facebook
    page or linkedin profile?

  22. constantly i used to read smaller content that also clear their motive, and
    that is also happening with this piece of writing which I am
    reading here.

  23. Attractive section of content. I just stumbled upon your web site and in accession capital
    to assert that I acquire actually enjoyed account your
    blog posts. Any way I will be subscribing to your feeds and even I achievement
    you access consistently quickly.

  24. We’re a gaggle of volunteers and opening a brand new scheme in our community.
    Your web site offered us with useful info to work on. You have done a formidable process and our whole neighborhood will probably be thankful to you.

    - cliquez ici
    - cliquez ici
    - cliquez ici
    - cliquez ici
    - cliquez ici
    - cliquez ici

  25. I all the time used to read paragraph in news papers but now as I am a uswr of internet so from noow I am
    using net for articles or reviews, thanks to web.

  26. I loved as much as you will receive carried out right here.
    The sketch is tasteful, your authored material stylish.
    nonetheless, you command get bought an shakiness over that you wish be delivering the following.
    unwell unquestionably come further formerly again as exactly
    the same nearly very often inside case you shield this hike.

  27. Hi there Dear, are you genuinely visiting
    this site regularly, if so then you will without doubt obtain fastidious know-how.

  28. Human female pheromones are invisible and undetectable. The law of attraction, revealed by Abraham-Hicks is the most powerful force in the universe.

    The science behind sexual smell and pheromones has
    been studied time and time again.

  29. http://www.akhbarbm.com

    أخبار بني ملال

    موقع أخبار مدينة بني ملال ، تجدد على مدار الساعة

    beni mellal, region tadla azilal, beni mellal et regions,
    أخبار بني ملال, بني ملال, بني ملال و
    النواحي , تادلة أزيلال ,
    فقيه بن صالح ,آخر أخبار بني ملال ,أخبار المغرب ,

  30. Lavendwr oikls are naturally caustic against harmful pathogens.

    They actually scavenge the dead remains of other bacteria that perished tto antibiotic.
    It is possible, however, foor such hosts to give the bacteria on others.

  31. I’ve been surfing online more than 3 hours today, yet I never found any interesting article like yours.
    It is pretty worth enough for me. In my opinion, if all webmasters
    and bloggers made good content as you did, the web will be a lot more useful
    than ever before.|
    I couldn’t resist commenting. Perfectly written!|
    I will immediately snatch your rss feed as I can’t find
    your e-mail subscription link or newsletter service.
    Do you have any? Please allow me recognize so that I may just subscribe.
    Thanks.|
    It’s appropriate time to make some plans for the future and it is time to be happy.

    I have read this post and if I could I wish to suggest you some
    interesting things or tips. Maybe you could write next articles referring to this article.
    I wish to read even more things about it!|
    It’s appropriate time to make a few plans for the longer term and it is time
    to be happy. I’ve read this publish and if I
    may I want to recommend you few attention-grabbing
    things or advice. Perhaps you could write next articles regarding this article.
    I want to read more issues approximately it!|
    I have been surfing online greater than 3 hours nowadays, but I never
    found any interesting article like yours. It’s lovely price sufficient for me.
    Personally, if all web owners and bloggers made
    good content material as you did, the web will be a
    lot more helpful than ever before.|
    Ahaa, its good dialogue concerning this paragraph at
    this place at this web site, I have read all
    that, so at this time me also commenting here.|
    I am sure this post has touched all the internet visitors, its really really
    good paragraph on building up new weblog.|
    Wow, this piece of writing is pleasant, my younger sister is
    analyzing such things, therefore I am going to convey her.|
    bookmarked!!, I love your site!|
    Way cool! Some very valid points! I appreciate you penning
    this article plus the rest of the site is also really good.|
    Hi, I do think this is a great web site. I stumbledupon it ;) I will come back yet again since I book-marked it. Money and freedom is the
    greatest way to change, may you be rich and continue to guide others.|
    Woah! I’m really digging the template/theme of this blog.

    It’s simple, yet effective. A lot of times it’s challenging to get that
    “perfect balance” between user friendliness and appearance.
    I must say you have done a excellent job with this.
    Also, the blog loads super fast for me on Opera. Excellent Blog!|
    These are genuinely enormous ideas in concerning blogging.
    You have touched some good points here. Any way keep up wrinting.|
    Everyone loves what you guys tend to be up too. This type
    of clever work and reporting! Keep up the wonderful works guys I’ve added you guys to my own blogroll.|
    Hi there! Someone in my Facebook group shared this site
    with us so I came to check it out. I’m definitely enjoying
    the information. I’m bookmarking and will be tweeting this to my followers!
    Superb blog and excellent design.|
    I enjoy what you guys tend to be up too.

    This sort of clever work and reporting! Keep up the awesome works guys I’ve included you
    guys to our blogroll.|
    Howdy would you mind stating which blog platform you’re using?

    I’m looking to start my own blog soon but I’m having a
    tough time selecting between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different then most
    blogs and I’m looking for something unique.
    P.S My apologies for being off-topic but
    I had to ask!|
    Hi there would you mind letting me know which web host you’re working with?
    I’ve loaded your blog in 3 completely different browsers
    and I must say this blog loads a lot quicker then most.
    Can you suggest a good internet hosting provider at a reasonable price?

    Thank you, I appreciate it!|
    Everyone loves it when individuals come together and share views.
    Great site, keep it up!|
    Thank you for the auspicious writeup. It in fact was a amusement account it.

    Look advanced to more added agreeable from you!
    However, how can we communicate?|
    Hello just wanted to give you a quick heads up.
    The words in your article seem to be running off the screen in Safari.
    I’m not sure if this is a formatting issue or something to do with internet browser compatibility but I
    figured I’d post to let you know. The design look great though!
    Hope you get the issue solved soon. Cheers|
    This is a topic that’s near to my heart… Best wishes!
    Where are your contact details though?|
    It’s very easy to find out any matter on web as compared to textbooks, as I found this article at this website.|
    Does your website have a contact page? I’m having trouble locating it but, I’d
    like to shoot you an email. I’ve got some creative
    ideas for your blog you might be interested in hearing.
    Either way, great blog and I look forward to seeing it improve over time.|
    Greetings! I’ve been reading your website for some time now and finally got the
    courage to go ahead and give you a shout out from Dallas Texas!
    Just wanted to say keep up the fantastic work!|
    Greetings from Florida! I’m bored at work so I decided to check
    out your blog on my iphone during lunch break. I enjoy the info you provide here and can’t
    wait to take a look when I get home. I’m shocked
    at how fast your blog loaded on my mobile .. I’m not even using WIFI,
    just 3G .. Anyways, awesome blog!|
    Its such as you learn my mind! You seem to know so much about this,
    such as you wrote the guide in it or something. I think that you
    could do with some p.c. to drive the message house a bit, however instead
    of that, this is excellent blog. A great read. I will certainly be back.|
    I visited multiple web pages except the audio quality for audio songs current at this site
    is actually marvelous.|
    Hello, i read your blog from time to time and i own a similar one and
    i was just curious if you get a lot of spam feedback?
    If so how do you reduce it, any plugin or anything you can suggest?
    I get so much lately it’s driving me crazy so any support
    is very much appreciated.|
    Greetings! Very useful advice in this particular post! It is the little changes that will make the most important changes.
    Thanks a lot for sharing!|
    I absolutely love your website.. Very nice colors & theme.
    Did you make this website yourself? Please reply
    back as I’m wanting to create my own personal site and would love to learn where
    you got this from or exactly what the theme is called. Kudos!|
    Hi there! This blog post couldn’t be written any better! Reading through this post reminds me of my
    previous roommate! He always kept preaching about this. I will
    send this article to him. Fairly certain he’s going to have a very good read.
    Thanks for sharing!|
    Wow! This blog looks exactly like my old one!
    It’s on a entirely different subject but it has pretty much the
    same page layout and design. Superb choice of colors!|
    There is definately a lot to learn about this topic. I love all of the
    points you made.|
    You made some good points there. I checked on the internet for additional information about the issue and found most individuals will go along with your
    views on this website.|
    Hello, I log on to your new stuff like every week.
    Your writing style is witty, keep up the good work!|
    I just could not depart your web site prior to suggesting that I extremely loved the standard info a person provide on your guests?

    Is going to be back incessantly to investigate cross-check
    new posts|
    I wanted to thank you for this great read!! I certainly enjoyed every little bit of it.
    I have you saved as a favorite to look at new things you post…|
    Hello, just wanted to tell you, I liked this article. It was inspiring.

    Keep on posting!|
    I drop a leave a response when I like a article on a site or if I have something to
    add to the discussion. It’s caused by the fire displayed in the post I looked at.
    And after this post Annoying cyborgs attack, distort analytics [UPDATED][SOLVED-ish] | StkyWll.
    I was actually moved enough to drop a thought ;-) I actually do have 2 questions for you
    if you usually do not mind. Could it be just me or does it appear like some of the responses look as if they are coming from
    brain dead folks? :-P And, if you are writing at additional social sites,
    I would like to follow you. Could you make a list all of all your communal pages like your linkedin profile, Facebook page or twitter feed?|
    Hi there, I enjoy reading through your post. I like to write a little
    comment to support you.|
    I every time spent my half an hour to read this web site’s
    articles daily along with a cup of coffee.|
    I constantly emailed this web site post page to all my contacts,
    because if like to read it next my friends will too.|
    My coder is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the expenses.
    But he’s tryiong none the less. I’ve been using Movable-type on a number of
    websites for about a year and am worried about switching
    to another platform. I have heard fantastic things about blogengine.net.
    Is there a way I can import all my wordpress content into it?
    Any kind of help would be greatly appreciated!|
    Howdy! I could have sworn I’ve been to this website before but after looking at some of the posts I realized it’s new to me.
    Anyways, I’m definitely pleased I came across it and I’ll be book-marking it and checking back
    frequently!|
    Great work! This is the type of information that are supposed to be shared
    around the internet. Disgrace on the search engines for not
    positioning this post upper! Come on over and visit my website .
    Thanks =)|
    Heya i am for the first time here. I came across this board and I find It really useful & it helped me out much.

    I hope to give something back and help others like you aided
    me.|
    Greetings, There’s no doubt that your web site may be having internet browser compatibility problems.
    Whenever I take a look at your site in Safari, it looks fine however, if opening in Internet
    Explorer, it has some overlapping issues. I just wanted
    to give you a quick heads up! Apart from that, fantastic blog!|
    Somebody essentially assist to make significantly articles I would state.
    This is the first time I frequented your website page and up to now?
    I amazed with the research you made to make this actual submit extraordinary.

    Fantastic activity!|
    Heya i’m for the primary time here. I came across this board and
    I in finding It really useful & it helped me out much.
    I’m hoping to give one thing again and aid others like you aided me.|
    Good day! I simply want to offer you a huge thumbs up for the great info you
    have got here on this post. I will be returning to your website for more soon.|
    I all the time used to study piece of writing in news papers
    but now as I am a user of internet so from now I am using net
    for articles, thanks to web.|
    Your mode of telling everything in this article is in fact nice, every one be able
    to simply know it, Thanks a lot.|
    Hi there, I found your website by means of Google
    at the same time as searching for a related subject, your site came up, it seems great.
    I’ve bookmarked it in my google bookmarks.

    Hi there, just became aware of your blog via Google, and found that it’s really informative.
    I am going to watch out for brussels. I will be grateful if you
    proceed this in future. Many folks might be benefited from your writing.
    Cheers!|
    I’m curious to find out what blog platform
    you have been utilizing? I’m experiencing some small security problems with my latest
    site and I would like to find something more safeguarded.

    Do you have any solutions?|
    I am extremely impressed with your writing skills and also with the layout on your blog.
    Is this a paid theme or did you customize it yourself?
    Either way keep up the nice quality writing, it is rare to see a great
    blog like this one nowadays.|
    I am really inspired along with your writing abilities as smartly as with the format to your blog.
    Is that this a paid topic or did you modify it your self?
    Either way stay up the nice high quality writing, it is uncommon to
    look a great blog like this one today..|
    Hi, Neat post. There’s a problem with your web site in internet explorer, may test this?
    IE still is the marketplace leader and a large part of other people will omit your great writing because of this problem.|
    I am not sure where you are getting your information, but great topic.
    I needs to spend some time learning more or understanding more.
    Thanks for excellent information I was looking for this information for my mission.|
    Hi, i think that i saw you visited my website so i came to “return the favor”.I
    am trying to find things to improve my website!I suppose its ok to use a few of your
    id
    \

  32. You just need to get the confirmation that the lawyer is updated with the latest changes or not.
    Only a specialized lawyer is well-versed with the clauses of the law and can determine how much compensation you are entitled
    to get. Too many drivers do not even hold a legal driving license.

  33. Hi! I could have sworn I’ve visited this website before but after going through many of the posts I realized it’s new
    to me. Anyhow, I’m definitely happy I stumbled upon it and I’ll be bookmarking it and checking back
    regularly!

  34. That is why it is always advisable to go in for loans which the interest can easily
    be handled or those that will not call for fines.
    re living from a absolutely noting stability objective, or if ou can’t reach no equilibrium
    regular monthly, ten keep the last expensive balances t i pssible to.
    Unlike monetary donations, it won’t help in the future;
    it is not applicable for the future.

  35. Good day! This is kind of off topic but I need some guidance from
    an established blog. Is it very hard to set up your own blog?
    I’m not very techincal but I can figure things out pretty quick.
    I’m thinking about making my own but I’m not sure where to start.
    Do you have any tips or suggestions? Appreciate it

  36. Shelley says:

    whoah this weblog is great i really like reading your
    posts. Stay up the good work! You recognize,
    lots of people are hunting rond for this info, you can aid them
    greatly.

  37. Alyssa says:

    Outlook will display a dialog box informing you that a junk e-mail has
    been detected:. Online pharmacy is a legitimate platform to buy prescription and non prescription drugs without even visiting the
    doctor. Pharmacists, who must earn a qualifying degree, prepare
    and dispense prescribed medications.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 555 other followers