Web Application Firewall 2.0

The web is the most vulnerable interface on most servers. Having a powerful web application firewall is an essential part of the defense toolset if you host any web content. The biggest challange of a WAF is to find the balance between security level and false positive rate. You don’t want a weak web application firewall, but you can’t afford many false positives either. Finding this balance was the leading cause of the WAF 2.0 BitNinja module to be born.

Introduction

Generally speaking, web application firewalls monitor, filter, and block incoming and outgoing traffic on HTTP protocol. Through analysis, it’s a good protection shield against common web hacks, like injection flaws (eg. SQL injection), cross site scripting (XSS), session hijacking, remote and local file inclusions and other attacks.

We created an application layer firewall that is also known as a proxy-based firewall. You can easily activate it on the Dashboard (see instructions below).

BitNinja WAF 2.0 only scans incoming traffic currently, in real-time, with the built-in HTTP proxy solution. (For outbound traffic filtering there is a different module, the Outbound WAF.) Upon activation, BitNinja adds an iptables rule to your nat iptables table to redirect all traffic coming from port 80 to a port dedicated for BitNinja WAF 2.0. The WAF 2.0 process analyzes the incoming traffic and makes a connection to your local HTTP server to fetch the actual content.

We maintain filter patterns, and update and fine-tune regularly with new BitNinja versions. If BitNinja finds any suspicious connections, it will greylist the IP and send the incident to the BitNinja Analyzer Central.

Technical overview

Through the development of the original WAF module, we realized there are way too many features that we need to implement just to catch up with the open-source solutions. That’s why we decided to change our technology stack here and use industry-standard technologies. We started to look for different WAF models and then decided to choose a model used by CloudFlare and Incapsula. The WAF 2.0 is based on a very fast reverse proxy engine, the well-known nginx 1.12. Plus, supported by a well-known and very reliable WAF engine (mod_security v3) and a battle-tested ruleset from the Open Web Application Security Project, the OWASP crs v3.

This setup works very well. The performance is killer. In the near future, we will constantly update and finetune the ruleset, add other rulesets and implement more and more rules. We monitor the false positive rates of all rules and based on this we can tweak risky rules on the fly, and release the fixes very often.

We ship the custom-built and tuned nginx + lib-modsecurity package with bitninja. The package is called bitninja-waf and it installs in /opt/bintinja-waf

The waf-manager BitNinja module is responsible for managing the nginx+lib-modsecurity stack. It automatically installs, starts, configures, reconfigures and stops the nginx instances.

When nginx is started and it’s ready to accept connections for filtration, the waf-manager module sets up the redirection rules to redirect incoming traffic to the WAF 2.0. The WAF 2.0 then uses the currently used http server as a backend and pass the filtered traffic on.

Compatibility

Our bitninja-waf package is currently NOT compatible with the following Linux distributions:

  • Ubuntu < 14.04
  • Debian <= 6.0 (Squeeze)
  • CentOS/CloudLinux <= 5

Pre-requirements

Before you enable the WAF 2.0 module, there are some steps you have to take to enjoy the full functionality of the WAF module.

Add your server’s IP address(es) as trusted proxy

It’s very important to set your server’s IP(s) as trusted proxy to see the real visitors’ IP address in your server’s access log files, as well as to update the LogFormat directive in your web-server’s config file (e.g. httpd.conf for Apache) to use %a instead of %h:

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Using cPanel on LiteSpeed servers

On a LiteSpeed web server with cPanel you simply need to enable the Use Client IP in header section to “Trusted IP only” in the General Settings section on the Configuration page and add proxy IPs to the trusted IP list, as shown on the following page: https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:config:show-real-ip-behind-a-proxy

Installing mod_remoteip

When the WAF 2.0 module is active, all requests will be accepted by it and it will open a new connection to your appropriate local server. This way your local server will ‘see’ only the IP address of the WAF. A typical symptom of this error is when your HTTP server logs every request as it was made from the local IP or 127.0.0.1.

There is a standard solution for this problem. The WAF 2.0 will add an HTTP header to every request with the real IP address. The name of the header we use is X-Forwarded-For.

If you use apache2 as a web server, there are 3 modules you can use. They all can provide this functionality. We recommend mod_remoteip.

  • mod_remoteip
  • mod_rpaf (only up to apache 2.2)
  • like mod_extract_forwarded

Installing mod_remoteip on cPanel servers

Please follow the instructions provided on this website to configure your apache with mod_remoteip: https://grepitout.com/install-mod_remoteip-cpanel/

Additionally, you can use the following lines:

wget -O /usr/local/src/mod_remoteip.c https://raw.githubusercontent.com/infinitnet/mod_remoteip-httpd22/master/mod_remoteip.c
cd /usr/local/src/
apxs -i -c -n mod_remoteip.so mod_remoteip.c

Configuring mod_remoteip

On some of the RHEL-based distrubitions, the RemoteIp module can be a better choice, as Apache2 contains it by default. You can find an example configuration below. Please specify every IP of your server in the web-server’s config file.

<IfModule remoteip_module>
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy INSERT YOUR IP ADDRESS(ES) HERE
</IfModule>

You may also need to change the log format. The %h format string should be replaced with %a, for example, if you are using this log format

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

you’ll need to modify it to

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Installing mod_rpaf on Debian/Ubuntu

You can install the rpaf module under Debian with apt-get like this:

apt-get install libapache2-mod-rpaf

Installing mod_rpaf on CentOs and others

In RHE/RHL machines there is no pre-packaged version of the rpaf module, but you can easily compile the module by following the instructions below

# Download the rpaf module
cd /root
wget https://github.com/y-ken/mod_rpaf/archive/master.zip
unzip master.zip

# Compile the module
cd mod_rpaf-master/
apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

Now create the file /etc/httpd/conf.d/mod_rpaf.conf with the content below at the sample rpaf configuration part, and restart apache.

If you use apache2 and the rpaf module has been installed, you can enable it with

a2enmod rpaf

Here is a sample mod_rpaf configuration.

LoadModule rpaf_module modules/mod_rpaf-2.0.so
<IfModule mod_rpaf-2.0.c>
    RPAFenable On

    # When enabled, take the incoming X-Host header and
    # update the virtualhost settings accordingly:
    RPAFsethostname On

    # Define which IP's are your frontend proxies that sends
    # the correct X-Forwarded-For headers:
    RPAFproxy_ips 127.0.0.1 xx.xx.xx.xx

    # Change the header name to parse from the default
    # X-Forwarded-For to something of your choice:
    #   RPAFheader X-Real-IP
</IfModule>

Finally, restart Apache to apply the changes:

service httpd restart

Activating and Deactivating the WAF 2.0 module

Here is how to activate the WAF 2.0 module:

  1. Log in to https://admin.bitninja.io/
  2. Navigate to https://admin.bitninja.io/modules
  3. Select the server you want to activate the WAF 2.0 module on.
  4. Switch WAF 2.0 on.

BitNinja WAF 2.0 will be activated within 15 seconds.

To deactivate the WAF 2.0 module you have 3 options:

  1. You can use the Dashboard and switch the WAF 2.0 module off (preferred)

  2. You can use the command line (not recommended)
    bitninjacli --waf=disabled
    
  3. You can shut down BitNinja (this is the most radical solution, but it will also

    disable the WAF appropriately)

    service bitninja stop
    

Configuration

BitNinja WAF 2.0 - just like other BitNinja modules - is designed to require no configuration. Essentially it can be simply dropped in and it works with the defaults, but to avoid false positives and tune the security level the WAF 2.0 can be configured using the config panel the Dashboard (https://admin.bitninja.io )

When you start the module, the WAF will start to proxy all HTTP connections. By default, a safe minimum ruleset will be applied on all traffic, and the WAF 2.0 is in log only mode. To switch on the active protection, change the configuration of the default domain pattern and set the Action to “Challange and greylist IP”

The module configuration file is located at: /etc/bitninja/WAFManager/config.ini, if you need to fine-tune some of the settings on your server. In some cases, you may need to use your own custom-built Nginx with libmodsecurity. You can change the binary-related settings in the Nginx config section.

[nginx]
;binaryLocation  = '/opt/bitninja-waf/sbin/nginx'
;configLocation = '/opt/bitninja-waf/etc/nginx.conf'
;workingDirectory = '/opt/bitninja-waf/etc/'
;pidFileLocation = '/var/run/bitninja-waf.pid'
;pidfile = '/var/run/bitninja-waf.pid'
;group = 'bitninja-waf'
;user = 'bitninja-waf'
;configPermission = '0644'
;sciptPermission = '0644'
;updateConfigOnStart = 1

By default, WAF 2.0 will only create port redirections for public IPs. We had many issues with local/private IP HTTP checks. You can change this behavior by changing the redirection options to a per network interface basis.

;
; Enable WAF on Interfaces only.
; The default redirection option is for public ips only.
;
[redirect_options]
;interface[]='eth0'
;interface[]='eth1'

ModSecurity has very powerful audit logging, which can log every detail of a HTTP request/response. It’s a great tool for System Admins to find attack attempts. But it can consume a lot of disk space, this is why it’s disabled by default. You can change this behavior by changing the action manager settings. The default actions are: - LogToFile: save triggered incidents to /var/log/bintninja/WAFManager - SendToShogun: creates incidents and sends it to the API server

Available actions: - AuditLog: saves ModSecurity audit logs to /var/log/bitninja-waf directory - BlackList: USE THIS WITH CAUTION! It will instantly blacklist the IP that triggers the specific rule. Please use this option only with secrule 10, which will be generated after creating a virtual honeypot location with bitninjacli –waf-honeypotify-uri=/path/to/malware.

[actionmanager]
;log_dir = '/var/log/bitninja/WAFManager'
;auditlog_dir = '/var/log/bitninja-waf'
;default_actions[]='LogToFile'
;default_actions[]='SendToShogun'
;default_actions[]='Auditlog'
; Rule specific changes
; To instantly blacklist IP caught by user defined honeypot uri-s
;secrule[10] = 'BlackList'

Domain pattern configuration

You can add custom domain patterns easily to lighten or restrict rules or completely switch off the WAF 2.0 on a per domain basis. To assign special rulesets to a domain pattern, select the pattern first and use the sliders to enable/disable any rules/rulesets.

../_images/waf-1.png

1. This is the domain pattern. A domain pattern consist of a string and asterisk signs (*) For example mydomain.com/hello/ will match any request which contain the “mydomain.com/hello/” string. So it will match mydomain.com/hello, www.mydomain.com/hello and www.mydomain.com/hello/abc too.

2. This is the rule set currently applied on the domain pattern. You can change it at the (7) Ruleset template section.

3. Here you can find how many times the rule was triggered and how many times BitNinja recorded a false positive.

  1. To completely disable the WAF 2.0 for this domain pattern, set the checkbox.

5. You can lock down any domain to disable POST requests and make the site read-only. This restricted mode leaves the site available for visitors but prevents most of the further hacking attempts. The lock down mode is a very agressive way to protect a site, use only as a last resort.

6. You can specify actions to be taken when a rule is triggered: - Choose „Challenge and greylist IP” to maximize security and avoid targeted attackers and botnets from bypassing the WAF 2.0. In this case, visitors will be challenged by the BitNinja CAPTCHA page. - Choose „Log only” if you don’t want to enable the protection for this domain, but you want to see all suspicious logs generated by WAF 2.0. You can use this option to test the WAF before production use.

7. A ruleset template is a set of enabled rules. There are built-in ruleset templates (the ones starting with „BitNinja – „) but you can define your own ruleset templates, too. When you change a ruleset template by adding or removing a rule, all changes will be applied to all domain patterns using that ruleset template.

You can divert from the ruleset template anytime by adding or removing rules in case of a domain pattern, but the changes to the ruleset template will be still applied.

8. We provide open-source and custom-made mod_security rules and keep them up-to-date. You can activate or deactivate any rule for a given domain pattern. If you apply a ruleset template, it will switch on a set of rules, but you can still add more rules or deactivate some.

9. The ‘forked’ column will indicate how many rules are different (added or removed) compred to the originally selected domain template.

Safe implementation best practices

The safest way to introduce the WAF 2.0 on your servers is doing it by small steps. During the internal testing phase, we followed this implementation plan:

1. Enable WAF 2.0 on 1 server only: To minimize risk, enable the WAF 2.0 module first on only 1 server in Log only mode with the BitNinja default ruleset. Choose a server with a relatively low traffic to minimize any damage in case of a failure. If you have the old WAF enabled, and it was working fine, then you can enable WAF 2.0. It will automatically disable the old WAF, install and enable the new WAF 2.0 in Log only mode without disrupting the traffic. If you don’t have the old WAF running on your server, please follow the pre-requirement steps described on our documentation site. The old WAF and the new WAF 2.0 has the same pre-requirements. (Later we plan to make this step automated, too.)

2. Observe the WAF 2.0: In log only mode, you can see all the logged (but not blocked) incidents using the Dashboard. https://admin.bitninja.io/ipmanager/analyze Choose BL_BN_WAF from the Incident types and the server name you have enabled WAF 2.0 on. You will see the Log only incidents in light grey color.

Log only incidents won’t greylist the IP and the connection is not interrupted by the WAF 2.0. You can monitor the incidents and manually greylist the IPs if you find positive hits (hopefully most of them will be positives without false positives ☺ ). If you find any false positives generated by the default ruleset we ship, please send it us so we can tweak the default rules.

3. Expand to other servers in Log only mode: After a couple days of observation with the first server and the generated log level incidents, you can expand the WAF 2.0 to other servers, too. Please feel free to send us any feedback about the new WAF 2.0. We are happy to help!

4. Enable active protection on some domains: If the WAF works well in Log only mode, there is a high chance it will work well in Active protection mode, too. You can enable the Active protection on some domains where you see security problems or outdated CMS versions. I recommend to use the default rule settings we provide, but you can change the settings, too if you want. We will keep updating the rules.

5. Activate global protection: The last step is to turn on the Active protection on a server level, and use the per domain settings to create exceptions. For example, if there are false positives on one website because of a default rule, it is better to disable that rule only on that domain, so you can keep the protection level high on all other sites. Or you can even choose to completely disable the protection on a domain, or use more strict settings then the defaults.

Relation to other modules

The WAF module is sending incidents to the Shogun and AntiFlood module.

Log files

There are two different logs related to the WAF 2.0 module. You can find the logs of the BitNinja module responsible for managing the nginx + lib-mod_security here: /var/log/bitninja/mod.waf_manager.log

The logs generated by nginx + lib-mod_security part of the WAF is located here: /var/log/bitninja-waf

The detailed logs about triggered rules are located under /var/log/bitninja/WAFManager

The log files are automatically rotated if you have logrot installed on your server. We ship the log rotating rules with BitNinja.

You can use https://admin.bitninja.io/ipmanager/analyze to inspect the latest WAF 2.0 events.

Modifying Nginx reverse proxy settings

We’ve tried to create a globally working Nginx reverse proxy configuration, but sometimes you may need to modify some settings.

You can change the global Nginx configuration by creating an nginx config file that matches the /opt/bitninja-waf/etc/local_configs/global_*.conf glob file pattern.

You can even change the settings for only a specific location, if you create configuration files matching the /opt/bitninja-waf/etc/local_configs/<LOCATION_PATTERN_CONFIG_HASH>_*.conf glob file pattern.

A LOCATION_PATTERN_CONFIG_HASH is a 16 character long hash, containing numbers and characters of the English alphabet. E.g.: 098f6bcd4621d373

This hash is generated when you create a location pattern on the Dashboard. You can find yours - if needed - in the /opt/bitninja-waf/etc/nginx.conf file.

The created local configuration files need to be valid Nginx configuration files. You can check whether your configuration file’s syntax is correct with the following command:

service bitninja-waf configtest

Configuration changes can be reloaded with:

service bitninja-waf reload

Is it production ready?

We have tested the new WAF 2.0 on more than 100 production web servers in the last 4 months of alpha test period without any problems. Many new features and bugfixes have been applied on it through the alpha period and now the module is ready for production deployments.

How can I handle false positives?

If you have found a false positive you have multiple choiches to handle it. The recommended method is to create a domain pattern and disable the rule causing false positive.

You can also decide to create a domain pattern to manage the false positive on multiple domains, too. For example, let’s say a rule is causing false positive on every Wordpress sites with a module called my_module. You can create a domain pattern like this:

/modules/my_module/

and deactivate the rule causing false positive.

As a last resort you can decide to change the default pattern, too. The / pattern matches every request that is not matched by any other domain pattern and you can change which rules are applied on it, just like any other domain pattern.

Does the WAF 2.0 work on AWS or Azure?

Yes, it does, and you only need to configure all the interfaces that the WAF 2.0 should listen on in the /etc/bitninja/WAFManager/config.ini file in the section called [redirect_options]:

You can check your interfaces using the following command:

ip -4 -f inet addr show

You should change the interface’s name in the config.ini according to your own settings, and remove the comment character from the start of the line, like this:

;
; Enable WAF on Interfaces only.
; The default redirection option is for public ips only.
;
[redirect_options]
interface[]='eth0'

After you’ve saved the config file, restart the WAF 2.0 module with:

bitninjacli --module=WAFManager --restart