MalwareDetection

BitNinja has an excellent module for file-based malware detection. If attackers can break through the defense line of honeypots and the web application firewall (see our other detection modules here: Detection modules), malware detection is the next line of defense to stop them infecting your server. It is also very useful to scan your server if you just started to use BitNinja for the first time, as there could be different malwares and backdoor programs on your server. Cleaning your files is vital.

We ship new malware definition files automatically so it is always up to date without any manual intervention.

Prerequirements

The module uses the inotify-tools package to monitor the filesystem for changes. You need this package to be available before you can use the module. We ship our own inotify-tools package with BitNinja to ensure the installation. If you need to install inotify-tools manually for some reason, please follow the steps stated here: Optional: manual installation of inotify-tools

Before you enable the Malware Detection module, there are some steps you have to follow on your server to prepare it for the launch of the module.

On most servers, default number of user watches have to be expanded from the default to a much higher number. BitNinja Linux Client increases it automatically up to 30000000.

If it is not large enough, you can increase it manually, e.g.:

echo 35000000 > /proc/sys/fs/inotify/max_user_watches

Now check if everything is ready to start the malware detection module:

cat /proc/sys/fs/inotify/max_user_watches
// This should be the result:
// 30000000
whereis inotifywait
// Something similar should be the result
// inotifywait: /usr/bin/inotifywait /usr/bin/X11/inotifywait /usr/share/man/man1/inotifywait.1.gz

// check if inotify-tools is on the latest version
// on deb-based systems
dpkg-query --showformat='${Version}' --show inotify-tools
// on rpm-based systems
rpm -q --qf "%{VERSION}\n" inotify-tools
// result: 3.14-2

You are ready to start the malware detection module now! Before lauching the module please read through the configuration section to prepare the module.

Configuration

To change the default configurations of the malware detection module you have to edit the module config file at /etc/bitninja/MalwareDetection/config.ini. We always try to set up reasonable defaults for our config files so you don’t have to mess with configs, but in this case there are some configurations you are probably willing to do before starting the module.

Example content for the config.ini file:

; Core settings
;
[core]
; do not check files over scan_max_file_size bytes
;scan_max_file_size = 786432 ; 768 KB 1048576 ; 1 MB
; scan process CPU scheduling priority
;scan_cpunice = 19
; scan process IO scheduling priority
;scan_ionice = 6
; move malicious file to quarantine; enabled|disabled
quarantine = 'enabled'
; replace malware to honeypot; enabled|disabled
honeypotify = 'enabled'
; if replace is enabled, this file will be copied instead of malware
;honeypot_file = '/opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php'

;
; File whitelist settings
;
[whitelist]
; do not scan files match these extensions
;extensions[] =
; do not scan files match these path definition expressions
;paths[] = '/var/lib/bitninja'
;paths[] = '/var/log/bitninja'
;paths[] = '/var/cache/awstats'
;paths[] = '/opt/bitninja'
;paths[] = '/bin'
;paths[] = '/sbin'
;paths[] = '/lib'
;paths[] = '/lib32'
;paths[] = '/usr/bin'
;paths[] = '/usr/sbin'
;paths[] = '/usr/lib'
;paths[] = '/usr/lib32'
;paths[] = '/home/accesslog'
;paths[] = '/home/virtfs'
;paths[] = '/home/cagefs-skeleton'
;paths[] = '/usr/share/cagefs-skeleton'
;paths[] = '/home/cpeasyapache/src'
;paths[] = '/proc'
;paths[] = '/dev'
;paths[] = '/sys'
; do not scan files match these types based on file command
;types[] =

;
; Inotify settings
;
[inotify]
; ignore files match these path definition expressions ; e.g: .log
;exclude[] = '^/var/tmp/mysql.sock$'
;exclude[] = '^/tmp/mysql.sock$'
;exclude[] = '^/var/cache/buagent/md0.cache.data$'
;exclude[] = '^/var/tmp/#sql_.*\.MYD$'
;exclude[] = '^/tmp/#sql_.*\.MYD$'
;exclude[] = '^/tmp/#sql_.*\.MYI$'
;exclude[] = '^/tmp/#sql_.*\.MAD$'
;exclude[] = '^/tmp/#sql_.*\.MAI$'
;exclude[] = '^/tmp/lshttpd/*\.sock*'
;exclude[] = '^/tmp/lshttpd/\.rtreport\.*'
;exclude[] = '^/var/tmp/clamav-.*'
;exclude[] = '^/tmp/clamav-.*'
;exclude[] = '^/var/lib/bitninja'
;exclude[] = '^/var/log/bitninja'
;exclude[] = '^/var/cache/awstats'
;exclude[] = '^/usr/local/maldetect/quarantine'
;exclude[] = '\.sock$'
;exclude[] = '\.log$'
;exclude[] = '^.*_log$'
;exclude[] = '^.*_log\.processed$'
;exclude[] = '^.*_ssl_log\.webstat$'
;exclude[] = '^/home/accesslog'
;exclude[] = '^/home/virtfs'
;exclude[] = '^/home/cagefs-skeleton/'
;exclude[] = '^/usr/share/cagefs-skeleton/'
;exclude[] = '\.sock$'
;exclude[] = '^/home/.*?/mail/'
;exclude[] = '^/home/cpeasyapache/src/'
;exclude[] = '^/var/tmp/clamav-.*'
;exclude[] = '^/var/lib/bitninja/quarantine$'

; inotify monitoring array
; This is were you can define what to be monitored by the
; malware detection module.
;file_path[] = '/tmp'
;file_path[] = '/home'
;file_path[] = '/var/www'

Quarantine behaviour

The quarantine behaviour depends on a setting in the module’s config.ini file. If the ‘quarantine’ option is enabled, BitNinja moves the infected file to quarantine.

You can find the quarantine directory here: /var/lib/bitninja/quarantine so you can restore any files if you need. This module is also integrated with another BitNinja module, the WebHoneypot module (you can read more about the web honeypot module here: Web honeypot). When a botnet attacks your system it will try to infect your files. If you only quarantine the file, the botnet will try to upload other malware and other variations of the infection and will eventually succeed and infect your system. If the ‘honeypotify’ config option is enabled, the module will automatically copy a honeypot file in place of the original malware. This way when the botnet comes back to check the operation of the malware infection, the honeypot will collect the IP address of the attacking bot and add it to the greylist, immediately protecting all your servers against this attacker.

Example quarantine directory structure:

├── var
│   ├── lib
│   │   ├── quarantine
│   │   │    ├── 2016
│   │   │    │    ├── 12
│   │   │    │    │    ├── 10
│   │   │    │    │    ├── 17
│   │   │    ├── 2017
│   │   │    │    ├── 01
│   │   │    │    ├── 02
│   │   │    │    │    ├── 01
│   │   │    │    │    ├── 05
│   │   │    │    │    ├── 10
│   │   │    │    │    ├── 17
│   │   │    │    ├── 03
│   │   │    │    │    ├── 12
│   │   │    │    │    ├── 13

Example quarantinized malware information…

7650_jquery94.php 7650_jquery94.php.info

… where file with .info extension contains information about the malicious file (original owner, group, file permissions, file path, malware name, file size, create time), while the other is the malicious file itself. Both files are prefixed with a unique identifier.

Disabling the quarantine functionality

If you’d like to disable the quarantine behavior of the Malware Detection module, you have to change a setting in the module’s config.ini file:

  1. Locate the config file at /etc/bitninja/MalwareDetection/config.ini
  2. Look for the Core settings section
  3. Insert this line after the last [core] setting (e.g. the honeypot_file setting)
quarantine = 'disabled'

The example config.ini file’s core section should look something like this:

; Core settings
;
[core]
; do not check files over scan_max_file_size bytes
;scan_max_file_size = 786432 ; 768 KB 1048576 ; 1 MB
; scan process CPU scheduling priority
;scan_cpunice = 19
; scan process IO scheduling priority
;scan_ionice = 6
; replace malware to honeypot; enabled|disabled
honeypotify = 'enabled'
; if replace is enabled, this file will be copied instead of malware
;honeypot_file = '/opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php'
quarantine = 'disabled'

Whenever you’d like to enable the quarantine function again, you should simply change the quarantine setting to ‘enabled’.

Note

If you edit the content of the /etc/bitninja/MalwareDetection/config.ini file, you have to reload the Malware Detection config for the changes to take effect. You can do this by using the following command:

bitninjacli --module=MalwareDetection --reload

Scan path

The most important config change you will probably do is defining what directories the module should monitor for file changes. The defaults are the /tmp, /home and /var/www directories, but you can add any other directories by defining new paths. For example:

file_path[] = '/var'

Exclude patterns

You can define patterns to be excluded from being scanned. This is useful to exclude files like log files.

exclude[] = '/var/www'

Don’t forget to reload the configuration of Malware Detection after altering it for the settings to take effect.

Module initialization

When the Malware Detection module starts at the first time (or any time the inotifywait process stops or is killed) the system have to reinitialize the inotify process. This is a very CPU and disc intensive process and can take hours to finish. (It should traverse all directories and set a flag on them to be monitored. Sorry, this is how the Linux kernel handles the inotify functionality.)

You can find the module logs at /var/log/bitninja/mod.malware_detection.log

Note

We’d like to help you by sharing our own experiences regarding the module’s resource usage. We have 30+ production servers at our shared hosting company right now. In one of the bigger ones, the number of files is 11 714 930. In this case the initial step takes about 5 hours and it doesn’t cause any load issues after that. The average load on this server is 3.

Manual scanning

BitNinja is continuously listening for file changes, but you are also able to run manual scanning with the CLI tool. E.g. in order to look after malware in the /var/www folder, use this command:

bitninjacli --module=MalwareScanner --scan=/var/www

If you’d like to cancel a malware scan that’s already been started, you can do it using the following command:

bitninjacli --module=MalwareScanner --cancel

Manual scanning works fine without installing inotify-tools.

Restoring files from quarantine

You can manually restore files from the quarantine using the following command:

bitninjacli --restore=/path/to/file

Post-detection, post-quarantine and post-honeypotify scripts

You can implement your own business logic to run after each detected malware. You should write your own scripts in an .sh file and place it in the /etc/bitninja/MalwareDetection/detection.conf.d folder.

For post-detection scripts, name your file using the pd_ prefix, e.g. like this: pd_my_script.sh. These scripts will run immediately after the module detects an infected file.

For post-quarantine scripts, the prefix should be pq_, e.g. pq_my_script.sh. These scripts will be executed after the module moves the malware file to quarantine.

For post-honeypotify scripts, the prefix should be ph_, e.g. ph_my_script.sh. Scripts that start with ph_ will run after the module replaces the infected file with a honeypot file.

Note

For the post-quarantine scripts to run, the quarantine function must be enabled in the module’s config.ini file. For the post-honeypotify scripts to work, the honeypotify function must be enabled in /etc/bitninja/MalwareDetection/config.ini.

Example for a post-detection script: pd_example.sh

#!/bin/bash

# This post detection script just echoes out the path to the detected malware

echo "Malware found at path [$1] Malware name is [$2]" > /path_you_wish_to_save_outcome/malware.txt

Optional: manual installation of inotify-tools

Install inotify-tools on .DEB based systems

apt-get update && apt-get install inotify-tools

Install inotify-tools on .RPM based systems

yum --enablerepo=epel install inotify-tools

Note

On some systems the inotify-tools package is not available by default, so you have to enable the EPEL repository.

Package inotify-tools has to be on the latest version (v3.14).

This will install the inotify-tools package on your server so the module can use it.