MalwareDetection
BitNinja has an excellent module for file-based malware detection. If attackers can break through the defense line of honeypots and the WAF, Malware Detection is the next line to detect file changes and uploads and check them against our database. It is also advantageous to scan your server if you have just started to use BitNinja for the first time, as there could be different malware and backdoor programs on your server. Also, our malware database is ever-growing, so it is recommended to schedule malware scans. Cleaning your files is vital.
We ship new malware definition files automatically so it is always up to date without any manual intervention.
Prerequirements
The module uses the inotify-tools package to monitor the filesystem for
changes. You need this package to be available before you can use the module.
We ship our own inotify-tools package with BitNinja to ensure the installation.
If you need to install inotify-tools manually for some reason, please follow the steps stated here:
Post-detection, post-quarantine and post-honeypotify scripts.
Before you enable the Malware Detection module, there are some steps you have to follow on your server to prepare it for the launch of the module.
On most servers, default number of user watches have to be expanded from the default to a much higher number. BitNinja Linux Client increases it automatically up to 30 000 000.
If it is not large enough, you can increase it manually:
echo 35000000 > /proc/sys/fs/inotify/max_user_watches
Now check if everything is ready to start the malware detection module:
cat /proc/sys/fs/inotify/max_user_watches
// This should be the result:
// 30000000
whereis inotifywait
// Something similar should be the result
// inotifywait: /usr/bin/inotifywait /usr/bin/X11/inotifywait /usr/share/man/man1/inotifywait.1.gz
// check if inotify-tools is on the latest version
// on deb-based systems
dpkg-query --showformat='${Version}' --show inotify-tools
// on rpm-based systems
rpm -q --qf "%{VERSION}\n" inotify-tools
// result: 3.14-2
You are ready to start the malware detection module now! Before lauching the module please read through the configuration section to prepare the module.
Configuration
To change the default configurations of the malware detection module you
have to edit the module config file at /etc/bitninja/MalwareDetection/config.ini.
We always try to set up reasonable defaults for our config files to provide an out-of-the-box experience,
but, in this case, there are some configurations you are probably willing to do before starting the module.
Disabling the MalwareDetection module from the dashboard will only disable the real-time malware detection. It will NOT disable the scheduled malware scans. A malware scan be still initiated via cron or via the BitNinja CLI as well.
Example content for the config.ini file:
;
; Core settings
;
[core]
; do not check files over scan_max_file_size bytes
;scan_max_file_size = 786432 ; 768 KB 1048576 ; 1 MB
; scan process CPU scheduling priority
;scan_cpunice = 19
; scan process IO scheduling priority
;scan_ionice = 6
; move malicious file to quarantine; enabled|disabled
;quarantine = 'enabled'
; replace malware to honeypot; enabled|disabled
;honeypotify = 'disabled'
; if replace is enabled, this file will be copied insted of malware
;honeypot_file = '/opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php'
; memory limit in MB
;memory_limit = 600
; Change where to store quarantined files
;quarantine_directory= '/var/lib/bitninja/quarantine'
; Enable malware source upload to Cloud.
; This feature helps you with the malware validation process.
; Uploaded source files only visable for the server owner.
;upload_malware_source=0
; How many scans can be run at once
;maxConcurrentScans=10
;
; File allow list settings
;
; Note: Do not scan these files even, if the file marked as scanable by someone.
;
[whitelist]
; do not scan files match these extensions
;extensions[] =
; do not scan files match these path definition expressions
; Glob pattern can be used Eg.: '/home*/virtfs'
;
;paths[] = '/var/lib/bitninja'
;paths[] = '/var/log/bitninja'
;paths[] = '/var/cache/awstats'
;paths[] = '/opt/bitninja'
;paths[] = '/bin'
;paths[] = '/sbin'
;paths[] = '/lib'
;paths[] = '/lib32'
;paths[] = '/usr/bin'
;paths[] = '/usr/sbin'
;paths[] = '/usr/lib'
;paths[] = '/usr/lib32'
;paths[] = '/home*/accesslog/*'
;paths[] = '/home*/virtfs/*'
;paths[] = '/home*/cagefs-skeleton/*'
;paths[] = '/usr/share/cagefs-skeleton'
;paths[] = '/home*/cpeasyapache/src/*'
;paths[] = '/proc'
;paths[] = '/dev'
;paths[] = '/sys'
; do not scan files match these types based on file command
;types[] =
;
; Inotify settings
;
[inotify]
; ignore files match these path definition expresssions ; e.g: .log
; Glob and regex pattern can be used as a value
;exclude[] = '^/var/tmp/mysql.sock$'
;exclude[] = '^/tmp/mysql.sock$'
;exclude[] = '^/var/cache/buagent/md0.cache.data$'
;exclude[] = '^/var/tmp/#sql_.*\.MYD$'
;exclude[] = '^/tmp/#sql_.*\.MYD$'
;exclude[] = '^/tmp/#sql_.*\.MYI$'
;exclude[] = '^/tmp/#sql_.*\.MAD$'
;exclude[] = '^/tmp/#sql_.*\.MAI$'
;exclude[] = '^/tmp/lshttpd/*\.sock*'
;exclude[] = '^/tmp/lshttpd/\.rtreport\.*'
;exclude[] = '^/var/tmp/clamav-.*'
;exclude[] = '^/tmp/clamav-.*'
;exclude[] = '^/var/lib/bitninja'
;exclude[] = '^/var/log/bitninja'
;exclude[] = '^/var/cache/awstats'
;exclude[] = '^/usr/local/maldetect/quarantine'
;exclude[] = '\.sock$'
;exclude[] = '\.log$'
;exclude[] = '^.*_log$'
;exclude[] = '^.*_log\.processed$'
;exclude[] = '^.*_ssl_log\.webstat$'
;exclude[] = '^/home\d+/accesslog'
;exclude[] = '^/home\d+/virtfs'
;exclude[] = '^/home\d+/cagefs-skeleton/'
;exclude[] = '^/usr/share/cagefs-skeleton/'
;exclude[] = '\.sock$'
;exclude[] = '^/home\d+/.*?/mail/'
;exclude[] = '^/home\d+/cpeasyapache/src/'
; inotify monitoring array
; this is where you can define what to be monitored by the MalwareDetection module
; Glob pattern can be used as a value eg.: /home*/
;
;file_path[] = '/tmp'
;file_path[] = '/home*/'
;file_path[] = '/var/www'
[FileSystemMonitor]
;monitor_type = 'inotify'
;monitor_type = 'auditd'
;monitor_order[] = 'auditd'
;monitor_order[] = 'inotify'
;monitor_order[] = 'nullMonitor'
[Detectors]
; disabled[]='Md5WhiteGlobal'
; disabled[]='Md5WhiteUserOld'
; disabled[]='Md5BlackUser'
; disabled[]='Md5BlackGlobal'
; disabled[]='Hex'
; disabled[]='FirstLine'
; disabled[]='FirstLineV2'
; disabled[]='HexaBotnet'
; disabled[]='WPVCD'
; disabled[]='SAWhiteGlobal'
; disabled[]='SABlackUser'
; disabled[]='SABlackGlobal'
; disabled[]='SASnippetUser'
; disabled[]='SASnippetGlobal'
[quick_cache]
;l1_chunks = 30
;l1_chunk_size = 400
;l1_to_l2_threshold = 3
[database]
;database_cache = 1
;build_php_structure_data = 1
;invalidate_results = 1
; Disable database save when free space in /var/lib/bitninja below then 1000 Mb
;disable_database_save_limit = 1000
; This config options disables the active malware scan to save not just php files in the file cache db
; only_save_php = 1
; This config options disables the manual malware scan to save not just php files in the file cache db
; manual_scan_only_save_php = 0
Quarantine behaviour
The quarantine behaviour depends on a setting in the module's config.ini file.
If the 'quarantine' option is enabled, BitNinja moves the infected file to quarantine.
You can find the quarantine directory at
/var/lib/bitninja/quarantine so you can restore any files if you need.
This module is also integrated with another BitNinja module, the
Web Honeypot module. When a botnet attacks your system it will try to infect
your files. If you only quarantine the file, the botnet will try to upload
other malware and other variations of the infection and will eventually
succeed and infect your system.
If the honeypotify config option is
enabled, the module will automatically copy a honeypot file in place
of the original malware. This way when the botnet comes back to check the
operation of the malware infection, the honeypot will collect the IP address
of the attacking bot and add it to the challenge list, immediately protecting
all your servers against this attacker.
Example quarantine directory structure:
├── var
│ ├── lib
│ │ ├── quarantine
│ │ │ ├── 2016
│ │ │ │ ├── 12
│ │ │ │ │ ├── 10
│ │ │ │ │ ├── 17
│ │ │ ├── 2017
│ │ │ │ ├── 01
│ │ │ │ ├── 02
│ │ │ │ │ ├── 01
│ │ │ │ │ ├── 05
│ │ │ │ │ ├── 10
│ │ │ │ │ ├── 17
│ │ │ │ ├── 03
│ │ │ │ │ ├── 12
│ │ │ │ │ ├── 13
Some examples for quarantinized malware information:
7650_jquery94.php, 7650_jquery94.php.info
Where the file with the .info extension contains information about the malicious file
(original owner, group, file permissions, file path, malware name, file size,
create time), while the other is the malicious file itself. Both files are prefixed
with a unique identifier.
MalwareDetection worker caps
In the module's config file you can change the maximum number of MalwareDetection workers.
If the corresponding variable is not present in the /etc/bitninja/MalwareDetection/config.ini file then add it manually under the [core] section in the config file.
Example:
[core]
; If file_path_pull_limit is set to 20 then max number of workers is 80
file_path_pull_limit = 20
The file_path_pull_limit variable's default value is set to 0, which means the maximum number of files processed by the SA and YARA workers each will be the double of the number of CPU cores in the server. The upper limit of the simultaneously processed files by each worker pool is 40. So with the default settings, the total number of MalwareDetection workers is limited to 80, (40 SA and 40 YARA workers) in case of any number of available CPU threads.
The maximum total number of MalwareDetection workers will be four times the 'file_path_pull_limit' variable's vaule.
If the file_path_pull_limit is raised it is highly recommended to also increase the memory limit of the module, because of the increased number of files processed simultaneously.
The memory_limit variable can be modified in the config file under the [core] section.
Throttle file processing
In the Malware Detection module's config file you can throttle the file processing to decrease the scans' impact on the server's load if it causes issues.
Example:
; The higher this value is, the fewer resources scans will consume (while also being slower).
; Ranges from 0 to 1
;scan_niceness = 0.002
The scan niceness is on a 0-1 scale. The closer it is to 1 the less effect it has on the server's load, also making the scans slower.
The default value for the scan_niceness variable is 0.002.
If this variable is not present in your BitNinja agent's config file, then make sure that the version of the agent is 2.38.4 or newer, and then add the lines above manually to the config file.
Filesystem cache settings
The Malware detection module creates a Filesystem cache, to speed up the malware scan process while lowering the consumed server resources.
From the MalwareDetection module's config file the Filesystem cache can be limited and the behavior of the cache can be also changed.
The corresponding variables can be found under the [database] flag in the /etc/bitninja/MalwareDetection/config.ini file.
By default, the Filesystem cache's size is not limited. However, using the filesystem_cache_size_limit variable the limit can be set in megabytes.
By setting the filesystem_cache_threshold_behavior variable to a value you can define what happens when the Filesystem cache reaches the limit.
By default, the filesystem_cache_threshold_behavior is set to readonly which means that the Filesystem cache won't be updated.
With the delete option, the Filesystem cache will be deleted when it reaches the limit.
The filesystem_cache_threshold_behavior variable only takes effect, if there is a size limit set for the Filesystem cache.
; The maximum size of the filesystem cache, in megabytes. Default: 0, which means the cache size is not limited.
;filesystem_cache_size_limit = 0
; What to do if the size of the filesystem cache exceeds the limit. Possible values: 'readonly', 'delete'
;filesystem_cache_threshold_behavior = 'readonly'
Disabling the quarantine functionality
If you'd like to disable the quarantine behavior of the Malware Detection module,
you have to change a setting in the module's config.ini file:
- Locate the config file at
/etc/bitninja/MalwareDetection/config.ini - Look for the Core settings section
- Insert this line after the last
[core]setting (e.g. thehoneypot_filesetting)
quarantine = 'disabled'
The example config.ini file's core section should look something like this:
; Core settings
;
[core]
; do not check files over scan_max_file_size bytes
;scan_max_file_size = 786432 ; 768 KB 1048576 ; 1 MB
; scan process CPU scheduling priority
;scan_cpunice = 19
; scan process IO scheduling priority
;scan_ionice = 6
; replace malware to honeypot; enabled|disabled
honeypotify = 'enabled'
; if replace is enabled, this file will be copied instead of malware
;honeypot_file = '/opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php'
quarantine = 'disabled'
Whenever you'd like to enable the quarantine function again, you should simply change the quarantine setting to enabled.
If you edit the content of the /etc/bitninja/MalwareDetection/config.ini file, you have to reload the Malware Detection config for the changes to take effect. You can do this by using the following command: bitninjacli --module=MalwareDetection --reload
Scan path
The most important config change you will probably do is defining what
directories the module should monitor for file changes. The defaults are
the /tmp, /home and /var/www directories, but you can add any other directories by defining new paths. For example:
file_path[] = '/var'
Exclude patterns
You can define patterns to be excluded from being scanned. This is useful to exclude files like log files.
exclude[] = '/var/www'
Don't forget to reload the configuration of Malware Detection after altering it for the settings to take effect.
Module initialization
When the Malware Detection module starts at the first time (or any time the
inotifywait process stops or is killed) the system have to reinitialize the
inotify process. This is a very CPU and disc intensive process and can take
hours to finish. (It should traverse all directories and set a flag on them
to be monitored. Sorry, this is how the Linux kernel handles the inotify functionality.)
You can find the module logs at /var/log/bitninja/mod.malware_detection.log.
We'd like to help you by sharing our own experiences regarding the module's resource usage. We have 30+ production servers at our shared hosting company right now. In one of the bigger ones, the number of files is 11 714 930. In this case the initial step takes about 5 hours and it doesn't cause any load issues after that. The average load on this server is 3.
Manual scanning
BitNinja is continuously listening for file changes, but you are also able to run
manual scanning with the CLI tool. E.g. in order to look after malware in the
/var/www folder, use this command:
bitninjacli --module=MalwareScanner --scan=/var/www
If you'd like to cancel all malware scan that's already been started, you can do it using the following command:
bitninjacli --module=MalwareScanner --cancel
Manual scanning works fine without installing inotify-tools.
Restoring files from quarantine
You can manually restore files from the quarantine using the following command:
bitninjacli --restore=/path/to/file
Post-detection, post-quarantine and post-honeypotify scripts
You can implement your own business logic to run after each detected malware.
You should write your own scripts in an .sh file and place it in the /etc/bitninja/MalwareDetection/detection.conf.d
folder.
For post-detection scripts, name your file using the pd\_ prefix, like this: pd_my_script.sh.
These scripts will run immediately after the module detects an infected file.
For post-quarantine scripts, the prefix should be pq\_, like this: pq_my_script.sh.
These scripts will be executed after the module moves the malware file to quarantine.
For post-honeypotify scripts, the prefix should be ph\_, like this: ph_my_script.sh.
Scripts that start with ph\_ will run after the module replaces the infected file with a honeypot file.
Note For the post-quarantine scripts to run, the quarantine function must be enabled in the module's config.ini file.
For the post-honeypotify scripts to work, the honeypotify function must be enabled in /etc/bitninja/MalwareDetection/config.ini.
Example for a post-detection script: pd_example.sh
#!/bin/bash
# This post detection script just echoes out the path to the detected malware
echo "Malware found at path [$1] Malware name is [$2]" > /path_you_wish_to_save_outcome/malware.txt
Optional: manual installation of inotify-tools
Installing inotify-tools on .DEB based systems
apt-get update && apt-get install inotify-tools
Installing inotify-tools on .RPM based systems
yum --enablerepo=epel install inotify-tools
On some systems the inotify-tools package is not available by default, so you have to enable the EPEL repository.
Package inotify-tools has to be on the latest version (v3.14).
This will install the inotify-tools package on your server so the module can
use it.