insmod ERROR - could not insert module - Operation not permitted

Symptom

This problem is usually realized after a failed replication. The Server Backup Manager may contain the following error in the task logs.

 An exception occurred during the request. The replication driver was not detected
detail: An error occurred while loading the hcpdriver module, please check your system logs;
Please run "r1soft-setup --get-module " on your Agent to install one. ).   

Attempts to load the module manually result in :

 ERROR: could not insert module /lib/modules/r1soft/hcpdriver-cki-XXX.ko: Operation not permitted 

The following error can also be found in Syslog / Messages log :

 kernel: hcp: ERROR: Could not init mrf hash! 

And lastly, the following error is shown in the dmesg log after a manual attempt to load the module :

 insmod : page allocation failure: order:5, mode:0x40d0 

Cause

In this case, memory fragmentation can be responsible for our memory allocation request being denied by the Linux Kernel. There are not enough contiguous pages to allocate. This can happen on servers with longer uptimes, usually with high or varying usage by database engines and applications.

Resolution

The recommended solution is to reboot the system, which should allow the module to load before the memory becomes fragmented. NOTE: This behavior can return after a reboot if the memory becomes fragmented to a point where our memory allocation request is denied

Additional Info:
If rebooting the system is absolutely out of the question, it is also possible to work around this issue by compacting the memory and clearing the caches. This is only intended to be used as a last resort, as it can affect performance temporarily as resources are reloaded into the cache, and can cause high disk I/O if a high number of users or processes are suddenly accessing the disk while the cache is re-populated.

Compact memory with:

 # sync; echo 1 > /proc/sys/vm/compact_memory 

Clear the disk-cache with:

 # sync; echo 1 > /proc/sys/vm/drop_caches 

Where:

echo 1: Clears PageCache only
echo 2: Clears dentries and inodes
echo 3: Clears PageCache, dentries and inodes.

Start with level 1, as this option has the least impact, and only move up to higher levels if necessary.
WARNING : The 3rd option is not recommended for use in production. Please use caution when selection this option.

Lastly, restart the agent:

 # service cdp-agent restart 