[Hidden Linux Threats Part 1] How Syslogk Hides Itself

Syslogk, a Rootkit Hidden in the Linux Kernel

Techniques that modify the Linux kernel to conceal malware and signs of compromise have long been used. Syslogk is a rootkit that uses this technique to hide malicious files, processes, TCP communications, files, and directories. It also conceals the fact that it operates as a Linux Kernel Module (LKM).

Linux system administration tools do not inspect kernel internals directly when retrieving system information. When a user runs a command, the tool receives information provided by the kernel and displays the results. For example, ps, top, and pstree use process information from /proc to show running processes, while netstat displays network connection status based on TCP socket information. The ls command reads directory entries to show files and directories, and lsmod displays information about currently active modules.

Syslogk intervenes in this information return process. It intercepts the execution flow of legitimate functions or callback functions, removes only the items that meet threat actor defined conditions, and returns the remaining information normally. This hides its presence and related traces. It uses inline hooking to conceal process and TCP communication information, while VFS (Virtual File System) table hooking prevents specific files and directories from appearing in listings. It also removes its own information from the kernel module list to hide the presence of the rootkit itself. As a result, query commands continue to work normally and display other information as expected, making it difficult for users to recognize that some items have been intentionally hidden.

Hiding Processes and TCP Connections with Inline Hooking

Inline hooking modifies the beginning of a function so that a threat actor defined function runs before the original code when the function is called. This allows a threat actor to intervene in the legitimate function's processing, remove specific information, and then call the original function so the remaining operations continue normally.

To apply inline hooking, Syslogk first identifies the address of the kernel function it will modify. It uses the kernel symbol table /proc/kallsyms to locate the target function in memory. It then stores the identified address in the hookFuncAddress field of a predefined structure.

To modify the memory containing the target kernel function, Syslogk changes the Write Protect bit in the CR0 register to 0, disabling write protection, and grants write access in the PTE (Page Table Entry). This makes it possible to modify the data in the page.

Once inline hooking is applied, the function specified by Syslogk runs first whenever the target function is called. During this process, only the information the threat actor wants to hide can be removed from the output. Syslogk uses this method to hook proc_root_readdir to hide specific processes and tcp4_seq_show to remove specific TCP socket information from the query results.

Process Concealment Through proc_root_readdir Hooking

The proc_root_readdir function examines process entries in the /proc directory and calls a pre-registered callback function to process each entry. The callback adds the entry it receives to the process list. Utilities such as ps, top, and pstree use the information provided through this process to display the list of running processes.

Syslogk hooks proc_root_readdir to intervene in this process. It first saves the address of the existing legitimate callback function in bk_proc_filldir and then replaces it with the threat actor defined nw_proc_filldir. Once the callback is replaced, even when the original proc_root_readdir is executed, process entries read from /proc go through nw_proc_filldir instead of being passed directly to the original callback.

nw_proc_filldir examines each process entry it receives and determines whether to conceal it. If the process name matches was_sys_relay, it does not pass the entry to the original legitimate callback, preventing the process from appearing in the process list.

Syslogk hides not only the was_sys_relay process itself but also processes that have was_sys_relay as a parent or grandparent. It does this by checking the PID and parent process ID. Entries with a PID below 2 are not subject to exclusion. Processes that do not meet the concealment conditions also appear normally through the original legitimate callback.

As a result, was_sys_relay and its related processes continue to run, but their entries are excluded when process information is retrieved from /proc. They therefore do not appear in process lists displayed by commands such as ps, top, and pstree.

TCP Socket Concealment Through tcp4_seq_show Hooking

Syslogk conceals TCP communication information as well as processes. To do so, it hooks tcp4_seq_show, the function that outputs TCP socket information from /proc/net/tcp. tcp4_seq_show is called each time an individual socket is processed and writes the output information about that socket to the seq_file. Utilities such as netstat use this information to display TCP connection lists.

When the hooked tcp4_seq_show is called, Syslogk first runs the original tcp4_seq_show so the socket information is written normally to seq_file. It then converts the port number it wants to hide into a string and checks whether the recorded socket information contains that string. If the condition is met, it removes the section containing that socket information from the output buffer so the socket does not appear in the TCP connection list.

The actual TCP connection does not disappear. Only its information is removed from the query results. The socket remains open and communication continues, but the entry does not appear in standard TCP query results. Because only information that meets the specified port condition is excluded, other connections are displayed normally.

File and Directory Concealment Through VFS Table Hooking

Syslogk uses VFS (Virtual File System) table hooking to hide files and directories. VFS is a layer that enables Linux to handle different file systems through a common interface. It stores pointers to functions responsible for various file system operations.

Among the operations in the VFS table, Syslogk hooks readdir. It obtains the file operations structure for the target path, saves the existing readdir pointer, and replaces it with the address of a threat actor defined function. When the directory contents are queried, the threat actor defined function runs before the legitimate readdir function.

The modified readdir follows a processing flow similar to the one used to conceal processes. The defined function backs up the address of the legitimate filldir callback and passes its own nw_root_filldir callback when calling the original readdir. The original function reads the directory contents, but each file and directory entry passes through nw_root_filldir, which determines whether it will be included in the output.

nw_root_filldir checks whether a file or directory name contains the string was-patch. If the name meets this condition, the function does not pass the entry to the legitimate callback, preventing it from being added to the listing. Entries that do not contain was-patch are passed to the legitimate callback and displayed normally.

Therefore, when an administrator uses ls to inspect a directory, an item whose name contains was-patch does not appear in the listing even if it is present. However, Syslogk does not delete the file or block direct access to its path. If the administrator knows the exact path of the hidden directory, it can be accessed directly with cd, and the files within it can also be viewed.

Syslogk does not remove the files or directories themselves. It excludes only specific entries from the list returned by readdir. Other entries appear normally and the command runs without errors, making it difficult for users to recognize that some files or directories have been concealed.

Self-Concealment Through Kernel Module Concealment

Syslogk operates in the Linux kernel as an LKM (Loadable Kernel Module). An LKM is dynamically loaded into a running Linux kernel to extend kernel functionality and is also used for legitimate functions such as device drivers. While operating as an LKM, Syslogk hides processes, TCP communications, and files, it also conceals the fact that it is running as a kernel module.

The Linux kernel manages loaded modules in a doubly linked list. Each kernel module node is linked to the previous and next nodes in the list, and the lsmod command uses this list to display the kernel modules currently running. Syslogk removes its own node from the module list so it does not appear in lsmod results. Before removing the node, Syslogk checks whether mod_list already contains the location of its module node to determine whether the module is currently hidden.

If the module is not already hidden, Syslogk first saves the location of its module node in mod_list, then removes the node from the doubly linked list of loaded kernel modules. Removing it from the list does not stop the module, allowing Syslogk continues to operate in the kernel while remaining absent from lsmod results.

Syslogk hides its information not only from lsmod but also from /sys/module in sysfs. Linux registers kernel module information in sysfs as a kobject. Syslogk removes its own kobject so the module is not visible under /sys/module. During this process, state_in_sysfs is used as a state value that indicates whether the object is registered in sysfs. Removing the kobject changes this value to 0, but Syslogk resets it to 1 to maintain the appearance that the object remains registered.

These changes hide Syslogk presence from the kernel module information available to administrators allowing it to operate while undetected.

Countermeasures for hidden rootkits

Rootkits such as Syslogk can intervene in the operating system's information retrieval process and hide specific process, file, and communication information from the results. This makes it difficult to identify concealed processes, files, and communications by using standard Linux commands such as ps, ls, and netstat alone.

When a rootkit infection is suspected, the first step is to check for hidden kernel modules. Once a hidden module is detected and made visible again, previously hidden processes, files, and other system information can also be identified.

The identified processes and files should then be examined to determine whether they are malicious, and any kernel modules or malicious files associated with the rootkit should be removed. Responding to this type of rootkit requires more than checking what is already visible. Hidden processes, files, and other information must first be uncovered so they can then be analyzed and remediated.