I have came across the situation where multiple kblockd process are utilizing the CPU heavily and causing server load high in Linux servers. I was wondered what is kblockd and why it is taking high CPU utilization. But there is not much information available in internet about kblockd. After lot of research, the below is what I learned.
What is kblockd?
In a general, the kblockd kernel threads are responsible for performing low-level disk operations.
Why kblockd processes are heavily utilizing the CPU which causing server load?
A high utilization of these could indicate that the server IO queue is backed up and the server is not managing to perform its disk writes quick enough. Most of the times, At that point, the SD drivers fails the IO and fails it to EXT3, which then aborts the journal for safety reasons. So when it is middle of transaction, it forgets about it and retrying to rollback. Also the kblockd message is a symptom of "server running low on memory and starting to fail normal kernel memory allocations".
Most of the times this is a sign of pathological behavior by the kernel or merely a symptom of an overloaded server, depends on the workload of the server and its hardware. Please note that there is always a potential for hangs when something can't allocate memory.
What can be done to resolve this issue?
For now, I don't find any immediate resolution to fix this issue. However keeping your kernel version and block device driver modules up to date might help fixing this issue. Also Upgrading the Linux server to the latest available service pack level is recommended. These might fix any known bugs in earlier versions.
What is kblockd?
In a general, the kblockd kernel threads are responsible for performing low-level disk operations.
Why kblockd processes are heavily utilizing the CPU which causing server load?
A high utilization of these could indicate that the server IO queue is backed up and the server is not managing to perform its disk writes quick enough. Most of the times, At that point, the SD drivers fails the IO and fails it to EXT3, which then aborts the journal for safety reasons. So when it is middle of transaction, it forgets about it and retrying to rollback. Also the kblockd message is a symptom of "server running low on memory and starting to fail normal kernel memory allocations".
Most of the times this is a sign of pathological behavior by the kernel or merely a symptom of an overloaded server, depends on the workload of the server and its hardware. Please note that there is always a potential for hangs when something can't allocate memory.
What can be done to resolve this issue?
For now, I don't find any immediate resolution to fix this issue. However keeping your kernel version and block device driver modules up to date might help fixing this issue. Also Upgrading the Linux server to the latest available service pack level is recommended. These might fix any known bugs in earlier versions.