I have came across the situation where multiple kblockd process are utilizing the CPU heavily and causing server load high in Linux servers. I was wondered what is kblockd and why it is taking high CPU utilization. But there is not much information available in internet about kblockd. After lot of research, the below is what I learned.
What is kblockd?
In a general, the kblockd kernel threads are responsible for performing low-level disk operations.
Why kblockd processes are heavily utilizing the CPU which causing server load?
A high utilization of these could indicate that the server IO queue is backed up and the server is not managing to perform its disk writes quick enough. Most of the times, At that point, the SD drivers fails the IO and fails it to EXT3, which then aborts the journal for safety reasons. So when it is middle of transaction, it forgets about it and retrying to rollback. Also the kblockd message is a symptom of "server running low on memory and starting to fail normal kernel memory allocations".
Most of the times this is a sign of pathological behavior by the kernel or merely a symptom of an overloaded server, depends on the workload of the server and its hardware. Please note that there is always a potential for hangs when something can't allocate memory.
What can be done to resolve this issue?
For now, I don't find any immediate resolution to fix this issue. However keeping your kernel version and block device driver modules up to date might help fixing this issue. Also Upgrading the Linux server to the latest available service pack level is recommended. These might fix any known bugs in earlier versions.
What is kblockd?
In a general, the kblockd kernel threads are responsible for performing low-level disk operations.
Why kblockd processes are heavily utilizing the CPU which causing server load?
A high utilization of these could indicate that the server IO queue is backed up and the server is not managing to perform its disk writes quick enough. Most of the times, At that point, the SD drivers fails the IO and fails it to EXT3, which then aborts the journal for safety reasons. So when it is middle of transaction, it forgets about it and retrying to rollback. Also the kblockd message is a symptom of "server running low on memory and starting to fail normal kernel memory allocations".
Most of the times this is a sign of pathological behavior by the kernel or merely a symptom of an overloaded server, depends on the workload of the server and its hardware. Please note that there is always a potential for hangs when something can't allocate memory.
What can be done to resolve this issue?
For now, I don't find any immediate resolution to fix this issue. However keeping your kernel version and block device driver modules up to date might help fixing this issue. Also Upgrading the Linux server to the latest available service pack level is recommended. These might fix any known bugs in earlier versions.
5 comments
Write commentsThankyou for good information.
ReplyThanks for the Info.. What about kjournald? It also consumes high cpu, temporarily slows the system..
Replythanks, you are great.
ReplyMaybe I have solved the high-utilisation-problem with kblockd, which I also had in the past on my (rooted) android-phone:
ReplyI saw that the kblockd-process had a priority (nice) of -20 (minimum)
With an app I was able to switch it to 10 and within seconds kblockd was ready and went into cached mode.
I don't know whether it works allover, but for me it worked without booting :-)
greetings from cologne
When I have seen this kind of issue, it was due to not enough throughput from the file system.
ReplyI had some jenkins jobs competing for reads/writes on the same file system, and caused kblockd to spike.
What do you think about this Article? Add your Opinion..! EmoticonEmoticon