The Linux kernel provides variety of locking mechanisms. Each one of them functions differently and are meant to be used in different use cases. There could be situations where threads can sleep after acquiring the lock but in some situations, threads cannot be allowed to sleep after acquiring the lock. Due to the different locking requirements, programmer will have to decide what kind of locking primitive should be used among options like spinlock, mutex, semaphore etc. No matter which locking primitive is being used, due to increasing complexity in the Linux kernel, possibility of hitting locking errors in the kernel space have always been increasing. In this blog we will discuss what kind of locking issues can happen in the Linux kernel and how do we recover from those.

Kernel locking errors –

One of the prominent locking issue is deadlock caused by weakness in the coding done in the kernel space. There could be situations when one or more threads of execution acquires the lock but doesn’t release it for other waiting threads for certain amount of time or indefinitely. Different locking errors shows up as different symptoms. E.g., mutex deadlock can cause kernel hung task, spinlock locking issues can cause CPU core being locked up in the form of soft lockup, locking errors in RCU critical sections can cause rcu CPU stall.

Typical hung task logs caused by mutex deadlock

[ 242.654560] INFO: task kworker/0:1:35 blocked for more than 120 seconds.

[ 242.654586] Tainted: G C 5.10.46-1-MANJARO-ARM #1 [ 242.654594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[ 242.654605] task:kworker/0:1 state:D stack: 0 pid: 35 ppid:

2 flags:0x00000028

[ 242.654803] Workqueue: events_power_efficient reg_check_chans_work

[cfg80211]

[ 242.654823] Call trace:

[ 242.654842] __switch_to+0x114/0x170

[ 242.654857] __schedule+0x320/0x900

[ 242.654868] schedule+0x50/0x10c

[ 242.654881] schedule_preempt_disabled+0x24/0x3c

[ 242.654891] __mutex_lock.constprop.0+0x184/0x510

[ 242.654605] task:kworker/0:1 state:D stack: 0 pid: 35 ppid:

[ 242.654911] mutex_lock+0x54/0x60

[ 242.654922] rtnl_lock+0x24/0x30

[ 242.655038] reg_check_chans_work+0x34/0x410 [cfg80211]

[ 242.655050] process_one_work+0x1dc/0x4bc

[ 242.655069] worker_thread+0x148/0x47c

[ 242.655079] kthread+0x14c/0x160

[ 242.655091] ret_from_fork+0x10/0x38

Soft lockup error would appear like this –
[ 56376.979162] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s!
[ptlrpcd_00_00:12056]

RCU stall errors would appear like this –
[32801109.627094] INFO: rcu_preempt self-detected stall on CPU { 0} (t=60000
jiffies g=3805569957 c=3805569956 q=39067)
[32801109.627096] Task dump for CPU 0:

[32801109.627099] XXXXXXXXXXXXXXXX R running task 0 260624 260617

0x00080088
[32801109.627100] Call Trace

Additionally, there could be scenarios when one high priority kernel thread is not indefinitely holding up a locking resource but holds it too often. This could potentially starve a low priority thread when numbers of waiters are more than one.

What is LOCKDEP?
Lockdep detects and predicts potential deadlocks caused by wrong use of locking primitives in the Linux kernel. Mutex API misuse, RCU primitive misuse and validity of sleepability assumed by each lock when acquiring lock is detected by lockdep.
How to enable LOCKDEP
Lockdep can be enabled by kernel config CONFIG_DEBUG_LOCK_ALLOC which in turn enables kernel configs CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, CONFIG_LOCKDEP. So, we would end up enabling –
How does LOCKDEP help –
Scenario 1 – Simple deadlock example
spin_lock(&lockA); spin_lock(&lockB); spin_unlock(&lockB); spin_unlock(&lockA); spin_lock(&lockC); spin_lock(&lockA);
spin_unlock(&lockA); spin_unlock(&lockC); spin_lock(&lockB); spin_lock(&lockC); spin_unlock(&lockC); spin_unlock(&lockB);
LOCKDEP output
[ INFO: possible circular locking dependency detected ]
3.1.0-rc4-test-00131-g9e79e3e #2 -------------------------------------------------------
insmod/1357 is trying to acquire lock:
(lockC){+.+...}, at: [ ] pick_test+0x2a2/0x892 [lockdep_test] but task is already holding lock:
(lockB){+.+...}, at: [ ] pick_test+0x296/0x892 [lockdep_test] which lock already depends on the new lock.

Scenario 2 – Detecting lock inversion
[ INFO: possible irq lock inversion dependency detected ]
3.1.0-rc4-test-00131-g9e79e3e #2
---------------------------------------------------------
sshd/1327 just changed the state of lock:
(lockA){-.....}, at: [ ] probe_irq_entry+0x1a/0x6c [lockdep_test] but this lock took another, HARDIRQ-unsafe lock in the past:
(lockC){+.+...}
and interrupts could create inverse lock ordering between them.

There are many more use cases where LOCKDEP is highly useful for detecting locking issues e.g missing unlock calls in the paths, circular locks, uninitialized lock errors.