openrisc / linux Goto Github PK
View Code? Open in Web Editor NEWThis project forked from skristiansson/linux
Linux kernel source tree
License: Other
This project forked from skristiansson/linux
Linux kernel source tree
License: Other
While running linux 5.5 in qemu with full network stack connected to NFS and SSH server running.
Trace:
[ 1020.840000] watchdog: BUG: soft lockup - CPU#0 stuck for 187s! [ksoftirqd/0:7]
[ 1020.840000] Modules linked in:
[ 1020.840000] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 5.5.0-simple-smp #91
[ 1020.840000] CPU #: 0
[ 1020.840000] PC: c03aa6e0 SR: 00008a7f SP: c1025c68
[ 1020.840000] GPR00: 00000000 GPR01: c1025c68 GPR02: c1025c8c GPR03: 00000000
[ 1020.840000] GPR04: eb893541 GPR05: 00000000 GPR06: c8000000 GPR07: 14090179
[ 1020.840000] GPR08: 000003ca GPR09: c006c1f4 GPR10: c1024000 GPR11: fffffffc
[ 1020.840000] GPR12: c133e319 GPR13: 00000000 GPR14: 01726000 GPR15: 00000000
[ 1020.840000] GPR16: c0910c50 GPR17: eb893541 GPR18: a0000000 GPR19: 00000001
[ 1020.840000] GPR20: 8f6237da GPR21: 00000003 GPR22: 000000bf GPR23: 0000014c
[ 1020.840000] GPR24: 0000001a GPR25: 00000010 GPR26: 00027f10 GPR27: 0000000e
[ 1020.840000] GPR28: 007c85fd GPR29: 00000012 GPR30: 00000200 GPR31: c1d9e870
[ 1020.840000] RES: fffffffc oGPR11: ffffffff
[ 1020.840000] Process ksoftirqd/0 (pid: 7, stackpage=c101b400)
[ 1020.840000]
[ 1020.840000] Stack:
[ 1020.840000] Call trace:
[ 1020.840000] [<5344f851>] ktime_get+0x70/0x1bc
[ 1020.840000] [<88164d97>] tcp_mstamp_refresh+0x30/0x190
[ 1020.840000] [<497cf51c>] tcp_rcv_established+0x38/0x798
[ 1020.840000] [<1447d5c1>] tcp_v4_do_rcv+0x1e4/0x238
[ 1020.840000] [<21cdc523>] tcp_v4_rcv+0xd1c/0xdd0
[ 1020.840000] [<f65863ef>] ? __inet_lookup_established+0x6c/0x248
[ 1020.840000] [<278b79da>] ip_protocol_deliver_rcu+0x44/0x1e8
[ 1020.840000] [<7bb669e4>] ip_local_deliver+0xa4/0xf8
[ 1020.840000] [<3e703180>] ip_rcv+0x6c/0xa0
[ 1020.840000] [<cf154ebe>] __netif_receive_skb_one_core+0x50/0x68
[ 1020.840000] [<7a0f7052>] __netif_receive_skb+0x38/0x98
[ 1020.840000] [<72f95335>] netif_receive_skb+0x38/0x90
[ 1020.840000] [<9b4bb2ad>] ethoc_poll+0x280/0x900
[ 1020.840000] [<917ada6e>] net_rx_action+0x11c/0x3cc
[ 1020.840000] [<6554ac93>] ? mempool_kfree+0x14/0x24
[ 1020.840000] [<e4702a67>] __do_softirq+0x128/0x320
[ 1020.840000] [<0edefd55>] ? smpboot_thread_fn+0x0/0x26c
[ 1020.840000] [<ddaf11a2>] run_ksoftirqd+0x50/0x70
[ 1020.840000] [<6abd7f59>] smpboot_thread_fn+0x1b8/0x26c
[ 1020.840000] [<e436cf21>] ? __kthread_parkme+0x68/0xa8
[ 1020.840000] [<410cfcab>] kthread+0x120/0x150
[ 1020.840000] [<12695f1f>] ? kthread+0x0/0x150
[ 1020.840000] [<161c3108>] ret_from_fork+0x1c/0x9c
[ 1020.840000]
[ 1020.840000] Code:
When just running linux with every once in a while we get complete lockups with no errors.
Running in qemu in the monitor I can see:
c00075e0
See dumps below.
Note, the LR points to :
c0007550 <_external_irq_handler>:
c0007550: d4 01 10 08 l.sw 8(r1),r2
c0007554: d4 01 18 0c l.sw 12(r1),r3
c0007558: d4 01 28 14 l.sw 20(r1),r5
...
c00075d8: 48 00 40 00 l.jalr r8
c00075dc: 15 00 00 00 l.nop 0x0
c00075e0: 00 00 03 5a l.j c0008348 <_ret_from_exception>
c00075e4: 15 00 00 00 l.nop 0x0
c039bfd8 <do_IRQ>:
c039bfd8: 9c 21 ff f8 l.addi r1,r1,-8
c039bfdc: d4 01 10 00 l.sw 0(r1),r2
c039bfe0: d4 01 48 04 l.sw 4(r1),r9
c039bfe4: 1a 20 c0 8f l.movhi r17,0xc08f
c039bfe8: 86 31 cb fc l.lwz r17,-13316(r17)
c039bfec: 48 00 88 00 l.jalr r17
c039bff0: 9c 41 00 08 l.addi r2,r1,8
c039bff4: 85 21 00 04 l.lwz r9,4(r1)
c039bff8: 84 41 00 00 l.lwz r2,0(r1)
c039bffc: 44 00 48 00 l.jr r9
c039c000: 9c 21 00 08 l.addi r1,r1,8
c039c004 <__irqentry_text_end>:
c039c004: 00 00 00 00 l.j c039c004 <__irqentry_text_end>
PC=c039c004
R00=00000000 R01=34d08aec R02=c15b1804 R03=00000000
R04=ffffffff R05=00000000 R06=00000012 R07=0000000e
R08=0000001a R09=c00075e0 R10=c15b0000 R11=00000000
R12=0000001b R13=00000010 R14=c1076840 R15=00000005
R16=c1076840 R17=00000000 R18=c1f69758 R19=00000000
R20=00ca4800 R21=00000000 R22=00000428 R23=00000003
R24=01511e00 R25=ce63f7b2 R26=0000042c R27=22bdcb7b
R28=00000005 R29=3416c000 R30=ffffffff R31=c62b0000
A second later.
(qemu) info registers
PC=c039c004
R00=00000000 R01=b96c226c R02=c15b1804 R03=00000000
R04=ffffffff R05=00000000 R06=00000012 R07=0000000e
R08=0000001a R09=c00075e0 R10=c15b0000 R11=00000000
R12=0000001b R13=00000010 R14=c1076840 R15=00000005
R16=c1076840 R17=00000000 R18=c1f69758 R19=00000000
R20=00ca4800 R21=00000000 R22=00000428 R23=00000003
R24=01511e00 R25=ce63f7b2 R26=0000042c R27=22bdcb7b
R28=00000005 R29=3416c000 R30=ffffffff R31=c62b0000
Seen with 5.7-rc2
Example:
/ # cat /proc/690/stack
[108997.050000] Unable to handle kernel access
[108997.050000] at virtual address 0x7fc60f58
[108997.050000]
[108997.050000] Oops#: 0000
[108997.060000] CPU #: 0
[108997.060000] PC: c00097fc SR: 0000807f SP: d6f09b9c
[108997.060000] GPR00: 00000000 GPR01: d6f09b9c GPR02: d6f09bb8 GPR03: d6f09bc4
[108997.060000] GPR04: 7fc60f5c GPR05: c00099b4 GPR06: 00000000 GPR07: d6f09ba3
[108997.060000] GPR08: ffffff00 GPR09: c0009804 GPR10: d6f08000 GPR11: 00000000
[108997.060000] GPR12: ffffe000 GPR13: dbb86000 GPR14: 00000001 GPR15: dbb86250
[108997.060000] GPR16: 7fc60f63 GPR17: 00000f5c GPR18: d6f09bc4 GPR19: 00000000
[108997.060000] GPR20: c00099b4 GPR21: ffffffc0 GPR22: 00000000 GPR23: 00000000
[108997.060000] GPR24: 00000001 GPR25: 000002c6 GPR26: d78b6850 GPR27: 00000001
[108997.060000] GPR28: 00000000 GPR29: dbb86000 GPR30: ffffffff GPR31: dbb862fc
[108997.060000] RES: 00000000 oGPR11: ffffffff
[108997.060000] Process cat (pid: 702, stackpage=d79d6000)
[108997.060000]
[108997.060000] Stack:
[108997.060000] Call trace:
[108997.060000] [<598977f2>] save_stack_trace_tsk+0x40/0x74
[108997.060000] [<95063f0e>] stack_trace_save_tsk+0x44/0x58
[108997.060000] [<b557bfdd>] proc_pid_stack+0xd0/0x13c
[108997.060000] [<a2df8eda>] proc_single_show+0x6c/0xf0
[108997.060000] [<e5a737b7>] seq_read+0x1b4/0x688
[108997.060000] [<2d6c7480>] do_iter_read+0x208/0x248
[108997.060000] [<2182a2fb>] vfs_readv+0x64/0x90
[108997.060000] [<3ff02271>] ? iov_iter_get_pages_alloc+0x3d4/0x670
[108997.060000] [<bab8ec1a>] ? slob_alloc.isra.0+0xfc/0x348
[108997.070000] [<d58f9294>] ? __wake_up_common_lock+0x90/0xcc
[108997.070000] [<49544e36>] default_file_splice_read+0x1cc/0x3b0
[108997.070000] [<ab9ebd9e>] ? kmem_cache_alloc+0x48/0x104
[108997.070000] [<6dd300f2>] do_splice_to+0xa8/0x100
[108997.070000] [<caed44c0>] splice_direct_to_actor+0xf8/0x33c
[108997.070000] [<b2a5b2b1>] ? direct_splice_actor+0x0/0x70
[108997.070000] [<886bec27>] do_splice_direct+0xa4/0x100
[108997.070000] [<d3c7c436>] do_sendfile+0x2bc/0x4f0
[108997.070000] [<bd6b01e4>] sys_sendfile64+0x130/0x138
[108997.070000] [<7f032d7a>] ? _syscall_return+0x0/0x4
When running in qemu with latest 5.5 kernel with networking turned on.
We get transmit queue timeouts.
Trace:
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.670000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.680000] nfs: server 10.0.0.27 not responding, still trying
[ 259.700000] ------------[ cut here ]------------
[ 259.700000] WARNING: CPU: 0 PID: 7 at net/sched/sch_generic.c:442 dev_watchdog+0x2dc/0x2e4
[ 259.700000] NETDEV WATCHDOG: eth0 (ethoc): transmit queue 0 timed out
[ 259.700000] Modules linked in:
[ 259.700000] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 5.5.0-simple-smp-00006-g9737e2c5f0bc #90
[ 259.700000] Call trace:
[ 259.700000] [<(ptrval)>] show_stack+0x48/0x58
[ 259.710000] [<(ptrval)>] dump_stack+0x28/0x38
[ 259.710000] [<(ptrval)>] __warn+0xf0/0x120
[ 259.710000] [<(ptrval)>] ? dev_watchdog+0x2dc/0x2e4
[ 259.710000] [<(ptrval)>] warn_slowpath_fmt+0x7c/0x94
[ 259.710000] [<(ptrval)>] dev_watchdog+0x2dc/0x2e4
[ 259.710000] [<(ptrval)>] ? dev_watchdog+0x0/0x2e4
[ 259.710000] [<(ptrval)>] call_timer_fn.isra.0+0x24/0xa0
[ 259.710000] [<(ptrval)>] run_timer_softirq+0x238/0x544
[ 259.710000] [<(ptrval)>] ? dev_watchdog+0x0/0x2e4
[ 259.710000] [<(ptrval)>] ? mempool_kfree+0x14/0x24
[ 259.710000] [<(ptrval)>] ? mempool_free+0x54/0xec
[ 259.710000] [<(ptrval)>] ? rcu_process_callbacks+0x9c/0x144
[ 259.710000] [<(ptrval)>] __do_softirq+0x128/0x320
[ 259.710000] [<(ptrval)>] ? smpboot_thread_fn+0x0/0x26c
[ 259.710000] [<(ptrval)>] run_ksoftirqd+0x50/0x70
[ 259.710000] [<(ptrval)>] smpboot_thread_fn+0x1b8/0x26c
[ 259.710000] [<(ptrval)>] ? __kthread_parkme+0x68/0xa8
[ 259.710000] [<(ptrval)>] kthread+0x120/0x150
[ 259.710000] [<(ptrval)>] ? kthread+0x0/0x150
[ 259.710000] [<(ptrval)>] ret_from_fork+0x1c/0x9c
[ 259.720000] ---[ end trace 2e12edb0824fd0ae ]---
[ 260.660000] nfs: server 10.0.0.27 OK
[ 260.660000] nfs: server 10.0.0.27 OK
[ 260.700000] nfs: server 10.0.0.27 OK
[ 260.740000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
[ 260.760000] nfs: server 10.0.0.27 OK
Even after increasing memory to 512mb on qemu, I still occasionally see this.
/ # [90216.140000] watchdog: BUG: soft lockup - CPU#0 stuck for 186s! [kworker/u3:3:2073]
[90216.140000] Modules linked in:
[90216.140000] CPU: 0 PID: 2073 Comm: kworker/u3:3 Tainted: G W 5.7.0-rc2-simple-smp-00001-g6bd140e14d9a-dirty #109
[90216.140000] Workqueue: xprtiod xs_stream_data_receive_workfn
[90216.140000] CPU #: 0
[90216.140000] PC: c03c1694 SR: 00008a7f SP: d89a3c90
[90216.140000] GPR00: 00000000 GPR01: d89a3c90 GPR02: d89a3cb4 GPR03: 00000000
[90216.140000] GPR04: eb895d10 GPR05: 00000000 GPR06: c8000000 GPR07: 00000000
[90216.140000] GPR08: c029c738 GPR09: c006b898 GPR10: d89a2000 GPR11: fffffe5c
[90216.140000] GPR12: 1a69ac61 GPR13: 00000000 GPR14: df4f5800 GPR15: 00000000
[90216.140000] GPR16: c08c2c20 GPR17: eb895d10 GPR18: 20000000 GPR19: 00000001
[90216.140000] GPR20: 098a4d3e GPR21: 000001a3 GPR22: 000051df GPR23: 00000000
[90216.140000] GPR24: 0000001a GPR25: 00000000 GPR26: 01125cf2 GPR27: 00000000
[90216.140000] GPR28: 002c771e GPR29: c04c4fa1 GPR30: df4f5b9c GPR31: c04b63c0
[90216.140000] RES: fffffe5c oGPR11: ffffffff
[90216.140000] Process kworker/u3:3 (pid: 2073, stackpage=d8850000)
[90216.140000]
[90216.140000] Stack:
[90216.140000] Call trace:
[90216.140000] [<8558de7f>] ktime_get+0x70/0x1bc
[90216.140000] [<8d9c6218>] tcp_mstamp_refresh+0x30/0x190
[90216.140000] [<d9699fc9>] tcp_rcv_space_adjust+0x38/0x300
[90216.140000] [<93cbbb5f>] tcp_recvmsg+0x558/0xebc
[90216.140000] [<a2442133>] ? __switch_to+0x50/0x7c
[90216.140000] [<e17f0602>] ? __enqueue_entity.constprop.0+0xc4/0x104
[90216.140000] [<85044956>] inet_recvmsg+0x3c/0x64
[90216.140000] [<96e0a9e6>] sock_recvmsg+0x24/0x34
[90216.140000] [<a02d272c>] xs_sock_recvmsg.constprop.0+0x44/0x88
[90216.140000] [<4c68e7cc>] xs_stream_data_receive_workfn+0x110/0xaac
[90216.140000] [<e17f0602>] ? __enqueue_entity.constprop.0+0xc4/0x104
[90216.140000] [<ee8f26ff>] process_one_work+0x254/0x538
[90216.140000] [<a246f55d>] worker_thread+0x64/0x694
[90216.140000] [<3bf47ca9>] ? worker_thread+0x0/0x694
[90216.140000] [<818fceb2>] kthread+0x120/0x150
[90216.140000] [<0db975eb>] ? kthread+0x0/0x150
[90216.140000] [<336cbdd5>] ret_from_fork+0x1c/0x9c
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.