Intermittent hardware failure - restoring backup on a new system?

[gwar9999]

Hello all,

As of yesterday my Untangle NG system (16.6.2) has been experiencing crashes (32 and counting). Since the system is at least 5 years old I figure this would he a perfect time to replace it. I was able to successfully backup the system today and will have a replacement ready for install on Monday. After I install Untangle on the new system what can I expect by restoring from today's backup? The NIC devices will certainly be different as well as the CPUs Intel (old) vs AMD (new). Both systems have 2 NICs.

FWIW, I was tailing syslog during one of the crashes and this was the output:

Mar 17 14:01:20 untangle kernel: [ 8360.977316] BUG: kernel NULL pointer dereference, address: 0000000000000084
Mar 17 14:01:20 untangle kernel: [ 8360.977330] #PF: supervisor read access in kernel mode
Mar 17 14:01:20 untangle kernel: [ 8360.977334] #PF: error_code(0x0000) - not-present page
Mar 17 14:01:20 untangle kernel: [ 8360.977339] PGD 108890067 P4D 108890067 PUD 0
Mar 17 14:01:20 untangle kernel: [ 8360.977350] Oops: 0000 [#1] SMP NOPTI
Mar 17 14:01:20 untangle kernel: [ 8360.977358] CPU: 3 PID: 9969 Comm: TX#00 Tainted: G E 5.10.0-14-untangle-amd64 Home Debian 5.10.113-1+untangle3bullseye
Mar 17 14:01:20 untangle kernel: [ 8360.977363] Hardware name: To be filled by O.E.M. To be filled by O.E.M./2550L2D-MX, BIOS 4.6.5 05/30/2012
Mar 17 14:01:20 untangle kernel: [ 8360.977375] RIP: 0010:nf_conntrack_update+0x15c/0x370
Mar 17 14:01:20 untangle kernel: [ 8360.977381] Code: 83 e6 07 48 8b 83 b8 00 00 00 48 85 c0 0f 84 ef 00 00 00 0f b6 10 84 d2 0f 84 e4 00 00 00 48 01 d0 0f 84 db 00 00 00 48 8b 00 <f6> 80 84 00 00 00 01 0f 84 cb 00 00 00 0f b7 43 32 66 83 f8 02 0f
Mar 17 14:01:20 untangle kernel: [ 8360.977386] RSP: 0018:ffffbbd5053b7920 EFLAGS: 00010282
Mar 17 14:01:20 untangle kernel: [ 8360.977393] RAX: 0000000000000000 RBX: ffff94850ec19900 RCX: ffff94844452cc00
Mar 17 14:01:20 untangle kernel: [ 8360.977397] RDX: 0000000000000058 RSI: ffff94842c46f800 RDI: ffffffff8d619d80
Mar 17 14:01:20 untangle kernel: [ 8360.977401] RBP: ffffbbd5053b7990 R08: ffff9485016335d0 R09: ffffbbd5053b7bb0
Mar 17 14:01:20 untangle kernel: [ 8360.977406] R10: ffffbbd5053b79b0 R11: ffff948591030000 R12: ffff94842c46f800
Mar 17 14:01:20 untangle kernel: [ 8360.977410] R13: ffff94850ec19900 R14: 0000000000000000 R15: 0000000000000000
Mar 17 14:01:20 untangle kernel: [ 8360.977416] FS: 00007f9046ffd700(0000) GS:ffff948637d80000(0000) knlGS:0000000000000000
Mar 17 14:01:20 untangle kernel: [ 8360.977420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 17 14:01:20 untangle kernel: [ 8360.977424] CR2: 0000000000000084 CR3: 0000000106264000 CR4: 00000000000006a0
Mar 17 14:01:20 untangle kernel: [ 8360.977429] Call Trace:
Mar 17 14:01:20 untangle kernel: [ 8360.977449] ? nfqnl_reinject+0x38/0x60 [nfnetlink_queue]
Mar 17 14:01:20 untangle kernel: [ 8360.977456] nfqnl_reinject+0x38/0x60 [nfnetlink_queue]
Mar 17 14:01:20 untangle kernel: [ 8360.977465] nfqnl_recv_verdict+0x29f/0x4e0 [nfnetlink_queue]
Mar 17 14:01:20 untangle kernel: [ 8360.977475] nfnetlink_rcv_msg+0x14c/0x2a0
Mar 17 14:01:20 untangle kernel: [ 8360.977488] ? sched_clock+0x5/0x10
Mar 17 14:01:20 untangle kernel: [ 8360.977494] ? sched_clock_cpu+0xc/0xa0
Mar 17 14:01:20 untangle kernel: [ 8360.977500] ? newidle_balance+0x1d3/0x3c0
Mar 17 14:01:20 untangle kernel: [ 8360.977507] ? nfnetlink_net_exit_batch+0x60/0x60
Mar 17 14:01:20 untangle kernel: [ 8360.977514] netlink_rcv_skb+0x53/0x100
Mar 17 14:01:20 untangle kernel: [ 8360.977522] netlink_unicast+0x204/0x2d0
Mar 17 14:01:20 untangle kernel: [ 8360.977529] netlink_sendmsg+0x250/0x4a0
Mar 17 14:01:20 untangle kernel: [ 8360.977538] sock_sendmsg+0x5e/0x60
Mar 17 14:01:20 untangle kernel: [ 8360.977545] ____sys_sendmsg+0x22e/0x270
Mar 17 14:01:20 untangle kernel: [ 8360.977553] ? import_iovec+0x2d/0x30
Mar 17 14:01:20 untangle kernel: [ 8360.977558] ? sendmsg_copy_msghdr+0x7c/0xa0
Mar 17 14:01:20 untangle kernel: [ 8360.977565] ___sys_sendmsg+0x75/0xb0
Mar 17 14:01:20 untangle kernel: [ 8360.977572] ? do_futex+0x7c3/0xbc0
Mar 17 14:01:20 untangle kernel: [ 8360.977579] ? sched_clock+0x5/0x10
Mar 17 14:01:20 untangle kernel: [ 8360.977584] ? sched_clock_cpu+0xc/0xa0
Mar 17 14:01:20 untangle kernel: [ 8360.977590] ? psi_group_change+0x41/0x220
Mar 17 14:01:20 untangle kernel: [ 8360.977598] ? __fget_files+0x79/0xb0
Mar 17 14:01:20 untangle kernel: [ 8360.977604] __sys_sendmsg+0x59/0xa0
Mar 17 14:01:20 untangle kernel: [ 8360.977614] do_syscall_64+0x33/0x80
Mar 17 14:01:20 untangle kernel: [ 8360.977621] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Mar 17 14:01:20 untangle kernel: [ 8360.977628] RIP: 0033:0x7f905875be4d
Mar 17 14:01:20 untangle kernel: [ 8360.977634] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 ca ee ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 fe ee ff ff 48
Mar 17 14:01:20 untangle kernel: [ 8360.977638] RSP: 002b:00007f9046ffc850 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
Mar 17 14:01:20 untangle kernel: [ 8360.977645] RAX: ffffffffffffffda RBX: 00007f9050270dd0 RCX: 00007f905875be4d
Mar 17 14:01:20 untangle kernel: [ 8360.977649] RDX: 0000000000000000 RSI: 00007f9046ffc890 RDI: 0000000000000005
Mar 17 14:01:20 untangle kernel: [ 8360.977653] RBP: 00007f9046ffc8f0 R08: 0000000000000000 R09: 00007f9057a3a6b0
Mar 17 14:01:20 untangle kernel: [ 8360.977657] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000
Mar 17 14:01:20 untangle kernel: [ 8360.977661] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Mar 17 14:01:20 untangle kernel: [ 8360.977668] Modules linked in: xt_connbytes(E) xt_NFQUEUE(E) xfrm_user(E) xfrm_algo(E) twofish_generic(E) twofish_x86_64(E) twofish_common(E) serpent_sse2_x86_64(E) serpent_generic(E) blowfish_generic(E) blowfish_x86_64(E) blowfish_common(E) cast5_generic(E) cast_common(E) ctr(E) ecb(E) des_generic(E) libdes(E) cbc(E) algif_skcipher(E) camellia_generic(E) camellia_x86_64(E) aes_generic(E) libaes(E) crypto_simd(E) cryptd(E) glue_helper(E) xcbc(E) sha512_ssse3(E) sha512_generic(E) md4(E) algif_hash(E) af_alg(E) nf_conntrack_netlink(E) nfnetlink_queue(E) tun(E) l2tp_ppp(E) l2tp_netlink(E) l2tp_core(E) ip6_udp_tunnel(E) udp_tunnel(E) pppox(E) ppp_generic(E) slhc(E) xt_socket(E) nf_socket_ipv4(E) nf_socket_ipv6(E) xt_CT(E) xt_state(E) xt_recent(E) xt_IMQ(E) cls_u32(E) sch_fq_codel(E) sch_htb(E) imq(E) xt_REDIRECT(E) ip6t_REJECT(E) nf_reject_ipv6(E) ip6t_ah(E) ip6table_filter(E) ip6_tables(E) xt_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_NFLOG(E) xt_MASQUERADE(E) xt_set(E) ip_set_hash_ip(E)
Mar 17 14:01:20 untangle kernel: [ 8360.977792] ip_set(E) xt_multiport(E) xt_tcpudp(E) nf_log_ipv4(E) nf_log_common(E) xt_LOG(E) xt_policy(E) xt_mac(E) xt_connmark(E) xt_conntrack(E) xt_addrtype(E) xt_mark(E) xt_comment(E) ebt_redirect(E) ebt_ip(E) ebtable_broute(E) ebtables(E) iptable_filter(E) iptable_mangle(E) iptable_nat(E) iptable_tune(E) iptable_raw(E) nf_nat_irc(E) nf_conntrack_irc(E) ip_gre(E) ip_tunnel(E) gre(E) nf_nat_snmp_basic(E) nf_conntrack_snmp(E) nf_conntrack_broadcast(E) nf_nat_amanda(E) ts_kmp(E) nf_conntrack_amanda(E) nf_nat_h323(E) nf_conntrack_h323(E) nf_nat_pptp(E) nf_conntrack_pptp(E) nf_nat_tftp(E) nf_conntrack_tftp(E) nf_nat_ftp(E) nf_conntrack_ftp(E) bridge(E) stp(E) llc(E) nfnetlink_log(E) snd_hda_codec_hdmi(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) ledtrig_audio(E) snd_hda_intel(E) snd_intel_dspcfg(E) soundwire_intel(E) soundwire_generic_allocation(E) snd_soc_core(E) snd_compress(E) soundwire_cadence(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) soundwire_bus(E) snd_pcm(E)
Mar 17 14:01:20 untangle kernel: [ 8360.977918] snd_timer(E) snd(E) sg(E) ppdev(E) iTCO_wdt(E) soundcore(E) intel_pmc_bxt(E) iTCO_vendor_support(E) intel_powerclamp(E) watchdog(E) at24(E) coretemp(E) serio_raw(E) parport_pc(E) parport(E) evdev(E) pcspkr(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) i915(E) sd_mod(E) t10_pi(E) crc_t10dif(E) crct10dif_generic(E) crct10dif_common(E) broadcom(E) gma500_gfx(E) ahci(E) i2c_algo_bit(E) libahci(E) bcm_phy_lib(E) xhci_pci(E) drm_kms_helper(E) libata(E) psmouse(E) tg3(E) ehci_pci(E) uhci_hcd(E) scsi_mod(E) i2c_i801(E) xhci_hcd(E) lpc_ich(E) i2c_smbus(E) ehci_hcd(E) cec(E) libphy(E) usbcore(E) ptp(E) usb_common(E) drm(E) pps_core(E) fan(E) video(E) button(E)
Mar 17 14:01:20 untangle kernel: [ 8360.978044] CR2: 0000000000000084
Mar 17 14:01:20 untangle kernel: [ 8360.979771] ---[ end trace d8fe1c2b1830eb23 ]---
Mar 17 14:01:20 untangle kernel: [ 8360.979798] RIP: 0010:nf_conntrack_update+0x15c/0x370
Mar 17 14:01:20 untangle kernel: [ 8360.979849] Code: 83 e6 07 48 8b 83 b8 00 00 00 48 85 c0 0f 84 ef 00 00 00 0f b6 10 84 d2 0f 84 e4 00 00 00 48 01 d0 0f 84 db 00 00 00 48 8b 00 <f6> 80 84 00 00 00 01 0f 84 cb 00 00 00 0f b7 43 32 66 83 f8 02 0f
Mar 17 14:01:20 untangle kernel: [ 8360.979858] RSP: 0018:ffffbbd5053b7920 EFLAGS: 00010282
Mar 17 14:01:20 untangle kernel: [ 8360.979869] RAX: 0000000000000000 RBX: ffff94850ec19900 RCX: ffff94844452cc00
Mar 17 14:01:20 untangle kernel: [ 8360.979875] RDX: 0000000000000058 RSI: ffff94842c46f800 RDI: ffffffff8d619d80
Mar 17 14:01:20 untangle kernel: [ 8360.979882] RBP: ffffbbd5053b7990 R08: ffff9485016335d0 R09: ffffbbd5053b7bb0
Mar 17 14:01:20 untangle kernel: [ 8360.979889] R10: ffffbbd5053b79b0 R11: ffff948591030000 R12: ffff94842c46f800
Mar 17 14:01:20 untangle kernel: [ 8360.980105] R13: ffff94850ec19900 R14: 0000000000000000 R15: 0000000000000000
Mar 17 14:01:20 untangle kernel: [ 8360.980113] FS: 00007f9046ffd700(0000) GS:ffff948637d80000(0000) knlGS:0000000000000000
Mar 17 14:01:20 untangle kernel: [ 8360.980119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 17 14:01:20 untangle kernel: [ 8360.980125] CR2: 0000000000000084 CR3: 0000000106264000 CR4: 00000000000006a0

Considering the age of the system it's probably not worth troubleshooting.

Thanks for any info!

[gwar9999]

The hardware replacement went ok. The main problem was that I ordered a device that had unsupported Intel NICs so had to order a replacement-replacement. Anyhow, was able to restore my settings from the old device and things are mostly working. Though it appears that the UVM is crashing occasionally but need to view the logs to see if there's something wrong

[gwar9999]

Looks like my issues with the legacy hardware are re-occurring with the replacement hardware. Fortunately, the new system reboots much faster than the last so the network is only down for about 20 seconds vs about 90 seconds. Still, I suppose there is a bug in the kernel causing the issue but would think others must be seeing this:

Mar 23 10:56:28 untangle kernel: [ 8405.589792] BUG: kernel NULL pointer dereference, address: 0000000000000084
Mar 23 10:56:28 untangle kernel: [ 8405.589813] #PF: supervisor read access in kernel mode
Mar 23 10:56:28 untangle kernel: [ 8405.589821] #PF: error_code(0x0000) - not-present page
Mar 23 10:56:28 untangle kernel: [ 8405.589827] PGD 108bed067 P4D 108bed067 PUD 0
Mar 23 10:56:28 untangle kernel: [ 8405.589843] Oops: 0000 [#1] SMP NOPTI
Mar 23 10:56:28 untangle kernel: [ 8405.589853] CPU: 0 PID: 5996 Comm: TX#00 Tainted: G E 5.10.0-14-untangle-amd64 Home Debian 5.10.113-1+untangle3bullseye
Mar 23 10:56:28 untangle kernel: [ 8405.589863] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./NUC-EHL, BIOS P1.50 08/25/2021
Mar 23 10:56:28 untangle kernel: [ 8405.589879] RIP: 0010:nf_conntrack_update+0x15c/0x370
Mar 23 10:56:28 untangle kernel: [ 8405.589888] Code: 83 e6 07 48 8b 83 b8 00 00 00 48 85 c0 0f 84 ef 00 00 00 0f b6 10 84 d2 0f 84 e4 00 00 00 48 01 d0 0f 84 db 00 00 00 48 8b 00 <f6> 80 84 00 00 00 01 0f 84 cb 00 00 00 0f b7 43 32 66 83 f8 02 0f
Mar 23 10:56:28 untangle kernel: [ 8405.589900] RSP: 0018:ffffadae41acf920 EFLAGS: 00010286
Mar 23 10:56:28 untangle kernel: [ 8405.589909] RAX: 0000000000000000 RBX: ffff9693eb85f400 RCX: ffff969284567200
Mar 23 10:56:28 untangle kernel: [ 8405.589916] RDX: 0000000000000058 RSI: ffff969289b68900 RDI: ffffffff83819d80
Mar 23 10:56:28 untangle kernel: [ 8405.589923] RBP: ffffadae41acf990 R08: ffff969288822650 R09: ffffadae41acfbb0
Mar 23 10:56:28 untangle kernel: [ 8405.589929] R10: ffffadae41acf9b0 R11: ffff9692c03f0000 R12: ffff969289b68900
Mar 23 10:56:28 untangle kernel: [ 8405.589936] R13: ffff9693eb85f400 R14: 0000000000000000 R15: 0000000000000000
Mar 23 10:56:28 untangle kernel: [ 8405.589945] FS: 00007f0904a2e700(0000) GS:ffff9695f0400000(0000) knlGS:0000000000000000
Mar 23 10:56:28 untangle kernel: [ 8405.589953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 23 10:56:28 untangle kernel: [ 8405.589959] CR2: 0000000000000084 CR3: 00000001211e4000 CR4: 0000000000350eb0
Mar 23 10:56:28 untangle kernel: [ 8405.589966] Call Trace:
Mar 23 10:56:28 untangle kernel: [ 8405.590073] ? _ilk_disable_lp_wm+0x20/0xe0 [i915]
Mar 23 10:56:28 untangle kernel: [ 8405.590088] ? nfqnl_recv_verdict+0x29f/0x4e0 [nfnetlink_queue]
Mar 23 10:56:28 untangle kernel: [ 8405.590100] ? nfqnl_reinject+0x38/0x60 [nfnetlink_queue]
Mar 23 10:56:28 untangle kernel: [ 8405.590108] nfqnl_reinject+0x38/0x60 [nfnetlink_queue]
Mar 23 10:56:28 untangle kernel: [ 8405.590119] nfqnl_recv_verdict+0x29f/0x4e0 [nfnetlink_queue]
Mar 23 10:56:28 untangle kernel: [ 8405.590131] nfnetlink_rcv_msg+0x14c/0x2a0
Mar 23 10:56:28 untangle kernel: [ 8405.590146] ? update_load_avg+0x7a/0x5d0
Mar 23 10:56:28 untangle kernel: [ 8405.590155] ? dequeue_entity+0xc6/0x450
Mar 23 10:56:28 untangle kernel: [ 8405.590162] ? newidle_balance+0x282/0x3c0
Mar 23 10:56:28 untangle kernel: [ 8405.590171] ? nfnetlink_net_exit_batch+0x60/0x60
Mar 23 10:56:28 untangle kernel: [ 8405.590180] netlink_rcv_skb+0x53/0x100
Mar 23 10:56:28 untangle kernel: [ 8405.590189] netlink_unicast+0x204/0x2d0
Mar 23 10:56:28 untangle kernel: [ 8405.590198] netlink_sendmsg+0x250/0x4a0
Mar 23 10:56:28 untangle kernel: [ 8405.590209] sock_sendmsg+0x5e/0x60
Mar 23 10:56:28 untangle kernel: [ 8405.590218] ____sys_sendmsg+0x22e/0x270
Mar 23 10:56:28 untangle kernel: [ 8405.590227] ? import_iovec+0x2d/0x30
Mar 23 10:56:28 untangle kernel: [ 8405.590234] ? sendmsg_copy_msghdr+0x7c/0xa0
Mar 23 10:56:28 untangle kernel: [ 8405.590242] ___sys_sendmsg+0x75/0xb0
Mar 23 10:56:28 untangle kernel: [ 8405.590251] ? do_futex+0x7c3/0xbc0
Mar 23 10:56:28 untangle kernel: [ 8405.590259] ? check_preempt_curr+0x29/0x60
Mar 23 10:56:28 untangle kernel: [ 8405.590265] ? ttwu_do_wakeup+0x17/0x130
Mar 23 10:56:28 untangle kernel: [ 8405.590273] ? smp_call_function_single_async+0x17/0x20
Mar 23 10:56:28 untangle kernel: [ 8405.590281] ? update_process_times+0xb0/0xc0
Mar 23 10:56:28 untangle kernel: [ 8405.590288] ? __hrtimer_init+0xd0/0xd0
Mar 23 10:56:28 untangle kernel: [ 8405.590297] ? __fget_files+0x79/0xb0
Mar 23 10:56:28 untangle kernel: [ 8405.590304] __sys_sendmsg+0x59/0xa0
Mar 23 10:56:28 untangle kernel: [ 8405.590315] do_syscall_64+0x33/0x80
Mar 23 10:56:28 untangle kernel: [ 8405.590324] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Mar 23 10:56:28 untangle kernel: [ 8405.590333] RIP: 0033:0x7f0909dd6e4d
Mar 23 10:56:28 untangle kernel: [ 8405.590341] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 ca ee ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 fe ee ff ff 48
Mar 23 10:56:28 untangle kernel: [ 8405.590354] RSP: 002b:00007f0904a2d850 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
Mar 23 10:56:28 untangle kernel: [ 8405.590364] RAX: ffffffffffffffda RBX: 00007f0900270dd0 RCX: 00007f0909dd6e4d
Mar 23 10:56:28 untangle kernel: [ 8405.590370] RDX: 0000000000000000 RSI: 00007f0904a2d890 RDI: 0000000000000005
Mar 23 10:56:28 untangle kernel: [ 8405.590377] RBP: 00007f0904a2d8f0 R08: 0000000000000000 R09: 00007f09090b56b0
Mar 23 10:56:28 untangle kernel: [ 8405.590384] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000
Mar 23 10:56:28 untangle kernel: [ 8405.590390] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Mar 23 10:56:28 untangle kernel: [ 8405.590399] Modules linked in: snd_hda_codec_hdmi(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E) xt_connbytes(E) xt_NFQUEUE(E) xfrm_user(E) xfrm_algo(E) twofish_generic(E) twofish_x86_64_3way(E) twofish_x86_64(E) twofish_common(E) nf_conntrack_netlink(E) serpent_sse2_x86_64(E) serpent_generic(E) nfnetlink_queue(E) tun(E) blowfish_generic(E) blowfish_x86_64(E) blowfish_common(E) cast5_generic(E) cast_common(E) ctr(E) ecb(E) des_generic(E) libdes(E) cbc(E) algif_skcipher(E) camellia_generic(E) camellia_x86_64(E) xcbc(E) sha512_ssse3(E) sha512_generic(E) md4(E) algif_hash(E) af_alg(E) xt_socket(E) nf_socket_ipv4(E) nf_socket_ipv6(E) xt_CT(E) xt_state(E) xt_recent(E) xt_IMQ(E) cls_u32(E) sch_fq_codel(E) sch_htb(E) imq(E) xt_REDIRECT(E) ip6t_REJECT(E) nf_reject_ipv6(E) ip6t_ah(E) ip6table_filter(E) ip6_tables(E) xt_nat(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_NFLOG(E) xt_MASQUERADE(E) xt_set(E) ip_set_hash_ip(E) ip_set(E) xt_multiport(E) xt_tcpudp(E) nf_log_ipv4(E) nf_log_common(E) xt_LOG(E)
Mar 23 10:56:28 untangle kernel: [ 8405.590520] xt_policy(E) xt_mac(E) xt_connmark(E) xt_conntrack(E) xt_addrtype(E) xt_mark(E) xt_comment(E) ebt_redirect(E) ebt_ip(E) ebtable_broute(E) ebtables(E) iptable_filter(E) iptable_mangle(E) iptable_nat(E) iptable_tune(E) iptable_raw(E) nf_nat_irc(E) nf_conntrack_irc(E) ip_gre(E) ip_tunnel(E) gre(E) nf_nat_snmp_basic(E) nf_conntrack_snmp(E) nf_conntrack_broadcast(E) nf_nat_amanda(E) ts_kmp(E) nf_conntrack_amanda(E) nf_nat_h323(E) nf_conntrack_h323(E) nf_nat_pptp(E) nf_conntrack_pptp(E) nf_nat_tftp(E) nf_conntrack_tftp(E) nf_nat_ftp(E) nf_conntrack_ftp(E) bridge(E) stp(E) llc(E) snd_sof_pci(E) snd_sof_intel_byt(E) snd_sof_intel_ipc(E) intel_rapl_msr(E) intel_rapl_common(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) snd_sof_intel_hda_common(E) coretemp(E) snd_sof_xtensa_dsp(E) snd_sof(E) btusb(E) snd_sof_intel_hda(E) btrtl(E) snd_soc_hdac_hda(E) kvm_intel(E) snd_hda_ext_core(E) snd_soc_acpi_intel_match(E) btbcm(E) snd_soc_acpi(E) btintel(E) ledtrig_audio(E) bluetooth(E) kvm(E)
Mar 23 10:56:28 untangle kernel: [ 8405.590673] nfnetlink_log(E) jitterentropy_rng(E) irqbypass(E) snd_hda_intel(E) drbg(E) mei_hdcp(E) snd_intel_dspcfg(E) ghash_clmulni_intel(E) aes_generic(E) soundwire_intel(E) soundwire_generic_allocation(E) iwlmvm(E) mac80211(E) snd_soc_core(E) libarc4(E) nls_ascii(E) snd_compress(E) nls_cp437(E) iwlwifi(E) soundwire_cadence(E) vfat(E) fat(E) snd_hda_codec(E) iTCO_wdt(E) wmi_bmof(E) intel_cstate(E) snd_hda_core(E) snd_hwdep(E) soundwire_bus(E) intel_pmc_bxt(E) iTCO_vendor_support(E) pcspkr(E) snd_pcm(E) watchdog(E) intel_wmi_thunderbolt(E) snd_timer(E) efi_pstore(E) aesni_intel(E) snd(E) crypto_simd(E) cryptd(E) glue_helper(E) ansi_cprng(E) cfg80211(E) soundcore(E) mei_me(E) ecdh_generic(E) ecc(E) libaes(E) rfkill(E) mei(E) idma64(E) tpm_crb(E) tpm_tis(E) tpm_tis_core(E) tpm(E) rng_core(E) intel_pmc_core(E) evdev(E) acpi_cpufreq(E) acpi_tad(E) acpi_pad(E) fuse(E) configfs(E) efivarfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) ahci(E) nvme(E) libahci(E)
Mar 23 10:56:28 untangle kernel: [ 8405.590851] libata(E) nvme_core(E) t10_pi(E) scsi_mod(E) crc_t10dif(E) crc32_pclmul(E) i915(E) i2c_i801(E) i2c_smbus(E) r8169(E) crc32c_intel(E) crct10dif_generic(E) i2c_algo_bit(E) crct10dif_pclmul(E) crct10dif_common(E) realtek(E) mdio_devres(E) drm_kms_helper(E) libphy(E) cec(E) xhci_pci(E) xhci_hcd(E) drm(E) usbcore(E) usb_common(E) fan(E) wmi(E) video(E) button(E)
Mar 23 10:56:28 untangle kernel: [ 8405.590963] CR2: 0000000000000084
Mar 23 10:56:28 untangle kernel: [ 8405.590971] ---[ end trace f33766ef957b6694 ]---
Mar 23 10:56:28 untangle kernel: [ 8405.728049] RIP: 0010:nf_conntrack_update+0x15c/0x370
Mar 23 10:56:28 untangle kernel: [ 8405.728060] Code: 83 e6 07 48 8b 83 b8 00 00 00 48 85 c0 0f 84 ef 00 00 00 0f b6 10 84 d2 0f 84 e4 00 00 00 48 01 d0 0f 84 db 00 00 00 48 8b 00 <f6> 80 84 00 00 00 01 0f 84 cb 00 00 00 0f b7 43 32 66 83 f8 02 0f
Mar 23 10:56:28 untangle kernel: [ 8405.728068] RSP: 0018:ffffadae41acf920 EFLAGS: 00010286
Mar 23 10:56:28 untangle kernel: [ 8405.728074] RAX: 0000000000000000 RBX: ffff9693eb85f400 RCX: ffff969284567200
Mar 23 10:56:28 untangle kernel: [ 8405.728078] RDX: 0000000000000058 RSI: ffff969289b68900 RDI: ffffffff83819d80
Mar 23 10:56:28 untangle kernel: [ 8405.728082] RBP: ffffadae41acf990 R08: ffff969288822650 R09: ffffadae41acfbb0
Mar 23 10:56:28 untangle kernel: [ 8405.728087] R10: ffffadae41acf9b0 R11: ffff9692c03f0000 R12: ffff969289b68900
Mar 23 10:56:28 untangle kernel: [ 8405.728091] R13: ffff9693eb85f400 R14: 0000000000000000 R15: 0000000000000000
Mar 23 10:56:28 untangle kernel: [ 8405.728096] FS: 00007f0904a2e700(0000) GS:ffff9695f0400000(0000) knlGS:0000000000000000
Mar 23 10:56:28 untangle kernel: [ 8405.728101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 23 10:56:28 untangle kernel: [ 8405.728105] CR2: 0000000000000084 CR3: 00000001211e4000 CR4: 0000000000350eb0
Mar 23 10:57:19 untangle systemd-udevd[277]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Mar 23 10:57:19 untangle systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 23 10:57:19 untangle systemd[1]: Finished Coldplug All udev Devices.
Mar 23 10:57:19 untangle kernel: [ 0.000000] Linux version 5.10.0-14-untangle-amd64 ([email protected]) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) Home SMP Debian 5.10.113-1+untangle3bullseye (2022-07-26)
Mar 23 10:57:19 untangle kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-14-untangle-amd64 root=UUID=2a574850-d85a-4b2f-9579-1aaa3e8ad2ed ro net.ifnames=0 ramdisk_size=100000 lang=us apm=power-off nomce nodhcp nofstab panic=5 quiet splash
Mar 23 10:57:19 untangle kernel: [ 0.000000] x86/split lock detection: warning about user-space split_locks
Mar 23 10:57:19 untangle kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Mar 23 10:57:19 untangle kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Mar 23 10:57:19 untangle kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x3, context size is 576 bytes, using 'compacted' format.
Mar 23 10:57:19 untangle kernel: [ 0.000000] BIOS-provided physical RAM map:

I can see it rebooted when logging in right after the outage:

$ uptime
11:00:43 up 3 min, 2 users, load average: 0.61, 0.98, 0.46

I'm not sure when my legacy hardware upgraded to 16.6.2 (since I had auto updates enabled) so I wonder if that is what introduced the problems. Anybody else noticing reboot issues with 16.6.2? If you have faster hardware than me, you might not even notice it unless you ssh in and run "uptime".

[jcoffin]

What log is that from? I don't recognized the format.

What is the output from uname -a ?

[gwar9999]

That's from /var/log/syslog. I did google the bug and others have seen the error with different kernels.

$ uname -a
Linux untangle.foobar 5.10.0-14-untangle-amd64 Home SMP Debian 5.10.113-1+untangle3bullseye (2022-07-26) x86_64 GNU/Linux

[jcoffin]

The kernel is correct. At this point it seems to be a Debian Bullseye kernel incompatibility with your hardware. What processor and hardware is this?

[gwar9999]

uptime:	2m
Server:	custom
CPU Count:	4
CPU Type:	Intel(R) Celeron(R) J6412 @ 2.00GHz
Architecture:	amd64
Memory:	16.16 GB
Disk:	499.39 GB

My prior hardware, that also exhibited the same failures was:

CPU Type: Intel(R) Atom(TM) CPU D2550 @ 1.86GHz
CPU Count: 4


Which version of Untangle used a different kernel? I wonder if there's a way to downgrade Untangle to an earlier kernel which didn't seem to have this problem. It began happening I believe around 3/15 so not sure if my system was auto-updated around then. I did delete it from my appliances because I needed to release the license from it to attach to the new hardware. Hopefully the data is available somewhere.

[jcoffin]

NGFW uses the a slightly modified version of the Debian kernel. There was a kernel change when going from Debian Buster to Bullseye. The same kernel changed occurred with 16.5.2 (Buster based) to 16.6.2 (Bullseye). If there is a 16.5.2 settings backup, you could install 16.5.2 and restore from previous version backup. 16.6.2 backup is not compatible with 16.5.2.

[gwar9999]

Ok thanks. Looks like 16.5.2 isn't available from https://wiki.edge.arista.com/index.p...wall_Downloads. Though I'm not sure if I have any backups from that version. I know they were backed up to the cloud but not sure if they're available since I deleted the old appliance. Is there a way to restore it and access the backups?

I did manage to find a backup from 15.0 (which is avail from the downloads page but not sure if it's wise to go back that far. Assuming I can download 15.0 or later which versions are compatible with the 15.0 backup file (ie. could I install 16.5.2 and restore a 15.0 backup?)

[jcoffin]

16.5.2 is available on the ETM downloads page.

[gwar9999]

Ahh didn't notice multiple releases on that page. Will 16.5.2 be able to parse and restore a 15.0 backup? If not, what version can I downgrade to in order to handle a 15.0 backup file?

[gravenscroft]

Will 16.5.2 be able to parse and restore a 15.0 backup? If not, what version can I downgrade to in order to handle a 15.0 backup file?

Backups are good for the version they were taken on and one minor version forward. Yours can be used on 15.0 and 15.1.

[gwar9999]

Thanks! So my best path forward would be to install 15.1 restore from 15.0 backup. Backup 15.1, install 16.01 restore from 15.1, ... 16.5.1 since I believe from what jcoffin said 16.5.2 changed the kernel so I need to stick to 16.5.1.

[jcoffin]

Thanks! So my best path forward would be to install 15.1 restore from 15.0 backup. Backup 15.1, install 16.01 restore from 15.1, ... 16.5.1 since I believe from what jcoffin said 16.5.2 changed the kernel so I need to stick to 16.5.1.

Just upgrade after installing 15.1 and restoring. 16.5.2 is the last version on the previous kernel. 16.6.0 and on has the latest kernel.

[gwar9999]

Ok I think I got it but want to make sure.... install the 15.1 ISO, restore the 15.0 backup file and then within Untangle 15.1 I can upgrade and select 16.5.2 rather than it pick the latest? Right now, on 16.6.2 there is no upgrade available (as expected) so not sure what the Upgrade page looks like if I'm on 15.1.