Load Spike

[jcoehoorn]

Last night I had a load alert. Checking the graph this morning, it looks like this:



Now I've had high load before. This doesn't even set a record here (I've seen it over 2500 back before they rewrote the network stack for v10). I've even had spikes before. Just never anything even close to this... spikey. Nothing this intense (load over 500) was ever this brief (lasted less than 15 minutes, probably less than 5) or sudden.

What could cause a load spike like this?

[dashpuppy]

Anything going out or in on the WAN ?

[gravenscroft]

Off the top of my head: sudden influx of traffic, either inward or outward. It appears to be a very short spike, so it's hard to tell. How long did it last?

What time did it occur? NG Firewall does its maintenance stuff around 01:30 server time. Could be something got stuck during checks for updates, database cycling, &c.

[dashpuppy]

Off the top of my head: sudden influx of traffic, either inward or outward. It appears to be a very short spike, so it's hard to tell. How long did it last?

What time did it occur? NG Firewall does its maintenance stuff around 01:30 server time. Could be something got stuck during checks for updates, database cycling, &c.

That's what i was "thinking" it called home did a database update for maybe av or something. BUT would have to match the wan traffic around that time to see.

[sky-knight]

I haven't seen a spike like that unless the SSD went read only for a short time. But that's also on a busy server, this one isn't busy... and a dying SSD would be generating intermittent spikes in the depicted window.

So yeah, short of a very short duration DOS attack... I have no clue what could cause that.

[jcoehoorn]

Anything going out or in on the WAN ?

This was about 11:40pm last night. It definitely does not correspond to the nightly reports. I'm not sure of the exact duration, but it was less than 15 minutes from start to end, maybe even less than 5. There was a corresponding spike in upstream traffic about that time, but it was far from the biggest or steepest spike from that day:

We see the traffic spike near 4:00pm was bigger, and the traffic spikes near 8:00pm were just as steep.

I'd like to look at session logs, but this server can exceed 10 millions sessions per day, so the session reports have never been particularly useful here except in aggregate, and the aggregate for that time frame doesn't show anything unusual.

[sky-knight]

If you're stuck using aggregate reporting in general, then My assumption would be a rapid onslaught of new network sessions causing that spike. New sessions hit the CPU for setup, RAM for throughput. And if you get a burst of something, you'll see a huge, but short lived CPU spike as a result.