RTSP camera monitoring: the 60 Mbit/⁠s that never existed

We were adding a Bitrate column to our monitors list: click the header and the heaviest video streams float to the top. Before shipping it, we sorted real data with it. One customer runs 458 cameras in a single workspace, and we had about 21,000 of their video samples. The top row would have been a 60 Mbit/s sample. During that sample the camera had sent us 300 KB in two and a half seconds — roughly 1 Mbit/s.

Nothing was broken, neither the camera nor the measurement. The sample divided bytes by time, correctly; it just used the wrong time. This post walks through how we monitor IP cameras over RTSP, how we measure a stream without decoding a single frame, the hypothesis we got backwards, and the half-second rule that removed every bogus value while costing us 18 honest ones out of 20,991.

TL;DR

• “It pings” is not “there is video”. You need at least two layers: an RTSP DESCRIBE every 1–5 minutes and a look at the stream itself every 15 minutes.
• Frame rate and bitrate can be measured from RTP timestamps and packet sizes, with no decoding. Arrival time is the wrong clock: TCP delivers frames in bursts.
• Short samples lie loudly: two keyframes over 40 ms of media make 60 Mbit/s. We no longer store a bitrate measured over less than 0.5 s of media.
• The average over the wire lies too, downward, whenever the stream stalls. We trusted it over RTP at first and were wrong.

“It pings” is not “there is video”

Ping is answered by the camera’s network stack, port 554 by its RTSP server, and the video comes from the encoder. They fail independently. A camera can answer ping and RTSP requests for days while its stream carries zero frames.

Typical failures, from the field and from vendor forums:

• The RTSP service hangs while the network stays up. Ping works, the port is open, and neither the recorder nor a player gets a stream. Only a reboot helps.
• A power cut resets the camera to factory settings. New password, new stream, sometimes a new address. Ping still answers; the old stream URL doesn’t.
• All connection slots are taken. Many models serve only a few concurrent streams. A recorder, a couple of phones, the monitoring — and the next client is refused.
• The stream is there, but bad. The frame rate drops, keyframes come tens of seconds apart, packets get lost, delivery stalls for seconds.
• Night. The IR illuminator turns on, power draw rises, and on a tight PoE budget the camera starts rebooting.

That is why camera checks come in layers, from cheap to deep:

Picture analysis means decoding video; recorder health means talking to the recorder’s API. We stopped at the fourth layer: it catches nearly everything on the list above for a few seconds of stream every 15 minutes.

How we check a camera

RTSP does not carry video. It negotiates what the camera should send and how, and the picture travels separately, over RTP (RFC 7826, RFC 3550). That split is handy: you can question a camera without opening any video, or open the stream for a few seconds and see what is in it.

A few decisions that matter once you have hundreds of cameras:

• The video check takes the place of a regular one. When it is due, the checker opens the stream instead of sending OPTIONS and DESCRIBE. So with a 15-minute interval a sample lands about every 16 minutes: it waits for the next regular tick.
• The first sample lands at a random point in the interval. Otherwise a checker restart would open 458 streams at once.
• Samples are short. A sample ends as soon as 3 seconds have passed and two keyframes have arrived, and it never watches the stream for more than 10 seconds — usually about a megabyte of traffic. Keep this rule in mind; it comes back later.
• One login per check. Cameras lock logins after a few wrong passwords, so we never repeat one the camera refused. More on that in the recorder section.

Measuring a stream without decoding it

You don’t need to decode a single picture to get the frame rate and bitrate. Everything is in the RTP headers and the packet sizes.

• Every packet carries a sequence number. Gaps in the numbering give packet loss, computed the way appendix A of RFC 3550 describes.
• Every packet carries a media timestamp. The RTP clock for video runs at 90 kHz, and all packets of one frame share a timestamp: a new timestamp means a new frame.
• A keyframe shows up in the NAL unit type at the start of the payload — an IDR, type 5, for H.264. The resolution, and the frame rate the camera is set to when it declares one, come from the SPS, which the camera sends in the SDP or inside the stream.

Simplified, the math looks like this:

CLOCK = 90_000  # RTP clock for video, ticks per second

def rates(frames):
    """frames: [(rtp_timestamp, payload_bytes), ...], one entry per frame."""
    stamps = sorted(ts for ts, _ in frames)
    media_s = (stamps[-1] - stamps[0]) / CLOCK  # media covered by the sample
    fps = (len(stamps) - 1) / media_s
    kbps = sum(size for _, size in frames) * 8 / media_s / 1000
    return fps, kbps

Why RTP time and not the checker’s clock? Over TCP, frames arrive in bursts, and arrival time carries all of the network’s unevenness. One real camera read 15.8 fps by arrival time and its honest 25.0 by RTP timestamps.

We still record arrival time, for something else: if two frames arrive 2.5 seconds apart while their timestamps are 40 ms apart, delivery stalled. That is a separate metric, the delivery stall, and it will matter below.

Out of the box we call a stream bad when PLAY succeeds but no frame arrives, when delivery stalls for more than 2 seconds, when loss goes above 1 %, or when there are fewer than half the frames the camera is set to. You can add your own limits for frame rate, keyframe interval and bitrate, plus a format-change warning, which is how a factory reset gets caught. A video problem is reported only after a second sample confirms it, and the checker takes that sample right away, on the next tick.

Remember rates. The 60 Mbit/s was hiding in it.

The Bitrate column

With a dozen cameras, each camera’s own page is enough. With 458 you want one sort that shows the heaviest and the lightest streams. Hence the Bitrate column in the monitors list:

• it appears only where the workspace has cameras;
• the first click on the header sorts heaviest first;
• the cell shows the bitrate of the latest video sample, and its tooltip says when that sample was taken;
• paused and down cameras show “—” and sort last.

Zoom

It needed no new database column. Each check result already keeps the sample’s metrics in JSONB, so the list fetches the latest video sample of every camera on the page in one query:

SELECT cr.*
FROM unnest(
    CAST(:monitor_ids AS int[]),
    CAST(:sinces AS timestamptz[])
) AS m(id, since)
CROSS JOIN LATERAL (
    SELECT *
    FROM check_results cr
    WHERE cr.monitor_id = m.id
      AND cr.created_at >= m.since   -- at most two video intervals back
      AND cr.rtsp_metrics ? 'fps'    -- video samples only
    ORDER BY cr.created_at DESC
    LIMIT 1
) cr;

For 458 cameras it runs in 15 ms: each camera is one short index walk with LIMIT 1. What was left was making sure the sort could be trusted.

The investigation: we blamed the wrong bitrate

The data: 458 cameras, all with the video check every 15 minutes. Bitrates between 0.64 and 5.6 Mbit/s, median 2.3. And 22 samples out of about 21,000 — 0.1 % — between 3 and 60 Mbit/s.

There are two ways to compute a bitrate:

• from RTP: payload bytes divided by media time, as in rates;
• from the wire: every byte received, divided by how long the sample took on the checker’s clock.

We compared the two across all samples. For 19 cameras the RTP bitrate came out 2–3× higher than the wire average. The first hypothesis wrote itself: RTP overcounts — we mix up clocks somewhere, reassemble frames wrong, or count something twice.

It was the other way round. Those cameras’ streams stalled for 1.5–4 seconds. A sample that normally fits into 3 seconds stretched to 5–7: the same number of bytes divided by a longer time, so the wire average dropped. The RTP bitrate doesn’t see pauses, because it counts media time. The same camera confirmed it: 2,120 kbit/s by RTP on clean samples, 2,188 on the “inflated” ones.

The lesson we should have drawn on day one: the wire average answers “how much traffic arrived while we waited”, not “what does the camera stream”. A sort needs the second answer, so RTP stayed. But that didn’t explain 22 samples in the tens of megabits.

Where 60 Mbit/s came from

Here is the most striking of the 22, with the fields we need:

{"fps": 25.0, "bitrate_kbps": 60041, "wire_bytes": 303693,
 "sample_s": 2.511, "keyframe_interval_s": 0.04, "max_stall_ms": 2471}

How much media stands behind that bitrate?

• the sample received 303,693 bytes, which is 2.43 Mbit;
• 2.43 Mbit ÷ 60,041 kbit/s = 0.04 s, a single frame interval at 25 fps;
• on the checker’s clock the first and last frames were 2.51 s apart, and for 2.47 s of that the stream stood still (max_stall_ms);
• keyframe_interval_s is 0.04: two keyframes 40 ms apart in RTP time.

Reading the fields, the sample caught exactly two frames, both keyframes. The first arrived, the stream stalled for two and a half seconds, the second arrived — and the early-stop rule was satisfied: 3 seconds gone, two keyframes seen. A rule meant to save traffic cut the sample at two frames. Their timestamps were 40 ms apart, so rates dutifully divided 300 KB by 40 ms of media and got 60 Mbit/s. The same bytes over the wall-clock time give about 1 Mbit/s.

There is a second subtlety. In rates the bytes cover every frame of the sample, while the time covers the gaps between frames, and there is one gap fewer than there are frames. Over a hundred frames that fencepost error vanishes. Over two frames, one 40 ms gap gets charged with the bytes of two frames — and both were keyframes, which weigh several times more than regular frames.

That is 0.1 % of samples. Not much, but a heaviest-first sort pulls exactly those to the top: without a guard, a bogus value would have been the first row roughly every 40 minutes.

The fix: half a second of media, or no bitrate

We no longer store a bitrate measured over less than 0.5 s of media. The result has no field for the media span, but two stored numbers give it back:

MIN_BITRATE_MEDIA_S = 0.5

def bitrate_covers_too_little_media(m: dict) -> bool:
    media_s = m["wire_bytes"] * 8 / 1000 / m["bitrate_kbps"]
    return media_s < MIN_BITRATE_MEDIA_S

The threshold came from the data:

The rest of the sample — frame rate, keyframes, loss — stays; only the bitrate lies. The check runs on the server every time a check result is built: when it arrives from a checker and when it is read back from the database. So it covers every checker at once, including private ones that customers update on their own schedule, and the 22 old bogus values left the column and the charts with no migration. The server logs why each bitrate was dropped.

The price is 18 honest values out of 20,991. Until the next sample their cell shows “—” with the tooltip “The latest video check could not measure the bitrate.”

What else we could have done, and why we didn’t:

• Drop outliers against the camera’s usual bitrate. That would also hide real spikes, and those exist (next section).
• Show the median of several samples. The column would lag the camera by half an hour, and the per-camera query would get heavier.
• Sample longer. More traffic and camera load for 0.1 % of samples, and a stream can stall during a long sample too.

Real spikes: 0.8 to 30 Mbit/s in the evening

Not every outlier is a measurement error. Four cameras that usually stream about 0.8 Mbit/s climbed to 7–30 Mbit/s for roughly two hours every evening. The resolution didn’t change, and the wire bytes backed up the RTP bitrate: the stream really was that heavy.

Our hypothesis is night mode plus sensor noise in low light: a variable-bitrate (VBR) encoder faithfully spends bits on noise as if it were detail. We haven’t checked it against frames, so it stays a hypothesis.

Across the whole fleet, the hourly median bitrate doesn’t drop at night, even though the usual expectation is the opposite: a still night scene gives a VBR encoder little to encode. The practical takeaway: give a “bitrate at least” threshold a generous margin, and don’t treat an evening surge as an outage by default.

Sorting by the latest sample holds up

One question remained: is the sort itself noise? One sample per 16 minutes, and VBR cameras wander. We compared two snapshots 12 hours apart: 16 of the 20 heaviest cameras were still among the 40 heaviest. The middle of the list shuffles; the edges don’t. The column shows a property of the camera, not the luck of one sample.

The camera frame: the one place we decode

Bitrate and frame rate need no decoding. But sometimes you simply want to see what the camera sees: is it covered, turned away, taped over? That is what the camera frame on the camera’s page is for.

Zoom

How it works:

• the frame is taken only when you open the camera’s page or press “Refresh the frame”;
• the checker opens the stream, waits up to 10 seconds for the first complete keyframe and sends it along with the codec parameters (SPS and PPS);
• your browser decodes it with WebCodecs, and we don’t store it anywhere;
• workspace members with access to the monitor see it; it never appears on status pages or in notifications;
• for now it works for H.264, the format most cameras use for their substream.

Why the browser and not our checker? A video stream is untrusted input. Anything can call itself a camera, and parsing whatever H.264 it serves is safer inside a browser sandbox than with a video decoder on a checker that runs checks for every customer.

Recorders and password lockouts

Network video recorders (NVRs) come with two traps nearly everyone falls into.

Alert storms. A 16-channel recorder reboots, and monitoring that watches each channel separately sends 16 “camera down” messages. Nobody reads past the eighth. When the whole recorder stops answering, you want one message about the recorder; when a single channel breaks, a message about that channel.

Password lockouts. Cameras defend against password guessing. By default Hikvision blocks the IP address for 30 minutes after 7 wrong administrator passwords (5 for a regular user). Dahua locks the account after 5 wrong attempts, for 5 minutes by default. That leads to two unpleasant scenarios:

• the camera’s password changed, the monitoring keeps logging in with the old one, and a few checks later the address it comes from is blocked — behind NAT, that can be the address of the whole network;
• the monitoring shares an account with the recorder, and a locked account stops the recording.

So we log in to a camera at most once per check. If the camera refuses the password, we stay away from that device with that login for an hour, on every channel. Password checks against one device run one at a time, and we end each session with TEARDOWN so it doesn’t hold a camera slot until the timeout. Give the monitoring its own view-only account.

Where to run the checker

Cameras are checked only by checkers; the central server never connects to a camera. There are two placements, and both work:

• A public checker (Moscow, Novosibirsk, Minsk) for cameras with a public IP: port 554 is forwarded on the router and the check comes from the internet. Nothing to install on your side: add the stream URL and the checks start.
• A private checker inside the camera network, for cameras behind NAT or without a public IP. It connects out to PingZen over 443/TCP, so nothing is forwarded on the router.

A camera with a public IP is a normal, working setup, and basic hygiene is enough: your own login instead of admin, a long password, and a separate view-only account for monitoring. A long random password can’t be brute-forced, cameras lock out guessing on their own (see above), and we log in at most once per check and wait an hour after a rejected password, so the monitoring never triggers a lockout itself.

Setting one up is covered in the checker guide and on the local network monitoring page.

Checking an RTSP stream by hand

The quickest way is ffprobe from FFmpeg. It runs DESCRIBE, opens the stream and shows what is inside:

ffprobe -rtsp_transport tcp \
  "rtsp://login:password@CAMERA_ADDRESS:554/Streaming/Channels/101"

A line like Stream #0:0: Video: h264 ... means the stream is alive. 401 Unauthorized means a wrong login or password, 404 Not Found a wrong path. The stream path depends on the vendor and is listed in the camera’s manual and in its web interface, under the RTSP settings.

Checklist

• You check the RTSP answer to DESCRIBE, not just ping, every 1–5 minutes.
• Every 15–60 minutes you check the stream itself: frames, bitrate, keyframes, loss.
• Frame rate and bitrate come from RTP timestamps, not arrival times.
• A bitrate measured over a fraction of a second of media never reaches reports or sorts.
• The monitoring has its own view-only account, and a refused password is never retried in a loop.
• A “bitrate at least” threshold has a generous margin.
• A recorder outage arrives as one message.
• A public-IP camera has its own login instead of admin and a long password; cameras behind NAT are checked by a private checker.

What PingZen does today

• A “Camera” monitor type: a single camera, a stream link, or a whole recorder — 2 to 64 channels of a Hikvision, Dahua or Reolink box, with stream paths filled in for you.
• Availability every 1, 5 or 15 minutes, a video check every 5, 15 or 60 minutes, your own limits and a format-change warning. A real outage reaches you within a few minutes.
• The Bitrate column in the list and a “Video quality over time” chart on the camera’s page.
• A camera frame on demand.
• Public checkers, and private ones: up to 10 per account on the free plan.

What we don’t do: record or store video, analyse the picture or detect motion, or handle ONVIF events. Recording and viewing stay with your recorder.

If you have hundreds of cameras, CSV import beats adding them one by one. The short version of all this is on the IP camera monitoring page, everything else PingZen does is on the features page; to try it, sign up and add a monitor of the “Camera” type.

How do you watch your own cameras: ping, an ffprobe cron job, your VMS? And if you have seen evening bitrate spikes, what turned out to cause them? Tell us on Telegram; we’d like to compare notes.