IP Camera Troubleshooting Hell

TLDR: Among the many variables in an IP Camera networking setup, the POE standard required by the cameras is critical.

POE.  Three little letters.  A relatively simple concept.  What can go wrong?

So many things!

The Backstory

I have a zoneminder-powered home security setup running that I am very proud to have stood up all by myself. I pieced the server together, learned to make my own cables and eventually wound up with a reliable system.  Like all good things that aren't broken, I made every effort to avoid 'fixing' or 'improving' it.   Well, I'm going to have to add another verb that expression:  "If it ain't broke, don't MOVE it".   Even more correct might be "If IT ain't broke, don't move IT".

What should have been a minor re-organization of my garage required me to move my POE switch and router.  I went to bring the system back online and ... where were my cameras?    I moved the switch ten feet and suddenly nothing was working.

The rest of this post is the story of overcoming my frustration to solve this mystery.

The Struggle is Real

First and foremost, if you're struggling to get your POE IP cameras up and running, you're not alone.  The pitfalls are numerous - from faulty cables and networking to streaming video formats and zoneminder configuration.  Setting up a working system using all of these moving parts is complex and I'll be honest:   was beyond the extent of my patience.   More than once I became completely frustrated, cursed technology and any asshole that has ever used the words 'plug and play', and had to take breaks for days at a time.

At its core this is a troubleshooting challenge.  There is no greater frustration than following all the steps to a process and not getting the expected outcome.  One of the challenges of working with IP cameras is that nobody has exactly the same setup, so getting any kind of apples-to-apples advice is nearly impossible.  Even once you have a very specific question in mind, it will produce scant and disorganized google search results at best.

So its right about this point that you're probably wondering, "why bother?"  After all, we're well beyond the point where most people would have given up.  Yeah, well, 'challenge accepted'.  I'm not in the habit of letting technology win in a standoff like this. Technology works for me, and I will go hours out of my way to put it back in it's place.

Troubleshooting is Process is Checklists

To ensure that I hadn't overlooked or forgotten about any of the simple reasons for my camera blackout, I found a few "IP Camera Troubleshooting Checklists":

http://support.cleverloop.com/en/support/solutions/articles/6000177018-troubleshooting-poe-connected-cameras
https://ipvm.com/reports/top-10-camera-troubleshooting-tips
https://superuser.com/questions/1344007/ip-camera-dahua-is-not-tuned-on-but-detected-in-network

I tested cables, I checked wiring, I power cycled things in all various orders.   I calculated bandwidth, bought a new router, tried anything I could from the command line to determine if my cameras were connected. arp this, nmap that...

After I had exhausted the most comprehensive 15-item checklist that I could find and my issue wasn't solved, I started to feel.... special.  Surely whatever mistake I had made was so unique and awesome that nobody was even thinking about it as one of the 15 most common things that could go wrong.   Wonderful.  Now we're verifiably in the weeds.   I would not blame you for feeling a bit hopeless in this situation.  It is only by your own bootstraps that you are going to pull yourself out of this hole ( that you created! ).  Self confidence is critical here.

My Specific Setup

Note the following network diagram.   If you imagine that the cameras are missing from the diagram, that is what I was seeing.  

From the picture it's clear that my focus should have been on the POE switch, being that it is the component that bridges the clients/cameras from the router.  POE switches usually have 2 lights per port.  One will indicate that power is being delivered to the device, and the other will indicate network activity.   I was seeing the power LED illuminated for each connected camera, but no indication of network activity.

Because the cameras were receiving power ( which was also confirmed by camera clicks, infrared LED illumination ), I assumed that they should be joining the network.  Why the hell weren't they?  A common explanation would be an IP conflict ( two devices competing for the same IP ), but my devices are all configured to use static IPs and had not been factory reset.  I ensured that my DHCP server was using a range of IPs that included the statically assigned IPs ( nope ) - I tried the opposite ( nope ).

I finally picked up my POE switch when checking my connections for the 34th time, and that's when I heard a little rattle.  I tilted the switch back and forth, and yep - there was some shit rolling around in there.  I shook it all they way down to the end of the switch by a vent and saw that it was a large capacitor.   I assume this capacitor was responsible for doing something important, and so have to assume that important thing is no longer happening and the switch is ... less than 100% reliable ( aka useless aka dead to me ).

I weighted my options for replacing the switch on Amazon and had one shipped the next day.  This switch was even worse!  The cameras didn't even power on when plugged in!  I think this was the point of lowest self esteem because I really started to think I was barking up the wrong tree.  It did give me an important clue:  there was more to 'POE' than I understood, because there was something different about these two switches.  Mistakes are just valuable learning experiences :)

I spent several more hours reading tweaker IP camera geek forums after my wife went to bed.  Like a lone astronaut questing across the universe for answers.

POE has multiple specs

wiki: https://en.wikipedia.org/wiki/Power_over_Ethernet

The original POE spec is 802.3.af, which provides up to 15.4W per channel. This is compatible with most first generation POE stuff including basic IP cameras.

A higher power standard followed, 802.3at - "POE+" providing up to 25.5W per channel.  This powers more robust devices like IP cameras with PTZ.

The new switch I purchased was the IEEE 802.3az Energy-Efficient Ethernet (EEE) standard.   This has something to do with saving power when devices are not transmitting any data.  All I know is that this did not work at all with my IP cameras.

These standards are not always well labeled when shopping online.  Consider the following one that shows conflicting information. Which standard is it?

Conclusion

Troubleshooting is an art of the general and the specific.  You have to glean clues from whatever you can.  You have to rely on your inputs and experience, reconcile the facts they provide, and then not be misled by either.

A problem starts broadly. Something is broken. The cause is specific and a detailed investigation ensues. Any solution must solve the cause, whether fully understood or not.  Once again things are simple, working.

I think this is an inevitable cost of putting technology to real work.  Every machine requires maintenance, even the ones with no moving parts.  In the future more of our machines will contain computers, and being able to troubleshoot their (network connectivity) issues is going to consume a lot of hours of human effort.  

As a skill set, I feel that troubleshooting is underrated.