Sorry to double post, but I just figured out what my issue was. Maybe this will help someone else.
Before I started having this issue, I needed to manually change the nuitrack.config file so my Color stream would work. (Referenced in this thread: How can I show camera frame in my Unity Project with Nuitrack Sdk?)
By default it’s at -1, in my case it’s ID was 2. I believe when I plugged the other camera in, then unplugged it, the order was changed and this threw an error. Setting it to -1 again let me pass the compatibility test and now that Unity isn’t crashing when I play I can find the colorstream ID again (1).