Should I be owing NJ taxes if I live in NJ and work in NY, but my employer is not registered in NJ, thus only withholding NY state taxes on my paychecks?

The general rule is that you owe taxes to the state where you live and to any states where you physically worked (there are, of course, exceptions).

You certainly owe NJ taxes because you live there, and NY taxes because you work there.

It does not matter for whom your employer withheld taxes - that does not determine your tax liability.

If your employer is not required to withhold NJ taxes (perhaps this is why they choose not to register there), you are still liable for them.

It sounds like your NY employer's withholding was not quite sufficient to cover your tax (remember that withholding is just an estimate of what you owe and that you can adjust it if you want), and that you didn't owe any tax in NJ because of the taxes you paid to NY.

Puzzled about how you get money back from NJ when you didn't withhold anything for NJ? NJ has some refundable credits, that is, credits that refund you money over and above your tax bill. This may not be the reason on your return, but I see that the NJ Property Tax Credit has a refundable amount of $50.

So I can't declare that your returns are perfectly correct (not having seen them), but what you have presented is perfectly plausible.