Brotherly loan gone bad but I also have to pay tax on it?

If you don’t want to declare it as a bad debt and there’s a chance you will get paid there is nothing to report. Considering it a gift actually doesn’t cost you anything because it would be applied towards your estate gift allowance which is millions of dollars. But there’s no compelling reason to do that. 

(1) declare the loan as a bad loan and write it off. However, I think IRS want to see an effort to collect the loan. How do I show a collection effort without hurting him?

Do you have paperwork on this loan that would stand up in court if you were to pursue collections through legal means? If not, then it would be questionable if you could write it off as bad debt. If you do have such paperwork, then you would need to be able to show a valid attempt to collect. Most likely, someone's gonna get hurt in the process. (One reason I do not do business with family under any circumstances and with no exceptions.)

(2) report it as an on-going loan, because there's always a chance he would come upon some money and pay me back.

I would not bother unless it's a legally binding loan. Keep in mind that for such a situation there is "implied interest" that you would pay taxes on, weather you actually charged interest on the loan or not.

What paperwork does IRS require to support this filing?

As far as I know, paperwork would only come into play if you were audited.

(3) report it as a gift and pay gift tax since it exceeds the tax free gift amount. It sucks if I have to do this.

That's a viable choice. Keep in mind you're required to file IRS Form 709-Gift Tax Return only if the amount given in a tax year to any one individual exceeds $15,000.   Even if it does, no taxes are due provided the gift does not exceed $11.7M.  Understand that $11.7M is a lifetime limit, not a yearly limit. Although that increases to $12.06M in 2022.  Also you're really doing when filing a 709 is meeting a legally required reporting requirement.