I have sold my rental property, how to treat mortgage prepayment penalty paid to the bank on sale of rental property?

The prepayment penalty for the mortgage payoff is not a qualified selling expense so cannot be used as a selling expense for the rental property.

In Rev. Rul. 57 - 198, the IRS permitted the deductibility of prepayment penalties as interest expense under Sec. 163.

also see issue #2 Conclusion 2

https://www.irs.gov/pub/irs-wd/0006011.pdf 

the amount would be deductible even on a personal resience.

from 1098 instructions

Box 1. Mortgage Interest Received From
Payer(s)/Borrower(s)
Enter the interest (not including points) received on the
mortgage from borrowers during the calendar year.
Include interest on a mortgage, a home equity loan, or a
line of credit or credit card loan secured by real property.
Do not include government subsidy payments, seller
payments, or prepaid interest that does not meet the
exception explained under Prepaid Interest, earlier.
Interest includes prepayment penalties and late charges
unless the late charges are for a specific mortgage
service.