We stopped trying to rent a residence to prepare it to sell. Is the cost to paint it an increase in basis or an investment expense?

Assuming that you rented it for any period of time in 2016, it's classification as residential rental real estate does not change just because you are preparing it for sale. The only way it's classification changes, is if you convert it to personal use and live in it as your personal residence, or use it as a 2nd home. When you sell it, you will report the sale as the sale of residential rental real estate.

Note that your painting and stuff are most likely not a rental expense and it's not a capital improvement either. Much of that is part of your sales expenses - but I don't think painting is. Here's how the different expenses are classified for rental property.

• RENTAL POPERTY ASSETS, MAINTENANCE/CLEANING/REPAIRS DEFINED

Property Improvement.

Property improvements are expenses you incur that add value to the property. Expenses for this are entered in the Assets/Depreciation section and depreciated over time. Property improvements can be done at any time after your initial purchase of the property. It does not matter if it was your residence or a rental at the time of the improvement. It still adds value to the property.

To be classified as a property improvement, two criteria must be met:

1) The improvement must become "a material part of" the property. For example, remodeling the bathroom, new cabinets or appliances in the kitchen. New carpet. Replacing that old Central Air unit.

2) The improvement must add "real" value to the property. In other words, when  the property is appraised by a qualified, certified, licensed property appraiser, he will appraise it at a higher value, than he would have without the improvements.

Cleaning & Maintenance

Those expenses incurred to maintain the rental property and it's assets in the useable condition the property and/or asset was designed and intended for. Routine cleaning and maintenance expenses are only deductible if they are incurred while the property is classified as a rental. Cleaning and maintenance expenses incurred in the process of preparing the property for rent are not classified as cleaning/maintenance costs. They are instead classified as startup costs, amortized as such and depreciated over time.

Repair

Those expenses incurred to return the property or it's assets to the same useable condition they were in, prior to the event that caused the property or asset to be unusable. Repair expenses incurred are only deductible if incurred while the property is classified as a rental. Repair costs incurred in the process of preparing the property for rent are classified as startup costs, amortized as such and depreciated over time.

Startup Costs

Please note that if residential rental income is not your PRIMARY business, and your PRIMARY source of income, then your rental business is considered to be passive, and you flat out, no way, no how , are not allowed to deduct your startup costs. Period. The IRS says so. See https://www.irs.gov/pub/irs-drop/rr-99-23.pdf and please take note that rental property produces “passive” income, while other types of businesses produce “active” income. Your rental property is not classified as your “active” business, unless you are a real estate professional, an active participant in the management of the property, and it provides a substantial (more than half) amount of your taxable income for the year. All three requirements must be met. There are no exceptions

Start up costs are expenses incurred while preparing the property for rent, with the express purpose being to prepare it for rent, before it is available for rent. These costs do include repair, cleaning and non-recurring maintenance cost. It does NOT include property improvements. With a normal business that produces active income (rental income is passive) you would amortize these costs over 15 years. But you can’t do that with a rental property. However, you can deduct a maximum of $5000 in startup costs in the first year the rental is available for rent, PROVIDED your total startup costs do not exeed $50,000. This is reported on line 18, “Other Expenses” of SCH E, and should be labeled “start up expenses”.

Additional clarifications: Painting a room does not qualify as a property improvement. While the paint does become “a material part of” the property, from the perspective of a property appraiser, it doesn’t add “real value” to the property.

However, when you do something like convert the garage into a 3rd bedroom for example, making a  2 bedroom house into a 3 bedroom house adds “real value”. Of course, when you convert the garage to a bedroom, you’re going to paint it. But you will include the cost of painting as a part of the property improvement – not an expense separate from it.