Deductions reducing refund

One probable reason is that some income-based credits such as the Earned Income Credit will decrease when you enter more rental expenses and decrease your taxable income.

The Earned Income Tax credit (EITC) is a bell shaped curve against income. The EITC rises with income until a certain level of income when it starts to decrease to reach 0 at the maximum level of income for the category.

As an example, please look at the EITC tables from page 27 of this IRS publication.

Please note that the law requires you to deduct all allowable expenses and not leave out some to increase your income-based credits.

Be aware, that if EIC is the problem, you are required to report all your legitimate expenses even if they reduce your EIC.  Omitting some expenses to get a higher EIC may be considered tax fraud.

On the other hand, if you are entering the business first, and will enter other income later, you may not be eligible for any EIC, and you are only seeing this effect because you entered the self-employment first (when you had no other income listed in the program) instead of last.

Also lastly, if you lost money on the business, that loss can be carried forward to a future tax year to offset any future profits.  So you don't lose in the long run, and you should enter all your expenses.