I have a 1099-R form but it should not be considered taxable as it was transfer to another institution. The software should ask me if it was transferred but it doesn't.

You should re-enter your form and answer the questions that follow carefully. Once complete, continue out of the section and you'll be asked more questions that can impact the tax calculation. 

To do this, since it sounds like something may have been entered incorrectly, I recommend deleting your form and re-entering it.  To do so:

1. Login and click Take me to my return
2. Under 2017 Taxes on the left menu, select the Federal
3. Select the Income tab along the top
4. Scroll down to Retirement Plans and Social Security section
5. Click Update or Revisit next to IRA, 401(k), Pension Plans (1099-R)
6. You'll be at a listing of the 1099-Rs you've entered. click Delete to remove your prior entry. 
7. Click Add Another 1099-R or begin entering your form again. 
   
8. Answer the questions that follow carefully. 
9. ***When you've finished entering your form and are back at the summary screen of Form 1099-Rs, be sure to click CONTINUE. In some cases, this is where you will be asked about information that will change your taxability.

If you did a 'backdoor Roth' (contributing to an IRA and then converting to a Roth in the same year) be sure to enter your non deductible contributions, which will offset the taxability of converting to a Roth. 

Review or enter your current year contributions under Deductions & Credits >>> Retirement and Investments >>> Traditional and Roth IRA Contributions.