Do I have to have receipts to support costs I paid for, out-of-pocket, for books and materials? Would I be able to use a rough estimate?

Yes, you should have receipts.  You aren't permitted to deduct any expenses that you are unable to substantiate without adequate records or documentation. The IRS states when claiming educational expenses, "Estimates or approximations don't qualify as proof of an expense."

For purposes of the American Opportunity tax credit only, these items are deductible if they are required for your program. Per the IRS, "qualified tuition and related expenses include amounts paid for books, supplies and equipment needed for a course of study. You do not have to buy the materials from the eligible educational institution."

See Education credits: Questions and answers for more information.