How do I know what I can write off as an unpublished author? And do I have to set up a sole proprietorship to write off writing expenses?

You can deduct/write off all the ordinary and necessary expenses required to work as an author (published or unpublished). Generally the expenses for the author would include supplies, internet, subscriptions, mileage, phone, software, home office, advertising, taxes, insurance, education, etc. This is not an exhaustive list and many of these expenses may not apply to you. You just have to keep good records of all the expenses that you are incurring to perform your business. 

To write off these expenses, you can be a sole-proprietor or any other business form (like LLC, partnership, etc.) If you are not registered as any of the business entity, your income and expenses from this work will show as self employment income of sole-propritor in your personal tax return ( being a sole-proprietor you do not need to file a seperate business return).