How do I report a failed business losses/expenses on my tax return?

It depends. If you are trading part-time, investment expenses aren't deductible on your federal return, but some states allow you to deduct them on your state return.

From IRS Pub 529:

Investment fees, custodial fees, trust administration fees, and other expenses you paid for managing your investments that produce taxable income are miscellaneous itemized deductions and are no longer deductible.

To allow the expenses to flow to your state tax return, enter the amounts under Federal >> Retirement and Investments >> Other Investment Expenses.

If you are in the business of trading full-time, you would be able to deduct expenses The IRS is VERY strict on the Trader in Securities rules.  See Topic No. 429 Traders in Securities.

A trader eligible for trader tax status can deduct business and home-office expenses and make a timely Section 475 election on securities for tax loss insurance and a potential QBI deduction. Using an S-Corp, a trader can also deduct health insurance premiums and a retirement plan contribution.

The Mark-to-Market Election

A trader must make the mark-to-market election by the original due date (not including extensions) of the tax return for the year prior to the year for which the election becomes effective. You can make the election by attaching a statement either to your income tax return if filed without an extension or to a request for an extension of time to file your return. The statement should include the following information:

1. That you're making an election under section 475(f);
2. The first tax year for which the election is effective; and
3. The trade or business for which you're making the election.

Traders can choose to use the mark-to-market rules, investors can't. If a trader doesn't make a valid mark-to-market election under section 475(f), then he or she must treat the gains and losses from sales of securities as capital gains and losses and report the sales on Form 1040, Schedule D, Capital Gains and Losses and on Form 8949, Sales and Other Dispositions of Capital Assets, as appropriate. When reporting on Schedule D, both the limitations on capital losses and the wash sales rules continue to apply. However, if a trader makes a timely mark-to-market election, then he or she can treat the gains and losses from sales of securities as ordinary gains and losses (except for securities held for investment - see above) that must be reported on Part II of Form 4797, Sales of Business Property. Neither the limitations on capital losses nor the wash sale rules apply to traders using the mark-to-market method of accounting.

(Summarized from this 2021 post.)