Can all adjusted qualified expenses be capped at $4,000 when claiming an AOTC or must the total count against a qualified 529 expense?

The total of the qualified educational expenses would go to the 529 plan first before it applies to the AOT. 

For example, your total tuition expenses on Form 1098-T are $11,000, your distribution amount on 1099-Q is $5,000.  You would enter the 1099-Q first, then the Form 1098-T.  Since the entire distribution is used to pay for educational expenses, your 1099-Q is a nontaxable event.  After applying to the 529 plan, you have $6000 (1,1000-5,000) left to apply toward the AOT.  As the maximum allowable expenses amount to claim the AOT is $4,000, you get the full AOT credit of $2500.

However, if the difference between the distribution and the tuition amount paid is less than $4,000, the program will prompt you to a screen where it asks you the difference between the two sources.  The program will then calculate your share of the education credits on Form 8853 Education Credits.  For example, your 1099Q distribution is $3,000, Form 1098-T tuition is $5,000, you would then enter $2,000 on that screen.  The program will calculate your AOT based on the $2,000 instead of $4,000.  See the image below:

The adjustment will not exceed $4000, because $4000 is the maximum needed to claim the maximum AOTC ($2500).  That frees up the rest to be used against the 1099-Q.