How does having to pay self-employment tax affect how much I'm taxed for social security and medicare?

The self employment tax is just to pay the Social Security & Medicare.   

The self-employment tax associated with your clergy housing allowance is added to your income tax on your Form 1040 and paid in to the IRS.

The housing allowance would not affect your taxable income as reported on your tax return, so it will have no affect on the portion of your social security benefits that are taxable. It will result in your housing allowance being added to your income as reported to the social security administration and as such may result in you receiving a bigger social security benefit upon retirement.

As mentioned by @VolvoGirl , the self-employment tax you pay in goes to the social security administration just as does social security/Medicare taxes that are withheld from your wages.