I took a leave from work after having a new baby mid-2020. Can I take the self employed family leave credit if my leave wasn't directly due to COVID?

No, the reasons listed for the new sick and the family leave credits are all Covid related. The same rules apply to the self-employed.   Here is a link for reference.

The are no other family leave credits available for self-employed taxpayers.

The Family Leave credit is for periods of leave during which they are unable to work or telework due to a need for leave to care for a child of such employee if the child’s school or place of care has been closed (including the closure of a summer camp, summer enrichment program, or other summer program), or because the child care provider of the child is unavailable, for reasons related to COVID-19. 

The sick leave credit is for:

  An employee who is unable to work (including telework) because of coronavirus quarantine or self-quarantine or has coronavirus symptoms and is seeking a medical diagnosis. 

An employee who is unable to work due to caring for someone with coronavirus, or caring for a child because the child’s school or place of care is closed, or the paid child care provider is unavailable due to the coronavirus.