The Affidavit of Support is a contract between the person who filed it and the U.S. Government. It is not a contract between you and your ex-spouse. It is also not a contract to provide you with support.
The Affidavit of Support is essentially a means by which the Government can seek reimbursement from the person who filed the Affidavit of Support for the costs of any of the designated Federal, State or local means-tested public benefits, if the person for whom the Affidavit was filed receives them. Therefore, only the government can enforce the Affidavit of Support, and only if you are receiving benefits in one of the designated programs.
In a divorce situation, any support to be provided by an ex-spouse, if any, is governed by the orders issued by the Court granting the divorce.