While in the Action Plan  Manager, select your plan and in the “action Plan Details” window, click on “Manage Assigned contacts” 

In the “Action Plan Contacts” window, find the contact to remove

Click on the “Remove” button beside their name.