Brian,
You can do so with a custom action. In this custom action, you will ask for the user you want to remove/delete. Then, in the "Set Actions" section of the rule, you can use the "Delete Object" option and specify the following:
if(not(IsNull(GetLink('IssueTeamMembers',CurrentObject(),TeamMember))),GetLink('IssueTeamMembers',CurrentObject(),TeamMember),null)
The above code will only delete the user if he is, indeed, assigned to the team.
Hop on an Advanced Q&A Webinar for more information!
Rob