Please check and see if the port is not used. If you have an application or device attached to the port then it will be impossible to remove it. Trying to remove a port while it is active could have a serious impact on your computer's functionality.
Close down the application and make sure the port is open then try to close it using your favorite application.