Alittle misleading topic here, but still, if i understand you right, you want to delete/remove the button you already has created in Outlook.
Just remove the button in the same add-in, when the explorer/inspector you added it in closes. When you have written the code to remove the button here, build your add-in, and run it. This will correct your problem, deleting the button on close. When you then uninstall the program, the button won't exist. (if you temporarly want to disable your add-in, change the loadbehavior to 0).
It's good practice to release all the objects been used in the code, before terminating. This way you will prevent Outlook from "hanging" when you close it. (it looks like it's closed, but the process is still going, and sometimes has to be closed by killing it manually).
I'm new to outlook development, so if someone mean this is wrong, please correct me