You can stop being paranoid about this. No one is monitoring the activities in your Outlook folders. A vCard is a text file that contains information about a contact. You can see this for yourself by opening it in Notepad. It contains no information about activities. It contains no information about activities.
The Activities page builds a list *on the fly* of items related to the current contact. It builds that list from the *current user's* Outlook folders. If someone has activities related to you, they'll see them when they open the vCard you sent, because Outlook creates one of its own contacts from the vCard. If you open that vCard yourself, Outlook creates a contact with your information on it, and that contact's Activities page will show all the activities in your folders involving you.
If you still don't believe me, save the vCard to your hard drive and open it in Notepad.
