At this stage, you cannot set up any specific privacy settings regarding who can and can't see the items in your PAM Village - all items are fully visible to everyone in your Village.
For now, you can assign items to specific members in your Village and use the filter on the top right of the calendar screen to filter items assigned to a specific person.
You'll improve in this area by the first half of 2026.
If you're importing data from Google Calendar or Outlook Calendar you can choose if the events imported are visible just for you or the whole account.