Hi Jens,
You have my vote. And to support this idea, there are some SAP notes that here and there talks about that scenario impossibility, like note 1484558 that states: "This occurs during the check of work address duplicates which is not required for a person to person contact person relationship, since the work address assignment is not possible for this scenario."
Kind regards and have a nice weekend!
Garcia