Prince Harry's children could 'face difficulties' in later life meeting royals