Apparently he wasn't ready to settle down when they first met, so she went off and married Andrew Parker-Bowles. I'm not sure but I also think she might have been considered 'unsuitable' by the Royal hierarchy as she was a bit of a party girl in her day (wouldn't know it to look at her now, would you?)