Rae was born in Leeds, West Yorkshire to a Kittitian father and an English mother as the eldest of three daughters. Her sister is actress Rhea Bailey. She used to be racially abused on a regular basis. She says about the situation, "My sisters and I were different and people used to say, 'Ah, aren't they cute, the little chocolate children', and 'look at their hair'". "I know they were only being cute but it was over the top. Then people from other schools would shout, 'Paki' (a commonly used racist term used in the UK) to me because it was the most common racist insult of the time. 'At least if you're going to do it, get it right,' I used to shout back, 'But my dad isn't from Pakistan, he's from St Kitts—so there!'"

Rae began her musical career at school where she studied classical violin before she turned her attention to singing: "I started off singing in church, I suppose, but people think it must have been a gospel church because of the whole, you know, black assumption