We don't have Google Voice in my country so I'm going on what I can find. Is the linked number used in some way or was it only to verify? I've seen some people who appear to have removed it. I can't say if this works out not so maybe it might be worth asking in the Google Voice support pages or on Reddit with your problem to see if they can confirm our suggest a solution.

https://support.google.com/voice/thread/22480467?hl=en
https://www.reddit.com/r/Googlevoice...linked_number/