New Post: Delete a non-selectable folder

I have a legacy app that stores messages in IMAP with a unique folder structure. Each instance of the app logs in as one particular IMAP user, then each actual user of the app has his own subfolder containing his messages.
INBOX/userA/Inbox contains userA's Inbox
INBOX/userA/Sent contains userA's sent items
INBOX/userB/Inbox contains userB's Inbox
etc, you get the idea.

There's a bunch of empty folders from users who have never actually used the messaging feature, and I'm trying to clean them up. I can delete the bottom level folders just fine (i.e.)

client.Folders["INBOX"].SubFolders["userX"].SubFolders["Inbox"].Remove()

will correctly delete the Inbox folder, but

client.Folders["INBOX"].SubFolders["userX"].Remove()

will throw an error saying that folder is non-selectable and therefore can't be deleted (even after Inbox, Sent etc have all been removed).

Is there anything I can do to delete the non-selectable folder once it's empty?