There's a bunch of reasons why an army of 20 thousand is not twice as effective as an army of 10 thousand. And the main one, especially in a defensive situation, is that there's only so much space for them to fit, so much room for them to maneuver, so much for them to actually do which wouldn't hinder more then help.
Visual cue, the Battle of Bastards in the latest Game of Thrones season. There were some skirmishes, some maneuvers, and they all required room to move. Even the idiotic lone charge by Snow required room, if he'd have been cheek to jowl in the mess he could barely have started to walk out of sync, let alone run.
Picture a warehouse, with a large roller door. Sure, you could have 10 people walk through it holding hands, side by side. Now throw some pallets in the middle so it's not level, stacks of wood to make the doorway smaller, have a fire alarm going off, with smoke from the fire. Situation like that, not unlike a pitched battle, you'd be lucky to get 3 people through at the same time, and they'd certainly not be a problem for a small group of defenders to pick off with spears.
Back to GoT, remember when a large fraction of the Northerner force is trapped in the middle of a circle of shields? And how the character routinely acknowledged as the best swordsmen around, highly trained in personal combat, a very important and noticeable character, nearly dies smothered by corpses, without a single blade touching him? When 3000 of your troops are Joe Muggins, recently retired from dung farmer, he's not going to fare well either, when push comes to shove.
5000? 5000 bandits (defining bandits as "lightly armed, lightly armoured, lightly trained, lightly cohesive raiders who want easy loot, not to attack a solid defense) would be eviscerated. 1000 defenders could sally forth, and annihilate them. Rome Total War is a good demonstration of this, Roman heavy infantry vs gaellic light infantry, even a 5:1 ratio on open plains easily results in a Roman victory. Add in confusion, boredom, sitting outside the castle walls starving for a few weeks (unless they routinely carry around a month of food for each person, they'll be on bark rations quickly. And, go to the fridge and pull out what you'd like to eat for a month. And a fridge to keep it fresh in. And a generator). Increase the numbers, just makes it messier for the attacker. 50 thousand versus 1 thousand? Yeah, if they throw 30 thousand away on constant attacks. You ever signed up for a job where to get a paycheque, you had a 60% chance of dying in a field?
Not least, the biggest problem, is the biggest problem. A force of 5000, isn't a bandit force, it's not raiders, or reavers, or nothing. It's an army. And when an army of 5000 comes knocking, they don't move fast, and they telegraph where they are going. And then, the defenders friends with 15000 meets them at a bridge.
In short, you *might* get away with being a bandit company of 100. Provided you keep in mountainous areas, or forest, anywhere where it'd be too much of a pain to send in the troops, and no idea if they'd get you. After all, the best defence in this situation, is to not be where the enemy army is - they kinda take an axe to problems like bandits. And bandits want to have a good chance of good treasure, good drink, some excitement, and to live to do it again tomorrow. If they don't know and respect (or fear) their leader, and think there's a good chance he's watching at any given moment for deserters, then why would they stick around?