When a person has power, influence, higher position in society or a lot of money it is difficult for him/her to lose and to accept the fall
The bigger they are, the harder they fall
Examples
Don't envy him, Peter. It was very difficult to reach the power he has. And remember - the bigger they are, the harder they fall.