How is escrow not a scam?

The concept of "escrow" already annoys me. As a requirement for my mortgage, I have to put aside money to pay for my homeowners insurance and property taxes, into a special account managed by the mortgage company, plus some extra depending on the mortgage company's demands. I can understand why they have a vested interest in making sure these things get paid, since technically they do own the house; but it seems it operates under the assumption that I would not take responsibility for these payments, and it gives me no opportunity to prove otherwise. Instead, the mortgage company gets to take the money and earn the interest, or dividends on the investments, and I have to just trust that they will make the tax and insurance payments (that I am responsible for) on time.

(To their credit, I have not yet had a mortgage company fail to pay either of these things on time.)

But recent events make me even more ticked off at the whole racket. I have a homeowners insurance claim in progress for storm damage (nothing too serious, mostly hail damage on the roof; we had the same thing five years ago). If the check is over a certain amount, the insurance company makes the check out to us and the mortgage company, requiring endorsement from all parties before the money can be used. I don't fully understand this. The insurance policy is a contract I have entered into with the insurance company. The mortgage company did not co-sign or endorse this contract, outside of requiring me to have a contract that meets whatever minimum requirements they had to allow me to get the mortgage. The mortgage company does not put any money into this contract — while they do make the payment of funds every year, they do so out of money they took from me. So why should they have a claim to the money on the insurance policy I paid for?

Still, before now, it was just a hurdle I had to jump before I could start repairing my home.

This time around, it's worse. Because the insurance payment was over a specific amount, my mortgage company demands I endorse the check and send it to them, and they will deposit it in a separate escrow account, and only give me money from it when I turn in receipts for repairs.

This is ridiculous. For one thing, it means I will be paying money out of pocket first. I don't have thousands of dollars sitting around that I can "float", paying for the repairs and waiting for the mortgage company to validate the receipts and give me my money back. That's the reason I have an insurance policy, so I don't have to pay out of pocket for catastrophes. For another, if my insurance company overestimated the damage amount, or if I am able to find a contractor willing to make the repairs for much less than the amount the insurance company gave me, how do I get that money when I don't have a big receipt to cover it? (The fact that the insurance company may have overpaid me is a discussion between me and them alone, since we are the ones that hold that contract.)

And this is just on top of my initial annoyance with escrow, that I'm not able to so much as put that money in an interest-bearing account to make a lousy 1‰ until I get the repairs done.

Of course, the answer is going to be so that they know that damage to their property is repaired, because otherwise the value of their property will be lessened by my negligence, while I could pocket the money and run. Still, this should be something I should prove first, let them sue me for negligence later. I get weary fast of having to pay for someone else's mistakes, and it seems like that's what is happing now: because irresponsible people exist, I cannot be trusted to do the right thing.