Twice I negotiated with a coach, and when they asked for a contract extension, I agreed, clicking the appropriate button. But then they both quit during the next staff hiring phase.

Once the contact extension was accepted before the staff hiring phase (at very beginning of the year) and once the contract extension was accepted in the first round of the staff hiring phase.

Am I doing something wrong, or is this a bug?

I'll have to check into it. The coaches are supposed to leave during the "end season/begin new season" phase when their contract ends. The new year flip is what updates their contracts to the ones they signed during the previous year. It could be trying to sign extensions before the season starts (or at least after the hiring phase) could be conflicting with signing/rotating new coaches.

In one new season, an assistant coach with a couple of years remaining on his contract switched to a different team.

Yea, that's normal. Coaches can get poached from teams either for money, prestige, job status, etc.

So you can fire a coach mid-contract without penalty and coaches can quit mid-contract without penalty?

yes