Section 54. The mayor may remove the head of a department or member of a board by filing a written statement with the city clerk setting forth in detail the specific reasons therefor, a copy of which shall be delivered or mailed to the person thus removed, who may make a written reply, which, if he desires, may be filed with the city clerk; but such reply shall not affect the action taken unless the mayor so determines. This section shall not apply to the school committee, or to officials appointed by the governor, or to assessors if elected by the people.