Section 3Y. In any city or town which accepts the provisions of this section, inspector of buildings, building commissioner or local inspector so employed and designated under this section shall be appointed for a minimum term of three years and such three year term or more shall not be diminished by an unexpired term of a predecessor, and he shall not be removed from office during his term except for just cause and after a hearing has been held on charges by and before the appointing authority.