Section 7. Any person so notified shall be allowed until twelve o’clock noon of the day following the service of the notice in which to begin to remove such structure or make it safe, or to make it secure, and he shall employ sufficient labor speedily to make it safe or remove it or to make it secure; but if the public safety so requires and if the aldermen or selectmen so order, the inspector of buildings may immediately enter upon the premises with the necessary workmen and assistants and cause such unsafe structure to be made safe or taken down without delay, and a proper fence put up for the protection of passers-by, or to be made secure. If such a building or structure is taken down or removed, the lot shall be levelled to uniform grade by a proper sanitary fill to cover any cellar or foundation hole and any rubble not removed.