Castle is your settlement`s heart. Upgrading your castle will give you more builders and faster building speed. After upgrading your castle to level 15 you will be able to destroy your settlement`s buildings.