Similar thing happened to me, i contacted my legal team from my insurance company, who told me anything placed on my land belonged to me and that i had the right to remove it and i didnt need to give the neighbours anything back ( in my case it was a gate). i presume as long as you can prove the fence falls within your boundary you could do the same. hope this helps