Matt and Margareta's dreams of converting a former dairy have been thwarted by planning permission problems.