I think many "diets" fail because they are viewed as a short term solution to be abandon once a person reaches a certain goal weight. Which leads to people falling off of them, because the mentality sets in "because this is short term anyway, I'll just end the term a little early..." or similar type of thought.
I think the key is to tell yourself this is a lifestyle change, not a temporary measure. Once you can wrap your head around this is a permanent change, it's easier to stick to.
The other aspect is different diets accomplish what they do in very different ways. Jumping from one concept to another concept (as many people do) ends up giving your body contradictory signals.
And then, everyone is different. What exactly works for one person may not work for some else's metabolism.
The biggest issue is when faced with a elimination type diet (i.e. a plan that is focused on not eating certain items) it can be temping to over indulge on acceptable items.
For years and years I've eaten a low fat diet, with no red meat, lots of veggies and not to much sugar. But over time I still added weight because I over indulged on carbs (bread, potatoes, pasta) because "it's not fat" and the reality is to many carbs are just as bad as too much fat.
I also started riding a bike to work. So I started to say "it's OK to eat this doughnut, because I rode my bike today" but then reality is replacing and exceeding all the calories I burned doesn't lead to any weight loss.
For myself, I am not on any "diet" per se. I just made a choice to be better. Step one was to eliminate as much sugar (both real and artificial) as possible. No more sugar in my coffee. Rum&coke is replaced by small amount of rum with club soda. No more doughnuts or cakes or whatever other garbage is being served around the office.
My biggest thing was figuring out breakfast. For myself, I just was never hungry first thing in the morning. So I'd bike to work. Now I've arrived at work on an empty stomach, and I'd want to eat everything in site. And it didn't matter how many "healthy" breakfast choices I'd keep stashed at my desk, I'd eat them all!
So I turned to green smoothies. It doesn't matter if I'm not hungry, because it's a beverage.
Some kale, almond milk, a tablespoon of wheatgrass powder, and handful of assorted frozen fruit, and I'm good to go. I've experimented with different items to add in to the smoothies. Some work, some don't.
I also experimented with gluten free. It did nothing for my waist, but my asthma and allergies went away, and my wife says I stopped snoring.
Between the smoothies, no refined sugars, and gluten free, I have way more energy to work out in morning, which has lead me to being in way better overall shape. I wish I had started doing this years ago!