Sweeteners are everywhere. In diet colas, sugar-free yoghurts, chewing gum - even in things you wouldn't expect, such as sauces and medicines. For years, the message has been simple: they're safe, they're healthier than sugar, and they'll help with weight loss.

Like many people, I once assumed the main downsides of sweeteners were fairly minor: they might upset your gut, causing bloating or diarrhoea, and they keep your palate hooked on sweet tastes.

But recent research has raised concerns about them, suggesting they can reduce insulin sensitivity - which means your body is less able to deal with sugar in the blood, so rather than helping with weight control, they could undermine it.

However, what was once a vague anxiety in my mind about sweeteners has become a determination to get rid of them from my family's diet.

And that is because of a study published in the journal Neurology this month which made me look at sweeteners in a completely different light.

Researchers monitored the diets of more than 12,700 adults in Brazil, specifically how much low-calorie and no-calorie sweetener they consumed. They also tested the participants' memory and brain function after four years, and then eight years.

The results were striking. In adults aged under 60, those who regularly consumed sweeteners experienced a faster decline in memory and thinking skills. The effect was equivalent to their brains ageing an extra 1.6 years across the eight years.

Those who regularly consumed sweeteners, such as those in diet sodas, experienced a faster decline in memory and thinking skills

Amongst the over-60s, the pattern was similar. However, the data wasn't strong enough to show conclusively the link with sweeteners.

The results were even worse in the people with diabetes: sweetener users had about a 30 per cent higher risk of memory decline.

For me, that's it: I'm now trying to avoid sweeteners altogether, including not drinking any diet drinks (or full-fat versions - you can learn to love water when you focus on what's not been added to it). The idea that sweeteners are harmless is no longer tenable.

Of course, the sweetener industry has not taken this lying down. Trade bodies have already pushed back, arguing that the study is 'observational', and therefore can't prove that sweeteners caused these problems.

Amongst their other arguments, they point out that global health authorities, including in the UK, EU and US, have repeatedly declared sweeteners safe.

But I don't think their arguments stand up - some of their points are technical, about how the research was done.

So let me focus on a key complaint about the new research: that the study is only observational, it didn't prove that sweeteners caused memory decline.

But to prove sweeteners were the cause we'd have to give some people sweeteners for decades, then compare them with people who didn't have them. That's simply not possible - or ethical if you're investigating something that might cause harm.

It was like this with smoking. There was never a trial where half the population were made to smoke for years and then compared with people who didn't smoke. Instead, we relied on observational studies to establish that smoking is harmful.

For decades the tobacco industry hid behind the line that this only showed 'association, not causation'. But there came a tipping point.

Nutrition science works in the same way: when an association is strong and it's repeated across groups of people, and if biologically plausible - as it is with sweeteners - it's a warning signal we ignore at our own risk.

Yes, regulators have deemed sweeteners safe. But those decisions were based on toxicology and short-term studies on digestion and excretion, not on long-term outcomes such as cognitive decline. The absence of evidence on brain health was simply because nobody had looked. This study fills that gap.

So why might sweeteners affect the brain? One route is through the gut microbiome. Studies in both mice and humans have shown that sweeteners such as saccharin and sucralose can alter the balance of gut bacteria.

We're also trying to wean my three-year-old son off the cartoon-covered yoghurts he adores, which hasn't made me popular

Professor Rob Galloway is trying to wean his family off artificial sweetners

Those changes, in turn, affect how we handle glucose.

One landmark trial, published in the journal Nature in 2014, showed that healthy volunteers who were given saccharin developed poorer blood-sugar control within just a week.

Sweeteners may also interfere directly with insulin sensitivity - studies show they can lead to higher insulin levels, which causes increased fat storage.

That's why we haven't seen a reduction in obesity despite diet drinks becoming more popular over the past few years.

And we know that, over time, high insulin levels can damage blood vessels in the brain, reduce energy supply to neurons, and accelerate the kind of changes we see in dementia.

Sweeteners also deliver an intensely sweet taste without calories. That mismatch seems to confuse the brain's signalling systems, keeping our appetite for sugar switched on.

Studies on animals show that chronic over-activation of these pathways is linked to changes in the chemical messengers in the brain that influence memory, mood and cognition.

There are other possible routes for sweeteners to affect brain function, such as inflammation.

And while individually each of these lines of evidence might not be enough, together they build a picture: sweeteners alter metabolism, gut health and brain chemistry in ways that we know are tied to dementia. That is what scientists mean when they talk about biological plausibility.

Read More The hidden vaping side-effects no one is talking about - and how you can quit for good

So what should you do? If you occasionally have an ultra-processed yoghurt or diet drink, or slice of mass-produced cake, I wouldn't worry. The risks relate to regular, long-term intake.

The idea that sweeteners are a healthy alternative to sugar is now in doubt.

You see this pattern often in medicine. Big business finds a clever 'solution' for a problem - margarine instead of butter; vaping instead of smoking; sweeteners instead of sugar.

At first, it looks like progress. Then, years later, we discover the unintended consequences. Nature has a way of humbling us.

We didn't evolve to eat artificial sweeteners, so it's no surprise they can cause harm.

My family shopping trolley no longer contains diet drinks -much to the annoyance of my daughter, who used to drink no-sugar cola by the bucketload - and we're also trying to wean my three-year-old son off the cartoon-covered, artificially sweetened yoghurts he adores, which hasn't made me popular.

But the lesson is clear. If you want to look after your brain, keep your mind as sharp as possible and reduce your risk of dementia, don't lean on artificial shortcuts.

The safest drink for your health remains the simplest - water.

@drrobgalloway

Hidden side effect diet colas have on your MEMORY: DR ROB GALLOWAY