Capitao This is something that our team is routinely looking at, and I can assure you we've looked at tens of thousands of sites and the specific tradeoffs between speed and overall user experience. You are correct, it is an indicator of page experience, but even Google has walked back the significance of it recently. Check out this exceprt from an article about the topic - https://www.searchenginejournal.com/ranking-factors/are-core-web-vitals-a-ranking-factor/:
Fast forward to April 2023, when Google “simplified” its guidance around page experience signals in search rankings.
While Google still recommends maintaining strong CWV for a good user experience and still uses these metrics as ranking signals, improving one or more metrics does not guarantee better rankings.
Page experience continues to be a ranking factor but is evaluated holistically by Google’s core ranking systems rather than as a single “page experience signal.” Mueller also updated Google Search Central audiences on YouTube in July 2023 about INP coming to CWV in 2024.
To be candid, you're making assumptions that core web vitals are the end all be all for ranking a site, which isn't true. At the end of the day, every website owner will need to make decisions on what is important to them. Is it more important that I focus on good content, or make my site as fast as humanly possible? Is a specific plugin on my site that is slowing it down useful enough that I want to keep it running on the site? These are decisions you will be faced with when running a website.
My strong recommendation is to look at the user experience metric for your site and see if there is a negative impact on that end vs. this specific single metric. I'd also encourage you to check out some of the biggest publications out there, and how their core web vitals look. I think you would be surprised at the wide ranging scores for core web vitals.
This isn't to dismiss the concern - I can empathize that it keeps you up at night because at the end of the day its a tangible score that Google has created, and it's 'easier' to try and complete this than it is to write good content because that is arbitrary. But I have seen the same concern many many times (and is the whole reason why LEAP was created in the first place), and the amount of time spent trying to improve this almost never actually moves the needle from a search rank perspective because It's one aspect of the website and not fully representative of the users expereince.