He dado con el problema. Iba ya a darlo por perdido cuando me ha dado por quitar el DLSS 4.5 que habia activado desde un principio. Solucionado. La VRAM no sube a mas de 9GB-10GB, se anda reciclando en todo momento como deberia. Si pongo preset L o M, vuelve el problema de manera super obvia, si uso K, de la misma manera se va.
Sabiendo esto he ido a buscar "DLSS 4.5 Starfield" y sin falta de añadir algo como "fps drop" o algo asi a la busqueda:
Just to give a bit more context: this issue happened on my end (RTX 5090 + 7800X3D) after using the newest DLSS presets — specifically L and M. As soon as I reverted back to preset K, the problem disappeared completely.
So, just a heads-up: if you’re using preset L or M, especially on RTX 5000-series GPUs, it’s worth double-checking whether this issue shows up on your system.
Another important note: if you’re a Frame Generation user, I strongly recommend updating the Frame Generation DLL via DLSS Swapper to version 310.5.0. After doing so, I noticed fixes for some bugs, including audio muting during planet-to-planet ship traversal (an issue that occurred with Frame Generation enabled).
Also, keep in mind that the problems with presets L and M occurred specifically on a 5000-series GPU. So far, users on 4000-series GPUs haven’t reported the same behavior with these presets. 
