OK actually, this is not linked to gnome. Switching to kde led to the same issues. I found out that at random times, I get errors like such appearing, which are linked to the nouveau driver:
[12214.906574] nouveau 0000:65:00.0: gsp: mmu fault queued
[12214.975649] nouveau 0000:65:00.0: gsp: rc engn:00000001 chid:8 gfid:0 level:2 type:31 scope:1 part:233 fault_addr:0000003ff3dd0000 fault_type:00000002
[12214.975669] nouveau 0000:65:00.0: fifo:000000:0008:0008:[Xorg[3024]] errored - disabling channel
[12214.975679] nouveau 0000:65:00.0: Xorg[3024]: channel 8 killed!
This is apparently linked to Nvidia metadata error : I need to install the nvidia-driver. But if there is a solution for this, I will happily take it!