I wrote a long winded response earlier but it seems to have disappeared.
Yes, you can fit a 980M, 1060 or 1070
You need a modified driver, new thermal pads and thermal compound. I do not belive you need an unlocked BIOS for the 980M.
You are best to also upgrade the CPU to a 3840QM or 3920XM but you will probably need a new dual pipe heatsink.
Head to the Alienware forum at NotebookReview more more help as there are still a few of us using and working on these old machines.