I have this problem myself. One thing I'm curious about that you didn't mention however, is when you woke up, about how much sleep did you have? With me, I notice that I tend to be unable to hold the dream because I'm close to the end of my REM Cycle and have been at the point where I'd normally wake up. Because of that, the dream fades quickly and I wake up, even if I use the various techniques you've described above. I've tried all those myself, spinning, rubbing hands, shouting outloud, and more.

I only ask because you might have the same problem, and may simply need to get Lucid earlier in the dream/night. It's something I'm experimenting with myself, trying to become Lucid early on (through WILD), to see if that helps maintain the dream stability and helps it last longer. I also think this has something to do with it because your mind prepares to become more active the closer you are to waking up, which when combined with being conscious already, could force you to wake up.

This is just speculation on my part, but it's something I'm trying to experiment with.