As far as stories go, It's been a long while since I've read their works and the main ones that come to mind from Poe are "Nevermore" and "Telltale Heart" (I know, I'm a normie). The first features a man in grief over the death of his loved one, and is confronted about his grief by a talking Raven. Now Poe never establishes if this should be plausible in the scope of the story, but I think the idea is sound: Represent degradation of the mental state by showing outlandish things seriously. A talking bird is goofy, but the gravity with which the author treats it makes for a very interesting event. I can see it being implemented into a CYOA via introducing strange/outlandish concepts (talking animals being one example) but never explicitly saying whether these concepts follow the rules of the world you've created or are just a figment of the characters imagination. This can make the reader second guess themselves and their actions, forced to figure out what is real and what isn't, and make choices accordingly. You could go a step further and never even conclude if the MC was crazy or not, leaving it up to the reader.
Two other examples of media I can think of that involve mental problems are "A Beautiful Mind" and "Fight Club". Only watched the former, but I loved how the twist gives a whole new context to the entire movie. The movie is actually about Schizophrenia but I feel you could apply the same to insanity (hallucinations, assuming the world strange world around you is real). Something like that is probs your best bet, imo.
I've got the Necronomicon on my desk, but that thing is way to imposing and I don't feel like diving into it. Lovecraft does focus on existential horror, and the story generally ends with the character going insane, so probs not that great for a story focused around an insane MC.
On to the actual questions,
a) Finding a reason for character going insane
I feel everyone else has covered this in greater detail than I ever could. I would like to point out that you don't have to make it obvious to the reader that the MC has gone insane, instead just giving foreshadowing so they could pick out that they were insane in repeat playthroughs (it was so obvious all along/ how did I miss that/ect). Being Sci Fi, you can make up whatever reason you want. Just make sure to decide whether going insane is going to be your major selling point or a surprise to the reader. You can't have both (unless you use some tricky red herrings).
b) Depicting the transition to a state of insanity.
Either understate it and make it a shocking reveal later on or play it out for all its worth.
Regarding the latter, it could make for a fun game where the MC hallucinates constantly and has to use an understanding of the world around him and clues given to him to identify what is real and what is not. I don't know if it could make for a serious story, but if you do decide to go this way, you need to clearly state to the player that things are not as they seem. As in most cases of showing insanity, making the player see things that aren't there or have sudden outbursts at seemingly inconsequential things would probs be the best way to go about it. Might get annoying tho...
If you're going for the former, oh boy. So many ways.
You could pull the same stunt as "A Beautiful Mind" did, and have the character imaging the driving entities of the story (ie hallucinations that the player doesn't know are hallucinations), then have the stories shift gear into another arc once the MC is aware of their insanity, still fighting to come to grips with it.
You could go for a "Fight Club" or "Jekyll and Hyde" scenario, where the player controls (or the story focuses on) two MCs and it's later revealed that they were the same person all along or one was a figment of the other's imagination.
You could do a "There Will Be No Peace With The Furies"-like and literally have the player interact with their emotions, fist fighting them, making deals, and having conversations with their 'ghosts' (eg the MC, anger, rational, and empathy are all interrogating a suspect. The suspect only sees the MC, but the MC sees his emotions moving around the room and giving advice to him... and in the corner of the room anger is furiously chugging a protein shake).
Insanity is so broad a term in fiction, doubly so for sci fi, that you could tackle this whatever way you like. It really depends on the mood of the story you're trying to tell, how many protags you're using, style of writing (I think you mentioned 1st person?), and generally the themes you're trying to convey.
Maybe providing more details on the type of story you're writing would help people give you more appropriate feedback for the genre you're writing for/mood you're going with. I myself can't really offer up much more in the way of feedback, but I feel others could help you out more if they know whether this is going all grim-dark or is just good fun (sorry if you've already mentioned this in your original post, maybe I've forgotten about it).
If you'd like me to clarify something, let me know. I tend to rant and derail into unnecessary tangents.
Good luck,