"How do you create horror?
Atmosphere.
Atmosphere and willing participation.
You cannot scare someone who doesn't want to be scared. You are describing things verbally, and if the player refuses to imagine them, they will never be scared of them.
Atmosphere is how they will imagine them. Use small wrongs to build up tension, larger wrong to accelerate that tension, and when that tension is at its peak, hit them with a horror set-piece. They won't accept either of the latter early on, because they're not in the mood. The small wrongs are there to get them to set their disbelief back just that little bit, and push up the tension slightly higher. That lets you worm in something more wrong, and the process repeats. Never drop a set-piece until they are demonstrating physical signs of discomfort or you'll waste it. The walls, roof, and floor of the giant room you're in all growing faces and starting to sing folkloric music is silly if you aren't in the mood, but if you ARE in the mood, it will fuck you up chronically.
Something critical to your success, or failure, in a horror game will be rules.
Rules provide predictability. Rules provide stability. Rules provide a known quantity to which everything can be reduced. These are all bad things for horror. If you want to make horror that works, you have to be willing to adapt the rules of the game to meet your needs. You need to keep the players tense and on-guard, especially if you intend to use their paranoia against them. Follow the rules, more or less, but make sure you change them. Make large changes to the rules and keep them that way for a while. Make a one-off exception. Change something,then change it back soon after. If possible, use a system the players are not intimately familiar with. If the players lose the sense of predictability known rules provide, it firstly provides tension, and more subtly, forces them to listen to your descriptions as a whole, instead of just hearing the relevant stats and rolls. That in turn makes them more susceptible to horror tactics. Be sparing, though. Make sure every change is backed by, and appears to have been caused by, something operating in the game world. If they draw causality from there they will pay more attention to the game world, and that makes them listen more and imagine. This also ensures that your players will believe that there are rules. While you don't have to let the players know what the rules are, and you should never do so for horror, the players have to believe a set of rules exist or they will instantly have all immersion destroyed as they realize this is purely based on arbitrary whim. They won't see the point in playing if their cause doesn't lead to an effect.
The challenges the players face are the basis of the game. I like to divide the challenges the players will face into four categories here, for ease of description. First, normal threats. These are threats you can beat by fighting, as in a normal game. Normal things for that world. These challenges the players will beat by fighting. If they run, they escape, but the threat remains. If they ignore it, they get hurt. If they play along, well, work it.These should start the game, to establish the rules you're going to change. As the game progresses, these challenges should get more and more difficult to fight, without actually becoming implausible to beat, to maintain tension.
Next are the Unstoppables. These are challenges, threats, or things that the players can't reasonably overcome. They should run from these,as fighting will get them hurt, ignoring will get them hurt worse, and playing along will get them hurt worst of all. These should come up soon.As the game progresses, these become more dangerous, going from likely hurt, to certain hurt with death chance, to certain death with only a notional chance of survival.
Next, comes the phantasmal challenges. These are challenges that aren't, usually the result of twisted perceptions. They come up later in the piece, once the rules have been changing considerably. These threats cannot be fought without being hurt, and cannot be run from because they will just follow. If you play along you'll either get hurt or stay trapped. The only way out is to ignore them. These must come later in the piece, so that the tone of the campaign already fits things undefeatable by normal methods. The effectiveness of such threats is overwhelmingly based on your description. As the game progresses, these should be less and less obvious, and take more effort to discern the 'escape move'.
Finally, there are the ineffable challenges. These are challenges the players cannot beat, run from, or ignore safely. To escape these, the character must play along, or find a mental escape. Think the Creepy kid from movies and games. These are often the part of the Big Threatof a horror campaign. These have to come latest in the piece, because firstly you need something that is threatening enough that the players have to play along, and secondly you need to show them subtly how to do these. As you progress, these challenges should take more complex, vicious, or unwanted actions from players to play along.
At first, keep all the different challenge types distinct. Then, as the players start to figure out the differences, blur the visual lines. Then, blur the actual lines of the rules. Always leave the players in doubt as to what it is they should do here. Not crippling doubt, save for big setpieces,but ensure every time they plan to deal with something there's that niggling concern in their minds that this might not be the way they think it is.
Pacing is next. This will come down to your ability to read players, and set the pace appropriately, but I find that when I run campaigns a fast pace works best. Always keep things moving slightly faster than the players want to. Don't give them time to stop and think. They start analyzing, they start wondering, they start to figure out the rules and your part in playing them. Keep them moving fast, and don't let them quite catch their breath, and the tension will keep up. Vary the pace slightly, to keep it interesting. Generally accelerate things towards the end. I like to keep the denouement of my horror campaigns with things moving so fast the players barely have any idea what's going on, which is extremely effective in building up the tension. Force them to choose fast, and give them the worst result if they don't choose quick. Just make sure you don't push it too far, into the realm of being unfair instead of just tense, or they'll stop playing along.
Also, if you think you can do it well, try to throw in sudden stops near the end. Points at which everything just slows down. If you've turned on the little paranoia in the players' heads, this will drive them insane. Do this very rarely, though, or it gets very obvious and very old.
Be descriptive. Being descriptive is utterly vital in horror. A good descriptive GM can make anything seem scary. A mechanical rote DM will make having Yawgmoth propose to you seem mundane and uninteresting. I can't tell you how to be descriptive, since everyone has their own style, and trying to play to another person's style just doesn't work. What I can do is give you a few tricks.
The first, and most common, way of creating tension is the 'nearly there'. That's where you take something very ordinary, and change just the littlest bit. That exploits the players' familiarity, and can work quite well. Another trick if you can do it well is to exploit that familiarity further and not fill in that missing detail until later, playing it as the characters having overlooked it. If you can do that well, it works wonders, making their acceptance of something comfortable even more disturbing. Something like an ordinary beach, plain sand, plain water, buckets, pails,dead fish every so often, and a little seaweed. All rather ordinary, save that the waves are rushing outwards from the sand. Be subtle here,though. Doing this hamfistedly will just annoy your players, kill the mood, and destroy the tension you so painstakingly built up.
Next comes the opposite, or the things that are just wrong, except for one small detail. Focus more on the one small detail than the wrongs.This is more effective once tension has built up, and the players are more willing to accept the strange.
Then comes the outright wrong, where there are no redeeming or familiarizing features to a thing. This only works once tension has mounted considerably and the players accept these things without thinking. Because these depend on their alien-ness, use them as sparingly as possible. They make good cores for horror set-pieces.
Finally, there's the absolutely normal. Only working once the tension has driven your players paranoid, this is something that appears completely normal because it is. Once you've been encountering the horrific and ineffable for hours at a time, something completely normal represents a drastic shift in the rules. You can, if you're good, achieve more fear from something utterly normal than something ostensibly terrifying.
Critically for all descriptive GMing, be descriptive, not prescriptive. The more alien a thing is, the less direct the information should be. Ideally,you want most of your descriptions to raise far more questions than they answer. Even the most horrific known thing can only be as horrific as it is. The unknown, though, is as horrific as everything it can be.
Description is also where you produce real fear in your players. When something goes wrong in horror, never let it be a strict number, or simple YOU ARE DEAD. When something goes wrong, it should be incredibly unpleasant. If what the players fear about failure is that death means end of game no more rewards, you've failed. If the players fear failure because of what happened to their character the last time they fucked up, you're doing it right. A player doesn't fall to zero HP and die; they have their ribs torn open and feel the edges of their vision fading,as they desperately try to stop the bleeding from everywhere at once. They don't get hit by a shadow damage trap, they're sucked violently into a shadow on the floor, fingers gouging tracks in the wood, and after falling under screams of primal fear and pain can be heard from insidethe walls, before the character inexplicably falls through the roof, bloodied, and with a look of utter terror on their face.
Seeing as we're on description and its role in producing horror, there are some types of horror eminently suited for role playing, and many that are not. Don't use 'BOO' horror. It is purely visual and doesn’t work at all here. Even if it did it is the worst kind of horror and is only used in compensation for the failure of the creator to produce something meaningfully terrifying.
Disturbing horror, as in describing things that just make you feel personally uncomfortable, takes exceptional skill, since it's completely dependent on your ability to read your players. If you can do it and maintain it just below the point at which the player has had enough and just leaves the game, do so. If you can't read the player, don't try. Also, don't do it to any player unless you can do it to all, as it'll make that player feel you're trying to go after him personally.
Fear of consequence works well, since the total control over the character and the tension you're creating work to make players empathize with and become immersed in their characters. This is where the consequences mentioned above become truly upsetting.
Paranoia creation works perfectly here, since you have complete control over information, and the players are inherently using their imaginations. Use it, and use it well.
'WHAT THE SHIT IS THAT' may or may not work. That all comes down to your ability as a descriptive GM. If it does, exploit it mercilessly.
On a side note, the use of motifs can work very well in horror. Try to associate all the worst events in the game with one motif, and put them before it. The tinkly music box is one of them, or a certain NPC's voice, or anything. If you use that motif and the players immediately start to get visibly tense, you're doing it right.
There is also the option of fucking with the players, and turning them on each other, but that is a very individual choice. It may work; it may not, all depending on your group.
If you want to do like I do sometimes and create a setting where the players are almost outright hostile to each other but work together still solely because they need to survive, combine separation and concentration to produce mutual paranoia. Being alone makes them uncomfortable, since they don't have any support and they don't have anything to confirm that what they see is real. Being with someone else makes them uncomfortable, since there is an excellent chance that they aren't who they say they are or even real. Remember to slide a note to one or both or all players every time they meet, even if it's blank. Sometimes, it will say 'you are an angry/sad/violent/helpful hallucination/ghost/disguised enemy' or 'you are his friend, but have become slightly/significantly/murderously hostile/anti-pathic/helpful/lustful for reason XYZ;. It takes work, but making players completely distrust one another without once making an actual PC willingly betray them makes for excellent horror. You want to create that unique-to-horror feeling of immense relief followed by deep suspicion every time you see another living human.
Or, for a less personal feel, do like 'FEAR', and have the other players almost always good, but with the caveat that if you ever meet another person it's because something absolutely horrible is about to happen to both of you. This is my personal favorite, because they instinctively want to stay other players and NPC's to feel safe, but the moment they actually see someone else they try to get away from each other as soon as possible because something nasty will happen. Try to capture that feeling from FEAR where you're screaming at the screen NO YOU FUCKHEAD DON'T MEET UP WITH ME YOU'RE GOING TO GET BRUTALLY FUCKING KILLED.
One trick I like to use in that situation is to have the players roll up multiple characters. Not individually, since that creates a sense of private-ness, but as a group. Hand them ten character sheets and tell them to fill them out as a group. They take one character at a time, and you work the others as you need to in the plot or under their direction if not. This gives you a much greater ability to produce incredibly harsh consequence, without compromising the extent to which players empathize with their characters, and turn up the difficulty considerably to add to the horror without making the game objectively more difficult. It does add an enormous amount of extra administration, though, so be careful with it.
There's more, but at this point I've completely lost track of my mental organization of this stuff. Just remember the basic rules of horror GMing:
The rules system is irrelevant.
The content is unimportant.
Delivery is everything."
Truly inspiring stuff. For my next post, I think I'll expand upon his masterful guidance and provide some bite-sized samples/material for your enjoyment.