Difficult to define when something is a challenge and something is merely grinding/luck/whatever, especially since it can differ from person to person.
Also note that "artificial challenges" that the game maker may not have intended can be considered a challenge (eg low level runs, Nuzlocke etc).

Some things which may cause something to be challenging:
- bad controls - typically unfavorable
- increased game speed - perhaps more favorable than bad controls
- enemies take more hits - typically done; effect can vary
- smarter enemy AI - typically a good idea, but can be difficult to implement (unless it's something where computers can easily calculate the exact angle to fire etc)
- level modifications - can make the game rather different, but requires a fair bit of design work

Also, depends on whether it's challenging because you need to execute a precise sequence of things at the right time (eg near perfect jumps), forcing you to grind more, or forcing you to think of new strategies?

I'm not sure that challenges should be beatable the first time through - after all, they're typically meant for someone who's beaten the game at least once (and some games will enforce this by unlocking the extra hard difficulty modes after the game is completed).
Sometimes having hindsight on what happens later in the game can affect how you play, and a well designed challenge will take this into consideration.

Unfortunately, designing a challenge so that it changes the gameplay a fair bit can be somewhat difficult to do, so many just resort to handicaps or similar.

This post was written in a bit of a rush.