So many people are afraid of sleights, and as you say, so many people become reliant on packet tricks or gaffs.
Sleights and other utility moves are fundamental in being able to take your enjoyment of the artform to the next level.
'Purist' card magic, i.e. card magic reliant on totally un-gaffed, un-gimmicked decks, is something which I think you should at least look at if you're interested in card magic after using, for instance, a Svengali or Stripper deck.
Most people start their card magic interest with either of these, and in fairness, I still use them now. But the enjoyment of actually learning and perfecting your card sleights is paramount to becoming a pupil of card magic.
All of these things take a long time to learn, and like Midas Kid, I've been doing this for over 10 years now, practicing wherever and whenever I can. I'm far from being perfect, but what I can say is that things that I never EVER thought possible are now just a deck of cards away.
For a lot of people, it's more a lack of confidence that puts them off... and this can be caused by showing someone a sleight before you've mastered it... and them saying "Well, you just keep putting them back in your left hand...".
It is SO important to handle a deck confidently. Get to know the feel of the cards, play with them, fold them, bend them, pick them up, drop them, learn how much pressure it takes to slide and hold them.
It is very rewarding, and is really worth it in the end.