When you first sign up to Twitter, things will move very slowly comparatively speaking. The simple truth of the matter is that when you only have a couple of people in your feed and they are not online at the time, you won't have much to read. There are ways to speed things up. Adding your favorite newspaper - they almost always have a Twitter account, a couple of TV stations, some sports teams and a celeb or two will keep the tweets coming, albeit maybe not as immediately relevant as you would like. You should also find out which of your friends are on Twitter. Following them will give you a chance to swap jokes and chit chat when you're apart. 

After a while, through re-tweets from your friends and people picking up and following your feed, names will become familiar and you can add them if you feel so inclined. Before too long you will be getting more tweets than you want to read - and then comes the decision of who to "unfollow". This can be a more stressful pursuit than you think, because some people check their followers page frequently and they often do it when they notice their numbers have dropped. You may get an angry or hurt response from someone you unfollowed. 

Should you be worried or reluctant to unfollow someone because of this? It depends on how you feel about getting those messages. As long as you are confident in telling someone that your feed went too fast and you had a little cull, you shouldn't be too worried. Failing that, you can always just ignore their tweets!