This reply is late but I wanted to offer some solutions. Keep in mind I have no clue about your current circumstances, such as your age, socioeconomic status, or home life. My first recommendation would be a portable door lock. It's an easy solution that avoids the hassle of installing a better lock on your door. I bought mine on Amazon and it works like a charm! The only way someone would get in would be by breaking the door. However, I realize that this may not be an option for you depending on where you live. That's why I found some youtube videos for making a door lock with readily available materials.
The next option is to use a door wedge. This would probably be easier to do than the former option but also not as effective. With a door wedge, an intruder would still be able to get in if they pushed it out from under the door. Depending on how big the opening at the bottom of the door is, they could push it back. However, it would confirm that someone is going into your room. Assuming someone decides to go into your room by moving the wedge, they wouldn't be able to force it back under the door when they leave. Unless they have another exit in your room like a window.
The final option would be to talk to someone that could help. I know nothing of your family situation but I would hope your parents would be willing to help. If it comes down to it, having someone install a better lock might take one afternoon to solve. You could even ask for a better lock, without explaining your fears as I imagine you don't want to spread that (You could even make up a false reason). I understand that you would be scared of making assumptions because it could hurt your brother's feelings or the rest of your family for that matter. However, this is an important issue not just because of possible sexual assault, but because you won't have peace of mind until this is resolved.
Finally, I hope this message eased your mind. Originally I saw this post was five days old and was going to ignore it. However, seeing all the dismissive comments made me feel sad. Other people gave impractical solutions like setting up a camera, without taking into account the resources and time needed to set that up. I don't know if you're crazy but by the fact, you're asking whether your crazy or not I really doubt that. Maybe you're making a problem out of nothing, but I don't think it would hurt to try. This is all the help I am capable of giving. These methods are all prevention methods and don't solve whether you've been touched or not. I'm sorry but I have no solutions on how you could go about solving that besides looking for evidence. I hope you'll find the courage within yourself to solve this problem. I know this doesn't mean much from a stranger, but I do hope you find happiness. Don't give up!
Door Lock:
Keep in mind that I have no idea the specifications of your door. These door locks are for inward-facing doors which is standard for bedrooms. Note the lining of the door might not fit variations of door locks. Do your research on what may fit.
Door Stop Wedge:
Keep in mind that you can just make one yourself, following the general shape.