Honestly it isn't even that big of a deal, and I doubt that he would even bring it up to your brother. It would also be really awkward for him as well don't worry about it, and honestly screw him move on with your life.