If you're talking about ordinary yellow bananas, then they're a fruit. If you're talking about plantains then they're a starchy carb. Starchy carbs will be things like breads, pasta, legumes, potatoes, corn, rice and other grains. They are generally not sweet.
A banana is a fruit. All fruits have carbs. The carbs in fruit don't really count in the same way grains do when you are eating a balanced diet. The fruit is a source of certain vitamins, minerals, and fiber. Point - fruit is fruit.
The breakfast you ate is just not enough.
Double the peanut butter and add 2 grains (whole wheat toast, english muffin, multigrain waffles, oats or some kind of whole grain cereal to pour the almond milk over and mix in the chia seeds with. That would balance it out.
Eating simple carbs like banana isn't a reason to forgo important complex carbs like oats. Its not a situation where you eat a banana, and say to yourself "well, thats enough carbs for now". I would try and balance your intake of simple and complex carbs. For example, have a serving of oats, and put 1/2 a banana into it. Or have some whole wheat cereal, but toss in some fruit. That way you get your slow-burning complex carbs that give you energy throughout the day, along with the simple carb to give you that short-term spike you need in the morning.