Prestige Improvements at a Small School

Hi All, I am struggling to understand prestige and what helps/improves it -- I started at a small school (Rice) and I've certainly had some success over the past three seasons -->
2 Seasons Ago: 24-9, Won conference tournament, lost as an 11 seed; Prestige moved from a 22 to 27
1 Season Ago: 26-9, At-Large Bid, pulled tow upsets as an 11 seed to make the sweet 16; Prestige moved from a 27 to 33
Current Season: 24-10 At-Large Bid, won in round 1 as a 9 seed and then lost the 1 seed in round 2; Prestige moved down from 33 to 31.
What is going on here! 3 straight tournament appearances at a lowly C-USA school and I just won a game in the tournament, yet my prestige is moving down. Does anyone have any idea on what else has a material impact on prestige? Do I need to get closer to 30 wins and win tournament games to move above 40? Or is there some "baseline" prestige for Rice that I'm buoyed to?
2 Seasons Ago: 24-9, Won conference tournament, lost as an 11 seed; Prestige moved from a 22 to 27
1 Season Ago: 26-9, At-Large Bid, pulled tow upsets as an 11 seed to make the sweet 16; Prestige moved from a 27 to 33
Current Season: 24-10 At-Large Bid, won in round 1 as a 9 seed and then lost the 1 seed in round 2; Prestige moved down from 33 to 31.
What is going on here! 3 straight tournament appearances at a lowly C-USA school and I just won a game in the tournament, yet my prestige is moving down. Does anyone have any idea on what else has a material impact on prestige? Do I need to get closer to 30 wins and win tournament games to move above 40? Or is there some "baseline" prestige for Rice that I'm buoyed to?