Whisk together all the custard ingredients in a bowl, set aside
Begin by alternating layers of pancakes, pound cake crumbles, blueberries and raspberries- 2 layers of each in a 13x9 casserole dish
Pour custard over layers, kind of mashing down to soak the top layers-carefully though not to squish berries
Bake in a 350 degree oven till custard is set - approximately 1 - 1 1/2 hours
While baking prepare pastry cream-In a 2 quart saucepan, combine sugar, flour, salt and milk and whisk; over medium heat, cook while stirring until mixture thickens.
In a small bowl with a fork, beat egg yolks, beat small amount of milk mixture into yolks to temper the eggs - so they don't scramble by adding them to hot milk mixture
Slowly pour egg mixture back into milk mixture.
Stirring constantly, cook over medium low heat until mixture thickens.
Remove from heat and stir in vanilla extract. Chill cream until ready to use.
Once baked, sprinkle with powdered sugar and serve with pastry cream.