Preheat the oven to 350°F (175°C).
In a saucepan over medium heat, combine the blackberries, sugar, lemon juice, and vanilla extract. Cook for about 10 minutes, stirring occasionally until the mixture thickens. Remove from heat and set aside to cool.
While the compote cools, prepare the spicy candied pecans. In a mixing bowl, combine sugar, cayenne pepper, and cinnamon. Add the pecans and stir until evenly coated.
In a skillet over medium heat, melt the butter. Add the coated pecans and cook for about 5 minutes, stirring frequently until they are caramelized. Remove from heat and set aside to cool on a baking sheet lined with parchment paper.
For the baked brie, place the brie in a baking dish. Drizzle the honey over the top, and sprinkle with chopped rosemary and sliced almonds if using.
Bake in the preheated oven for 15-20 minutes until the brie is soft and melty.
To serve, spread the warm brie onto crackers or bread, topping it with blackberry compote and a sprinkle of spicy candied pecans.