Combine warm milk, yeast, and 1 teaspoon sugar in the bowl of a stand mixer. Rest for 10 minutes until foamy.
Add the remaining sugar, eggs, vanilla, orange zest, and lemon zest. Mix well.
Add flour and salt. Mix with the dough hook until a rough dough forms.
Gradually add softened butter, one tablespoon at a time.
Knead for approximately 10 minutes or until the dough becomes smooth and elastic.
Transfer to a greased bowl, cover, and refrigerate overnight for approximately 8–12 hours. Alternatively, let rise at room temperature until noticeably expanded.
Drain and thoroughly dry the raisins if soaked. Gently fold the raisins and candied orange peel into the dough.
Shape into a smooth ball and place in a suitably sized panettone mold.
Cover loosely and let rise until puffy and near the top of the mold, approximately 2–3 hours or longer.
Preheat oven to 350°F (175°C).
Score a shallow X on top and add a small pat of butter if desired.
Bake for approximately 40–45 minutes or until golden and the internal temperature reaches 190–195°F (88–91°C).
Cool completely, preferably inverted if the loaf is tall and delicate, before slicing.