Preheat the oven to 400°F (204C). Line baking sheet with parchment paper.
In a small bowl, whisk egg with a few drops of water to combine. Set aside.
Roll out puff pastry on a lightly floured surface just until it is slightly flattened. Use a knife or pizza cutter to cut sheet into even rectangles (about 3 inches by 5 inches each).
Transfer the rectangles to the prepared baking sheets. Spread the jam into the center of each rectangle.
Brush the egg wash around the edges of each pastry. Top each pastry with the remaining rectangles and then use a fork to crimp the dough all around the edges to seal.
Cut a small vent into the center of each pastry and then brush the egg wash all over. Bake until the pastries are golden brown, 18 to 22 minutes. Let cool completely.
In a small bowl, whisk the sugar with the heavy cream and vanilla extract to combine. Whisk until smooth; add more cream a little at a time, as needed, so the icing is at a drizzling consistency.
Drizzle the icing over the cooled pastries.
Serve the toaster pastries with milk.