In a medium bowl, whisk together flour, cocoa powder, baking soda, and salt.
In the bowl of an electric mixer fitted with the paddle attachment, cream together butter and sugar until light and fluffy, about 2 minutes. Add egg, milk, and extracts, and beat until well combined. Add the flour mixture and mix on low speed until combined.
Using a tablespoon, drop portions of dough onto ungreased baking sheets, about 2 inches apart. Bake until cookies begin to spread and become firm, 7 to 10 minutes.
Remove the baking sheets from the oven and place a marshmallow, cut-side down, in the center of each cookie, pressing down slightly. Return to the oven and continue baking until the marshmallows begin to melt, 2 to 2 ½ minutes.
Transfer the cookies to a wire rack to cool completely before frosting. To frost, spread about 1 tablespoon of frosting over each marshmallow, starting in the center and continuing outward until the marshmallow is covered.