EDIT: For those who are interested as per Tom J Nowell the problem was that I used it with my own custom widget which doesn't have before and after support.
I have this sidebar and everything works fine except that the before and after widget don't show up:
register_sidebar( array( 'name' => 'Footer Three', 'id' => 'footer-three', 'before_widget' => '<div class="footer-three-wrapper">', 'after_widget' => '</div>', ) );
And inside page:
<?php dynamic_sidebar( 'footer-three' ); ?>
Isn't before_widget and after_widget supposed to show up?