This feature is a part of Categorized Product Tree. Click here to see all Categorized Product Tree features.
Instead of showing the categories first and the product at the bottom of the product list, you can display it as a categorized listing.
Each category will be a section title with all the products inside.
To enable it to go to Catalog Settings > Catalog Design and select the Categorized Product Tree:
After that, select the template for products:
The categories will always be displayed as titles/links, but you can show the products with any other active template.
Here is how it would look with the Modern Grid:
Clicking on the category/subcategory will collapse/expand the list.