How to remove unwanted panels inside InspectorControls from core blocks in Gutenberg

Question

If I want to extend the core block I can use Block Filters where I found a filter like editor.BlockEdit.

Example from doc:

const { createHigherOrderComponent } = wp.compose;
const { Fragment } = wp.element;
const { InspectorControls } = wp.blockEditor;
const { PanelBody } = wp.components;
 
const withInspectorControls = createHigherOrderComponent( ( BlockEdit ) => {
    return ( props ) => {
        return (
            <Fragment>
                <BlockEdit { ...props } />
                <InspectorControls>
                    <PanelBody>My custom control</PanelBody>
                </InspectorControls>
            </Fragment>
        );
    };
}, 'withInspectorControl' );
 
wp.hooks.addFilter(
    'editor.BlockEdit',
    'my-plugin/with-inspector-controls',
    withInspectorControls
);

It works, but now i would like to remove unwanted panels from core blocks. Unfortunately, the documentation does not contain any such example so I can't understand how to do this.

So how to remove unwanted panels inside InspectorControls from core blocks in Gutenberg?

Note: I know about the newest feature theme.json, but it has very limited functions and won't remove everything it needs