What theme are you using?

I would probably start with making a child theme. This will give you more control over when styles are loaded.

If you need to add styles specifically for the admin area you can enqueue styles like this...

function my_admin_styles(){
    wp_enqueue_style(
        'admin_css', 
        get_stylesheet_directory_uri() . '/css/my-admin.css', array(), filemtime( get_stylesheet_directory() . '/css/my-admin.css') 
    );
}

add_action('admin_enqueue_scripts', 'my_admin_styles');

What theme are you using?

I would probably start with making a child theme. This will give you more control over when styles are loaded. Here is an example of loading the parent theme styles and then your child theme styles.

function my_theme_enqueue_styles() {

    $parent_style = 'parent-style'; // This is 'twentyfifteen-style' for the Twenty Fifteen theme.

    wp_enqueue_style( $parent_style, get_template_directory_uri() . '/style.css' );
    wp_enqueue_style( 'child-style',
        get_stylesheet_directory_uri() . '/style.css',
        array( $parent_style ),
        wp_get_theme()->get('Version')
    );
}
add_action( 'wp_enqueue_scripts', 'my_theme_enqueue_styles' );

If you need to add styles specifically for the admin area you can enqueue styles like this...

function my_admin_styles(){
    wp_enqueue_style(
        'admin_css', 
        get_stylesheet_directory_uri() . '/css/my-admin.css', array(), filemtime( get_stylesheet_directory() . '/css/my-admin.css') 
    );
}

add_action('admin_enqueue_scripts', 'my_admin_styles');

The above code would go in your child theme's functions.php.