There is no direct way, hence you'll have to hack it.
get_terms internally. There's a filter
terms_clauses which will allow you to modify the
WHERE conditions for getting the terms. Here's how you can use it:
add_filter('terms_clauses', 'term_filter', '', 1);
//You can change the entire SQL by examining the pieces array
$pieces['where'] .= '<SQL to get the term meta>';
//Here's how $pieces looks like:
[fields] => t.*, tt.*
[join] => INNER JOIN wp313_term_taxonomy AS tt ON t.term_id = tt.term_id
[where] => tt.taxonomy IN ('category') AND tt.count > 0
[orderby] => ORDER BY t.term_id
[order] => ASC
WHERE condition to the default
WHERE condition. I'm not very good with SQL (honestly, I couldn't find where term meta is stored!) hence didn't write the query. For more reference you can check the file
taxonomy.php (line. no. 1311) and these 2 posts: Link 1, Link 2.
Hope it helps!