5

I have many (100+) different WordPress installations across different servers and I'd need to have a centralized search... engine? to be able to search in all of the websites at once. I'm not running WordPress Multisite and installing it is not an option. Moving all the sites to the same server isn't an option either.

I've been researching for solutions and I've come across both ElasticSearch and Apache Solr, both seemingly built on top of Apache Lucene. There seems to be plugins for both systems, but none of them mention multi-website search.

How could I set this up? Do you have experience setting a search across multiple WordPress installations?

  • You mean like "your query" (site:www.stackexchange.com) OR (site:stackoverflow.com) in Google? – j8d Feb 22 '16 at 12:19
  • Exactly like that @j8d – José Tomás Tocino Feb 22 '16 at 12:21
  • So just use that. Or do you mean you want to put a search box on each of the websites that searches the current website as well as every other website? – j8d Feb 22 '16 at 12:26
  • I need to have our own search box to search in all the websites of our network, I cannot use Google's search, first because there are a lot of different domains involved and second because we'd rather have our own search server to get analytics from too. – José Tomás Tocino Feb 22 '16 at 13:29
  • 2
    Got it. I can't help then. Google Custom Search Engine does provide access to detailed analytics. developers.google.com/custom-search/docs/overview. – j8d Feb 22 '16 at 14:21
3

You should move your sites to the same server and same DB and then you will be able to run whatever queries you. Barring that, the only alternative I know to writing your own search engine is to use google's site search API which gives you access to the raw data instead of the snippets google usually displays. Cheapest tier is 100$/yr which is probably why you will not find many plugins for it.

Another option is to have some replication of the DBs into yet another server (or the one where the site with search functionality is), and run searches on it. But even that way it will be better if all servers are at least in a local network to have less potential security issues.

  • I cannot move my sites to the same server and DB, I'm dealing with a huge network of 100+ sites. Both elasticsearch and solr seem to have plugins to do something similar to what I'm looking for, so that's why I'm asking. – José Tomás Tocino Feb 24 '16 at 22:39
  • 1
    You will need to write middleware to connect the search engine with actual results and they will need the info stored at one place. Maybe I am just over pessimistic about the time and effort it takes to do things you and me don't understand, therefor the suggestion to use google API as middleware as it might be more cost effective than figuring how to setup and write web search over elastic search. – Mark Kaplun Feb 25 '16 at 4:14
  • 1
    @JoséTomásTocino Mark did not suggest moving your sites nor their databases to a single server - he suggested mirroring their data into a common database such that all data is indexed in one location. In this manner the database can be optimized to facilitate the singular task of providing your search functionality - every site still has and depends upon it's own database, and any searchable data is additionally replicated on the dedicated "search server". – bosco Feb 26 '16 at 0:38
3

I did similar thing to my client before, I created a meta search site for my client which use WordPress Rest API that search through all of my client sites.

Here is just a little idea to working with multiple site search in different server, this is just a rough example using PHP. If you're using WP site itself as the center of the search site, then it will be much easier with wp_remote_get function.

Requirements:

  • each site have WP REST API v2 Plugin installed http://v2.wp-api.org/

  • curl enabled on the site that want to request remote

Here are the codes:

Setting up API request using CURL

function call_api( $url ) {
    $curl = curl_init();

    curl_setopt_array( $curl, array(
        CURLOPT_URL => $url,
        CURLOPT_RETURNTRANSFER => true
    ));

    $result = curl_exec($curl);
    curl_close($curl);
    return json_decode($result);
}

Search function to handle form submit

function site_search( $sites, $search = '' ) {

    $results  = array();
    $api_url  = 'wp-json/wp/v2';
    $types    = 'posts';

    foreach( $sites as $site ) {
        $rest_url = "{$site}/{$api_url}/{$types}?search={$search}";

        $req = call_api( $rest_url );
        if( $req ) {
            $results = array_merge( $results, $req );
        }
    }

    return $results;
}

the variables

 //The search keyword
 $search_key = isset( $_GET['search'] ) ? $_GET['search'] : '';
 //lists of the sites you want to search with v2 rest api enabled
 $sites = array(
            'http://wordpress-site-with-v2-rest-api-enabled.com',
            'http://another-wp-site-with-v2-rest-api-enabled.com',
        );

the handler

$contents = array();
if( !empty( $search_key ) ) {
    $contents = site_search( $sites, $search_key );
}

the HTML

<!DOCTYPE html>
<html>
<head>
    <title>Test WP REST API</title>
</head>
<body>
    <form method="GET">
        <input type="text" name="search" value="<?php echo ( isset( $_GET['search'] ) ? $_GET['search'] : '' ); ?>" />
        <input type="submit" value="SUBMIT" />
    </form>
    <?php foreach( $contents as $content ) : ?>

            <div class="content">
                <h2><a href="<?php echo $content->link; ?>"><?php echo $content->title->rendered; ?></a></h2>
                <?php echo $content->excerpt->rendered; ?>
            </div>

    <?php endforeach; ?>
</body>
</html>

This is just a rough example, you'll need to work on security, etc. Hope this can help you get started.

  • 2
    This approach might work for couple of sites, but I'm not sure about 100+ sites as mentioned in the question ;-) ps: I would also recommend using WordPress HTTP API in your example. – birgire Feb 27 '16 at 11:11
  • Agree :D, this will require more work for 100+ sites :D, it's like building own search engine. In my case we use to crawl 12 different WordPress installations in a nodejs app, so it still working fine, but not sure if 100+ – nonsensecreativity Feb 27 '16 at 14:05
  • 2
    this approach would still be workable if you used a curl_multi_exec handler to get 5-10 results at a time rather than 1 by 1 like this... otherwise it would simply take too long. – majick Feb 29 '16 at 9:38
2
+50

Apache Solr can handle multiple DataImport Handlers. I would recommend setting up one handler for each MySQL database, and adding an additional field to the index for which server the data came from so that you could match slugs to websites.

Solr should easily handle the import of one hundred moderately-sized websites, but you will want the data partition on an SSD, not a spinning hard drive. I highly recommend putting the Solr instance on a Memory-Optimized Amazon AWS instance, such as r3.2xlarge or larger. This is going to be a RAM hog, especially during the initial import stages. After the initial import, you can tone down the server type if you see that the sites are not updated very frequently. I would still recommend using one of the Memory-Optimized AWS instances, though, as Solr really works well with them.

  • I'm giving you the bounty because your solution is the one closer to what I eventually ended up doing (using Elasticsearch). – José Tomás Tocino Mar 3 '16 at 8:22
  • I'm glad that it helped, José! Elasticsearch is based on Lucene, which Solr is also based on. Great choice! – dotancohen Mar 3 '16 at 8:23
1

At the end I've eventually done what I initially posted in the question. I've set up an Elasticsearch server using Docker, and I've installed the Elasticpress plugin, which properly indexes all the posts and pages in the elasticsearch index.

Then, I've created a simple front-end to search in Elasticsearch using simple bootstrap and jquery and it works like a charm.

0

Although this may not be the desired solution it is just an idea from where you can start to build your search. It still uses Google search engine in behind. You may will to use Google search engine to display 100 results and you will limit your search to specific web sites using the site:example.com site:example2.com "then your search query goes" .. So you will need to create your script that queries Google and then return the results to the end users.

  • sorry, but my reputation doesn't allow me to comment and still I wanted to share this idea since I find this is leading to a solution one way. Would you consider writing such a script as the answer to the question? – prosti Feb 24 '16 at 19:34
  • I actually was going to give the same answer, but I am not sure how off-topic the question is ;) – Mark Kaplun Feb 24 '16 at 20:02
  • I have several (100+) different websites and I cannot place all of them using "site:a.com site:b.com ..." in the query. I'd rather use a different, preferably self-hosted technology like the ones I've mentioned. – José Tomás Tocino Feb 24 '16 at 22:41
  • you are very humble person then, or I don't understand the several word. I always said: it may get complicated meeting several girls at the same time. Please update the original question then. – prosti Feb 24 '16 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.