SteemAdd semantic search to WordPress with AI embeddings
Published Updated 4 min read
WordPress's default search is keyword search. It looks for the visitor's words in titles and content, counts them, and ranks by frequency. That works when someone types the exact phrase a post uses. It fails when they describe what they want in their own words, use a synonym, or ask the question differently from how you answered it.
Relevanssi improves things a lot with stemming, fuzzy matching and field weights, but it is still keyword-based. A visitor searching 'how to cancel my account' will not find your 'membership termination policy' page unless you have set up synonyms or the two happen to share words. The gap widens as a site grows into thousands of posts, each phrased a little differently from how people ask.
This plugin ranks search results with OpenAI embeddings instead. Saving a post stores a 1,536-number vector that encodes its meaning. A search embeds the query the same way, compares it with every stored vector by cosine similarity, and returns the posts closest in meaning rather than those sharing the most words. One API key is the only outside dependency.
Why keyword search struggles at scale
Default search runs SQL LIKE queries on post_title and post_content, so the term has to appear literally. MySQL full-text search, which Relevanssi uses, adds stemming and relevance scoring, but the model is the same: the query's terms must be in the document.
That suits navigational searches for the exact name of something and fails conceptual ones. 'Refund policy' misses a page titled 'returns and cancellations'; 'setup guide' misses a post tagged 'getting started'. With a few dozen posts you can work around it with careful titles and tags. With thousands, you cannot cover every way someone might phrase the same question.
Embeddings fix it at the model level instead of through synonym lists. OpenAI's text-embedding-3-small turns text into a point in 1,536-dimensional space where similar meanings sit close together, so 'refund policy' and a post about returns and cancellations land near each other with no words in common.
How embeddings work in this plugin
The embeddings API takes a string and returns 1,536 floating-point numbers. Texts that mean similar things produce vectors that are numerically close, measured by cosine similarity, the cosine of the angle between them: near 1.0 is closely related, near 0.0 unrelated.
Each post's embedding is stored as serialised post meta under _semantic_embedding. A search embeds the query string, loops over every stored embedding computing similarity, and returns posts in descending order, bypassing the SQL keyword search entirely.
text-embedding-3-small costs roughly $0.02 per million tokens. A 500-word post is around 700 tokens, so indexing 1,000 posts costs under a cent, and each search is one short call costing a fraction of a cent. For thousands of posts and moderate traffic the monthly bill is typically under a dollar; very high-traffic sites should check OpenAI's pricing before relying on a call per search.
Indexing on save, ranking on search
A save_post hook generates and stores the embedding whenever a post or page is published or updated. Settings > Semantic Search holds the OpenAI key and a button to re-index everything already published, which you need once after activation. It runs synchronously, so on a large site the page can take a minute or two to come back.
On the front end, the posts_pre_query filter intercepts the main search query before it reaches the database, embeds what the visitor typed, scores it against every stored embedding and returns the top ten posts as WP_Post objects. They flow into your theme's search.php exactly as keyword results would.
There is no custom table and no JavaScript. Embeddings live in the standard post meta table and survive theme switches, caching plugin changes and host migrations. If the OpenAI call fails, the filter returns null and WordPress falls back to keyword search, so results showing up as keyword matches usually means a wrong key or one that has hit its usage limit.
Ten results, posts and pages, a few thousand at most
The scoring loop runs in PHP over every stored embedding on every search. With a few hundred posts it is quick. With tens of thousands it is slow, and beyond that it may time out on shared hosting. Past a few thousand indexed posts, use a service with native vector indexing.
Only posts and pages are indexed. WooCommerce products and custom post types from other plugins are left out, and adding them means adding them to the post_type array in two places. The embedding is built from the title and stripped post_content only; to include custom fields or ACF data, add their values to the $text variable in index_post.
Results are a fixed top ten with no pagination, and found_posts does not reflect the full set, so themes that show a result count or paginate search will show wrong numbers.
Typesense, Algolia or SearchWP's add-on
Where search performance matters at scale, a service that keeps embeddings in a vector database and queries them with approximate nearest-neighbour search wins. Typesense offers hybrid keyword and semantic ranking and can be self-hosted; Algolia has a WordPress connector and handles high query volumes. Both cost money beyond a free tier and need more integration than one plugin file.
SearchWP, the most widely used paid search plugin, has a semantic add-on that runs embeddings through a managed cloud service. If you already use SearchWP for other things, that add-on is the simpler path.
Where the problem is result quality rather than query volume, this build is a practical start. A follow-up message can extend it to WooCommerce products, adding the product post type in both places and including each product's short description in the embedded text.
What you get
Included
- Generates and stores OpenAI embeddings for published posts and pages when content is saved or updated
- Replaces WordPress search results with posts ranked by cosine similarity to the search query
- Admin settings page at Settings > Semantic Search with API key field and one-click bulk re-index button
- Falls back to WordPress keyword search if the API call fails, so search always returns something
Not included
- WooCommerce products: targets posts and pages only; product search requires additional hooks
- Large sites: cosine similarity runs in PHP over all indexed posts, which is slow past a few thousand
- Pagination: returns a fixed top ten results; found_posts is not updated for multi-page result sets
- Custom fields or ACF data: embeddings are generated from post title and post content only
- Result caching: each search calls the OpenAI API in real time, so repeated identical queries each incur an API call
SearchWP does the things on the right. This build covers the part most sites use.
Questions
Will this conflict with Relevanssi or SearchWP?
Yes. All three take over the WordPress search hooks, so run one at a time. Deactivate Relevanssi or SearchWP before activating this.
Build it in Steem
The button opens the builder with this guide's prompt filled in, so you can change anything before it runs. Nothing is built, and nothing is charged, until you send it.
Read the prompt (488 words)
Build a WordPress plugin called WP Semantic Search. Plugin Name: WP Semantic Search. Version: 1.0.0. Description: Replace WordPress keyword search with OpenAI embedding-based semantic search. Wrap all code in a class WP_Semantic_Search with a public init() method. Instantiate the class and call init() on plugins_loaded. Settings page: In init(), add_action('admin_menu', [$this, 'add_settings_page']). Register the page using add_options_page('Semantic Search', 'Semantic Search', 'manage_options', 'semantic-search', [$this, 'render_settings_page']). In init(), call register_setting('wp_semantic_search', 'wp_semantic_search_options'), add_settings_section('main', '', null, 'semantic-search'), and add_settings_field('api_key', 'OpenAI API Key', [$this, 'render_api_key_field'], 'semantic-search', 'main'). render_api_key_field(): $opts = get_option('wp_semantic_search_options', []); echo '<input type="password" name="wp_semantic_search_options[api_key]" value="' . esc_attr($opts['api_key'] ?? '') . '" class="regular-text">'; render_settings_page(): outputs a div.wrap with h1 'Semantic Search', a form with method post, settings_fields('wp_semantic_search'), do_settings_sections('semantic-search'), submit_button(), closing form. Then a second form with method post, wp_nonce_field('semantic_reindex'), hidden input name=semantic_reindex value=1, submit_button('Re-index all published posts and pages', 'secondary'). Show a notice if $_GET['reindexed'] is set: '<div class="notice notice-success"><p>Re-indexed ' . intval($_GET['reindexed']) . ' posts.</p></div>'. Generating embeddings: In init(), add_action('save_post', [$this, 'index_post'], 10, 3). In index_post($post_id, $post, $update): return early if wp_is_post_revision($post_id), if $post->post_status !== 'publish', or if !in_array($post->post_type, ['post', 'page']). $text = substr($post->post_title . "\n" . wp_strip_all_tags($post->post_content), 0, 8000). $embedding = $this->get_embedding($text). If !empty($embedding), update_post_meta($post_id, '_semantic_embedding', $embedding). In get_embedding(string $text): array: $opts = get_option('wp_semantic_search_options', []); $key = $opts['api_key'] ?? ''; if (empty($key)) return []; $resp = wp_remote_post('https://api.openai.com/v1/embeddings', ['headers' => ['Content-Type' => 'application/json', 'Authorization' => 'Bearer ' . $key], 'body' => wp_json_encode(['model' => 'text-embedding-3-small', 'input' => $text]), 'timeout' => 20]); if (is_wp_error($resp)) return []; $body = json_decode(wp_remote_retrieve_body($resp), true); return $body['data'][0]['embedding'] ?? []; Re-index action: In init(), add_action('admin_init', [$this, 'handle_reindex']). In handle_reindex(): if (empty($_POST['semantic_reindex'])) return; check_admin_referer('semantic_reindex'); $posts = get_posts(['post_type' => ['post', 'page'], 'posts_per_page' => -1, 'post_status' => 'publish']); $n = 0; foreach ($posts as $post) { $text = substr($post->post_title . "\n" . wp_strip_all_tags($post->post_content), 0, 8000); $emb = $this->get_embedding($text); if (!empty($emb)) { update_post_meta($post->ID, '_semantic_embedding', $emb); $n++; } } wp_redirect(add_query_arg(['page' => 'semantic-search', 'reindexed' => $n], admin_url('options-general.php'))); exit; Search override: In init(), add_filter('posts_pre_query', [$this, 'semantic_search'], 10, 2). In semantic_search($posts, WP_Query $query): if (!$query->is_main_query() || !$query->is_search() || is_admin()) return $posts; $s = trim($query->get('s')); if (empty($s)) return $posts; $qv = $this->get_embedding($s); if (empty($qv)) return $posts; $ids = get_posts(['post_type' => ['post', 'page'], 'posts_per_page' => -1, 'post_status' => 'publish', 'meta_key' => '_semantic_embedding', 'fields' => 'ids', 'no_found_rows' => true]); if (empty($ids)) return $posts; $scores = []; foreach ($ids as $id) { $emb = get_post_meta($id, '_semantic_embedding', true); if (is_array($emb) && count($emb)) $scores[$id] = $this->cosine_similarity($qv, $emb); } if (empty($scores)) return $posts; arsort($scores); $results = []; foreach (array_slice(array_keys($scores), 0, 10) as $id) { $p = get_post($id); if ($p) $results[] = $p; } return $results; In cosine_similarity(array $a, array $b): float: $dot = $na = $nb = 0.0; for ($i = 0, $len = count($a); $i < $len; $i++) { $ai = $a[$i]; $bi = $b[$i] ?? 0.0; $dot += $ai * $bi; $na += $ai * $ai; $nb += $bi * $bi; } return ($na && $nb) ? $dot / (sqrt($na) * sqrt($nb)) : 0.0; No custom database table, no JavaScript, no stylesheet.