SQL Server 2012 - Semantic Search

Business Intelligence
Semantic Search in SQL Server 2012

• Semantic search seeks to improve search
accuracy by understanding searcher intent
and the contextual meaning of terms as they
appear in the searchable dataspace.
What is Semantic Search

• Built on top of Full-Text Search
• Requires predefined external Database
• That database should be attached to SQL
Server Instance
• Semantic Search should be configured to use
that Database
Semantic Search in SQL Server 2012

• Exists in all Commercial editions of SQL Server
2012
�� Also in SQL Server 2012 Express Advanced
Services Edition
Supported in SQL Server Editions

Semantic Search Installation 1/3

-- do not use sp_attach_db stored procedure
-- it is obsolete
CREATE DATABASE SemanticsDB
ON (FILENAME = N'C:Program FilesMicrosoft
Semantic Language DatabasesemanticsDB.mdf')
LOG ON (FILENAME = 'C:Program FilesMicrosoft
Semantic Language Databasesemanticsdb_log.ldf')
FOR ATTACH;
GO
Attach Semantics DB

-- Register Semantics Languages Database
-- required once
EXEC
sp_fulltext_semantic_register_language_statisti
cs_db @dbname = N'SemanticsDB';
GO
Register Semantics DB

-- Verify the registration is succeeded
SELECT * FROM
sys.fulltext_semantic_language_statistics_database;
GO
Verify Registration

-- Check available languages for statistical semantic extraction
SELECT * FROM sys.fulltext_semantic_languages;
GO
Supported Languages

-- Reload filters (iFilter) and restart fulltext
-- host process if needed
EXEC sp_fulltext_service 'load_os_resources', 1;
EXEC sp_fulltext_service 'restart_all_fdhosts';
GO
Restart Processes

Full-Text Search
• Supports character-based columns:
1. char
2. varchar
3. nchar
4. nvarchar
5. text
6. ntext
7. image
8. xml
9. varbinary (max)
10. FileStream
Text

Full-Text Queries Specifics
• Full-text queries are not case-sensitive searching for
"Aluminum" or "aluminum" returns the same results
• Transact-SQL predicates:
– CONTAINS
– FREETEXT
• Transact-SQL functions:
– CONTAINSTABLE
– FREETEXTTABLE
Text

SELECT * FROM sys.fulltext_document_types;
File types supported by iFilters

Three Tabular Functions:
• SemanticKeyPhraseTable - returns the statistically
significant phrases in each document
• SemanticSimilarityTable – returns documents or
rows that are similar or related, based on the key
phrases in each document
• SemanticSimilarityDetailsTable – returns the key
phrases that explain why two documents were
identified as similar
Semantic Search Functions

-- select Full-Text Catalog items count
SELECT FulltextCatalogProperty
('FullTextCatalog', 'itemcount');
GO
Full-Text Catalog Items Count

-- check Population progress
SELECT fulltextcatalogproperty('FullTextCatalog', 'populatestatus');
GO
• 0 = Idle
• 1 = Full population in progress
• 2 = Paused
• 3 = Throttled
• 4 = Recovering
• 5 = Shutdown
• 6 = Incremental population in progress
• 7 = Building index
• 8 = Disk is full. Paused.
• 9 = Change tracking
Full-Text Catalog Population Status

-- Get all key phrases in the entire corpus
SELECT
K.score, K.keyphrase, COUNT(D.stream_id) AS Occurrences
FROM SemanticKeyPhraseTable
(dbo.Documents, (name, file_stream)) AS K
INNER JOIN dbo.Documents AS D
ON D.path_locator = K.document_key
GROUP BY K.score, K.keyphrase
ORDER BY K.score DESC, K.keyphrase ASC;
GO
Get all Key Phrases

-- Find documents by keyphrase – ‘sql’ in the case below
SELECT
K.score, K.keyphrase,
D.stream_id, D.name, D.file_type, D.cached_file_size,
D.creation_time, D.last_write_time, D.last_access_time
FROM dbo.Documents D
INNER JOIN semantickeyphrasetable (
dbo.Documents,
(name, file_stream)
) AS K
ON D.path_locator = K.document_key
WHERE K.keyphrase = N'sql'
ORDER BY K.score DESC;
Find Documents by Key phrase

-- find similar documents
DECLARE @Title NVARCHAR(1000) = (SELECT'Gurevich Vladimir.docx');
DECLARE @DocID HIERARCHYID =
(SELECT path_locator FROM dbo.Documents WHERE name = @Title);
SELECT
@Title AS source_title, D.name AS matched_title,
D.stream_id, K.score
FROM SemanticSimilarityTable(dbo.Documents, *, @DocID) AS K
INNER JOIN dbo.Documents AS D
ON D.path_locator = K.matched_document_key
GO
Find Similar Documents

-- find out Key Phrases that make two documents match
DECLARE @SourceTitle NVARCHAR(1000) = (SELECT ‘source.docx');
DECLARE @MatchedTitle NVARCHAR(1000) = (SELECT ‘target.docx');
DECLARE @SourceDocID HIERARCHYID =
(SELECT path_locator FROM dbo.Documents WHERE name = @SourceTitle);
DECLARE @MatchedDocID HIERARCHYID =
(SELECT path_locator FROM dbo.Documents WHERE name = @MatchedTitle);
SELECT
K.keyphrase, K.score, @SourceTitle AS source_title, @MatchedTitle AS matched_title
FROM SemanticSimilarityDetailsTable(dbo.Documents, file_stream, @SourceDocID,
file_stream, @MatchedDocID) AS K
GO
Why 2 Documents Are Similar

• The generic NEAR operator is deprecated in SQLServer2012
• It is a new operator and not an extension of the existing NEAR
operator
• Lets to query with 2 optional requirements that you could not
previously specify
1. The maximum gap between the search terms
2. The order of the search terms - for example, “John” must appear
before “Smith”
• Stopwords or noise words are included in the gap count.
CONTAINSTABLE(Documents, Content, ‘NEAR((John, Smith), 4, TRUE)’);
Full-Text Search NEAR Operator 1/2

• -- get documents that contain keywords "sql"
and "server" nearby
• SELECT D.name,
file_stream.GetFileNamespacePath() AS
relative_path
• FROM dbo.Documents D
• WHERE CONTAINS(file_stream, 'NEAR(("sql",
"server"), 1, FALSE)');
• GO
Full-Text Search NEAR Operator 2/2

-- get documents that contain keywords "sql" and
"server" nearby
SELECT D.name,
file_stream.GetFileNamespacePath() AS
relative_path
FROM dbo.Documents D
WHERE CONTAINS
(file_stream, 'NEAR(("sql", "server"), 1, FALSE)');
GO
Full-Text Search in Documents

• Full Text Catalog depend on language selected
Problems

SQL Server 2012 - Semantic Search

More Related Content

What's hot

What's hot (20)

Similar to SQL Server 2012 - Semantic Search

Similar to SQL Server 2012 - Semantic Search (20)

More from Sperasoft

More from Sperasoft (20)

Recently uploaded

Recently uploaded (20)

SQL Server 2012 - Semantic Search