+-
从 Python: tf-idf-cosine: to find document similarity开始,可以使用tf-idf余弦计算文档相似度.没有导入外部库,是否有任何方法可以计算2个字符串之间的余弦相似度?
s1 = "This is a foo bar sentence ."
s2 = "This sentence is similar to a foo bar sentence ."
s3 = "What is this string ? Totally not related to the other two lines ."
cosine_sim(s1, s2) # Should give high cosine similarity
cosine_sim(s1, s3) # Shouldn't give high cosine similarity value
cosine_sim(s2, s3) # Shouldn't give high cosine similarity value
最佳答案
一个简单的纯Python实现将是:
import re, math
from collections import Counter
WORD = re.compile(r'\w+')
def get_cosine(vec1, vec2):
intersection = set(vec1.keys()) & set(vec2.keys())
numerator = sum([vec1[x] * vec2[x] for x in intersection])
sum1 = sum([vec1[x]**2 for x in vec1.keys()])
sum2 = sum([vec2[x]**2 for x in vec2.keys()])
denominator = math.sqrt(sum1) * math.sqrt(sum2)
if not denominator:
return 0.0
else:
return float(numerator) / denominator
def text_to_vector(text):
words = WORD.findall(text)
return Counter(words)
text1 = 'This is a foo bar sentence .'
text2 = 'This sentence is similar to a foo bar sentence .'
vector1 = text_to_vector(text1)
vector2 = text_to_vector(text2)
cosine = get_cosine(vector1, vector2)
print 'Cosine:', cosine
打印:
Cosine: 0.861640436855
这里使用的余弦公式描述为here.
这不包括tf-idf对单词的加权,但是为了使用tf-idf,你需要有一个相当大的语料库来估计tfidf权重.
您还可以通过使用更复杂的方法从一段文本中提取单词,词干或对其进行词干化等来进一步开发它.
点击查看更多相关文章
转载注明原文:python – 给出2个句子字符串计算余弦相似度 - 乐贴网