gensim - Python库用于主题建模,文档索引和相似性检索大全集
gensim - Python库用于主题建模,文档索引和相似性检索大全集。目标受众是自然语言处理(NLP)和信息检索(IR)社区。 menshikh-iv released this
Assets
2
3.7.0, 2019-01-18
🌟
New features
-
Fast Online NMF (@anotherbugmaster, #2007)
-
Benchmark
wiki-english-20171001
Model Perplexity Coherence L2 norm Train time (minutes) LDA 4727.07 -2.514 7.372 138 NMF 975.74 -2.814 7.265 73 NMF (with regularization) 985.57 -2.436 7.269 441 -
Simple to use (same interface as
LdaModel
)from gensim.models.nmf import Nmf from gensim.corpora import Dictionary import gensim.downloader as api text8 = api.load('text8') dictionary = Dictionary(text8) dictionary.filter_extremes() corpus = [ dictionary.doc2bow(doc) for doc in text8 ] nmf = Nmf( corpus=corpus, num_topics=5, id2word=dictionary, chunksize=2000, passes=5, random_state=42, ) nmf.show_topics() """ [(0, '0.007*"km" + 0.006*"est" + 0.006*"islands" + 0.004*"league" + 0.004*"rate" + 0.004*"female" + 0.004*"economy" + 0.003*"male" + 0.003*"team" + 0.003*"elections"'), (1, '0.006*"actor" + 0.006*"player" + 0.004*"bwv" + 0.004*"writer" + 0.004*"actress" + 0.004*"singer" + 0.003*"emperor" + 0.003*"jewish" + 0.003*"italian" + 0.003*"prize"'), (2, '0.036*"college" + 0.007*"institute" + 0.004*"jewish" + 0.004*"universidad" + 0.003*"engineering" + 0.003*"colleges" + 0.003*"connecticut" + 0.003*"technical" + 0.003*"jews" + 0.003*"universities"'), (3, '0.016*"import" + 0.008*"insubstantial" + 0.007*"y" + 0.006*"soviet" + 0.004*"energy" + 0.004*"info" + 0.003*"duplicate" + 0.003*"function" + 0.003*"z" + 0.003*"jargon"'), (4, '0.005*"software" + 0.004*"games" + 0.004*"windows" + 0.003*"microsoft" + 0.003*"films" + 0.003*"apple" + 0.003*"video" + 0.002*"album" + 0.002*"fiction" + 0.002*"characters"')] """
-
See also:
-
-
Massive improvement of
FastText
compatibilities (@mpenkov, #2313)from gensim.models import FastText # 'cc.ru.300.bin' - Russian Facebook FT model trained on Common Crawl # Can be downloaded from https://s3-us-west-1.amazonaws.com/fasttext-vectors/word-vectors-v2/cc.ru.300.bin.gz model = FastText.load_fasttext_format("cc.ru.300.bin") # Fixed hash-function allow to produce same output as FB FastText & works correctly for non-latin languages (for example, Russian) assert "мяу" in m.wv.vocab # 'мяу' - vocab word model.wv.most_similar("мяу") """ [('Мяу', 0.6820122003555298), ('МЯУ', 0.6373013257980347), ('мяу-мяу', 0.593108594417572), ('кис-кис', 0.5899622440338135), ('гав', 0.5866007804870605), ('Кис-кис', 0.5798211097717285), ('Кис-кис-кис', 0.5742273330688477), ('Мяу-мяу', 0.5699705481529236), ('хрю-хрю', 0.5508339405059814), ('ав-ав', 0.5479759573936462)] """ assert "котогород" not in m.wv.vocab # 'котогород' - out-of-vocab word model.wv.most_similar("котогород", topn=3) """ [('автогород', 0.5463314652442932), ('ТагилНовокузнецкНовомосковскНовороссийскНовосибирскНовотроицкНовочеркасскНовошахтинскНовый', 0.5423436164855957), ('областьНовосибирскБарабинскБердскБолотноеИскитимКарасукКаргатКуйбышевКупиноОбьТатарскТогучинЧерепаново', 0.5377570390701294)] """ # Now we load full model, for this reason, we can continue an training from gensim.test.utils import datapath from smart_open import smart_open with smart_open(datapath("crime-and-punishment.txt"), encoding="utf-8") as infile: # russian text corpus = [line.strip().split() for line in infile] model.train(corpus, total_examples=len(corpus), epochs=5)
-
Similarity search improvements (@Witiko, #2016)
-
Add similarity search using the Levenshtein distance in
gensim.similarities.LevenshteinSimilarityIndex
-
Performance optimizations to
gensim.similarities.SoftCosineSimilarity
(full benchmark)dictionary size corpus size speed 1000 100 1.0× 1000 1000 53.4× 1000 100000 156784.8× 100000 100 3.8× 100000 1000 405.8× 100000 100000 66262.0× -
See updated soft-cosine tutorial for more information and usage examples
-
-
Add
python3.7
support (@menshikh-iv, #2211)- Wheels for Window, OSX and Linux platforms (@menshikh-iv, MacPython/gensim-wheels/#12)
- Faster installation
👍
Improvements
Optimizations
- Reduce
Phraser
memory usage (drop frequencies) (@jenishah, #2208) - Reduce memory consumption of summarizer (@horpto, #2298)
- Replace inline slow equivalent of mean_absolute_difference with fast (@horpto, #2284)
- Reuse precalculated updated prior in
ldamodel.update_dir_prior
(@horpto, #2274) - Improve
KeyedVector.wmdistance
(@horpto, #2326) - Optimize
remove_unreachable_nodes
ingensim.summarization
(@horpto, #2263) - Optimize
mz_entropy
fromgensim.summarization
(@horpto, #2267) - Improve
filter_extremes
methods inDictionary
andHashDictionary
(@horpto, #2303)
Additions
- Add
KeyedVectors.relative_cosine_similarity
(@rsdel2007, #2307) - Add
random_seed
toLdaMallet
(@Zohaggie & @menshikh-iv, #2153) - Add
common_terms
parameter tosklearn_api.PhrasesTransformer
(@pmlk, #2074) - Add method for patch
corpora.Dictionary
based on special tokens (@Froskekongen, #2200)
Cleanup
- Improve
six
usage (xrange
,map
,zip
) (@horpto, #2264) - Refactor
line2doc
methods ofLowCorpus
andMalletCorpus
(@horpto, #2269) - Get rid most of warnings in testing (@menshikh-iv, #2191)
- Fix non-deterministic test failures (pin
PYTHONHASHSEED
) (@menshikh-iv, #2196) - Fix "aliasing chunkize to chunkize_serial" warning on Windows (@aquatiko, #2202)
- Remove
__getitem__
code duplication ingensim.models.phrases
(@jenishah, #2206) - Add
flake8-rst
for docstring code examples (@kataev, #2192) - Get rid
py26
stuff (@menshikh-iv, #2214) - Use
itertools.chain
instead ofsum
to concatenate lists (@Stigjb, #2212) - Fix flake8 warnings W605, W504 (@horpto, #2256)
- Remove unnecessary creations of lists at all (@horpto, #2261)
- Fix extra list creation in
utils.get_max_id
(@horpto, #2254) - Fix deprecation warning
np.sum(generator)
(@rsdel2007, #2296) - Refactor
BM25
(@horpto, #2275) - Fix pyemd import (@ramprakash-94, #2240)
- Set
metadata=True
formake_wikicorpus
script by default (@Xinyi2016, #2245) - Remove unimportant warning from
Phrases
(@rsdel2007, #2331) - Replace
open()
bysmart_open()
ingensim.models.fasttext._load_fasttext_format
(@rsdel2007, #2335)
🔴
Bug fixes
- Fix overflow error for
*Vec
corpusfile-based training (@bm371613, #2239) - Fix
malletmodel2ldamodel
conversion (@horpto, #2288) - Replace custom epsilons with numpy equivalent in
LdaModel
(@horpto, #2308) - Add missing content to tarball (@menshikh-iv, #2194)
- Fixes divided by zero when w_star_count==0 (@allenyllee, #2259)
- Fix check for callbacks (@allenyllee, #2251)
- Fix
SvmLightCorpus.serialize
iflabels
instance of numpy.ndarray (@aquatiko, #2243) - Fix poincate viz incompatibility with
plotly>=3.0.0
(@jenishah, #2226) - Fix
keep_n
behavior forDictionary.filter_extremes
(@johann-petrak, #2232) - Fix for
sphinx==1.8.1
(last r (@menshikh-iv, #None) - Fix
np.issubdtype
warnings (@marioyc, #2210) - Drop wrong key
-c
fromgensim.downloader
description (@horpto, #2262) - Fix gensim build (docs & pyemd issues) (@menshikh-iv, #2318)
- Limit visdom version (avoid py2 issue from the latest visdom release) (@menshikh-iv, #2334)
- Fix visdom integration (using
viz.line()
instead ofviz.updatetrace()
) (@allenyllee, #2252)
📚
Tutorial and doc improvements
- Add gensim-data repo to
gensim.downloader
& fix rendering of code examples (@menshikh-iv, #2327) - Fix typos in
gensim.models
(@rsdel2007, #2323) - Fixed typos in notebooks (@rsdel2007, #2322)
- Update
Doc2Vec
documentation: how tags are assigned incorpus_file
mode (@persiyanov, #2320) - Fix typos in
gensim/models/keyedvectors.py
(@rsdel2007, #2290) - Add documentation about ranges to scoring functions for
Phrases
(@jenishah, #2242) - Update return sections for
KeyedVectors.evaluate_word_*
(@Stigjb, #2205) - Fix return type in
KeyedVector.evaluate_word_analogies
(@Stigjb, #2207) - Fix
WmdSimilarity
documentation (@jagmoreira, #2217) - Replace
fify -> fifty
ingensim.parsing.preprocessing.STOPWORDS
(@coderwassananmol, #2220) - Remove
alpha="auto"
fromLdaMulticore
(not supported yet) (@johann-petrak, #2225) - Update Adopters in README (@piskvorky, #2234)
- Fix broken link in
tutorials.md
(@rsdel2007, #2302)
⚠️
Deprecations (will be removed in the next major release)
-
Remove
gensim.models.wrappers.fasttext
(obsoleted by the new nativegensim.models.fasttext
implementation)gensim.examples
gensim.nosy
gensim.scripts.word2vec_standalone
gensim.scripts.make_wiki_lemma
gensim.scripts.make_wiki_online
gensim.scripts.make_wiki_online_lemma
gensim.scripts.make_wiki_online_nodebug
gensim.scripts.make_wiki
(all of these obsoleted by the new nativegensim.scripts.segment_wiki
implementation)- "deprecated" functions and attributes
-
Move
gensim.scripts.make_wikicorpus
➡gensim.scripts.make_wiki.py
gensim.summarization
➡gensim.models.summarization
gensim.topic_coherence
➡gensim.models._coherence
gensim.utils
➡gensim.utils.utils
(old imports will continue to work)gensim.parsing.*
➡gensim.utils.text_utils
Docs 💬
menshikh-iv released this
Assets
- 22.7 MB gensim-3.5.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 22.4 MB gensim-3.5.0-cp27-cp27m-manylinux1_i686.whl
- 22.4 MB gensim-3.5.0-cp27-cp27m-manylinux1_x86_64.whl
- 22.3 MB gensim-3.5.0-cp27-cp27m-win32.whl
- 22.4 MB gensim-3.5.0-cp27-cp27m-win_amd64.whl
- 22.4 MB gensim-3.5.0-cp27-cp27mu-manylinux1_i686.whl
- 22.4 MB gensim-3.5.0-cp27-cp27mu-manylinux1_x86_64.whl
- 22.6 MB gensim-3.5.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 22.4 MB gensim-3.5.0-cp35-cp35m-manylinux1_i686.whl
- 22.4 MB gensim-3.5.0-cp35-cp35m-manylinux1_x86_64.whl
- 22.3 MB gensim-3.5.0-cp35-cp35m-win32.whl
- 22.4 MB gensim-3.5.0-cp35-cp35m-win_amd64.whl
- 22.6 MB gensim-3.5.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 22.4 MB gensim-3.5.0-cp36-cp36m-manylinux1_i686.whl
- 22.4 MB gensim-3.5.0-cp36-cp36m-manylinux1_x86_64.whl
- 22.3 MB gensim-3.5.0-cp36-cp36m-win32.whl
- 22.4 MB gensim-3.5.0-cp36-cp36m-win_amd64.whl
- 21.9 MB gensim-3.5.0.tar.gz
- 22.6 MB gensim-3.5.0.win-amd64-py2.7.exe
- 22.9 MB gensim-3.5.0.win-amd64-py3.5.exe
- 22.9 MB gensim-3.5.0.win-amd64-py3.6.exe
- 22.5 MB gensim-3.5.0.win32-py2.7.exe
- 22.8 MB gensim-3.5.0.win32-py3.5.exe
- 22.8 MB gensim-3.5.0.win32-py3.6.exe
- Source code (zip)
- Source code (tar.gz)
3.5.0, 2018-07-06
This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs
Apart from the massive overhaul of all Gensim documentation (including docstring style and examples—you asked for it), we also managed to sneak in some new functionality and a number of bug fixes. As usual, see the notes below for a complete list, with links to pull requests for more details.
Huge thanks to all contributors! Nobody loves working on documentation. 3.5.0 is a result of several months of laborious, unglamorous, and sometimes invisible work. Enjoy!
📚
Documentation improvements
- Overhaul documentation for
*2vec
models (@steremma & @piskvorky & @menshikh-iv, #1944, #2087) - Fix documentation for LDA-related models (@steremma & @piskvorky & @menshikh-iv, #2026)
- Fix documentation for utils, corpora, inferfaces (@piskvorky & @menshikh-iv, #2096)
- Update non-API docs (about, intro, license etc) (@piskvorky & @menshikh-iv, #2101)
- Refactor documentation for
gensim.models.phrases
(@CLearERR & @menshikh-iv, #1950) - Fix HashDictionary documentation (@piskvorky, #2073)
- Fix docstrings for
gensim.models.AuthorTopicModel
(@souravsingh & @menshikh-iv, #1907) - Fix docstrings for HdpModel, lda_worker & lda_dispatcher (@gyanesh-m & @menshikh-iv, #1912)
- Fix format & links for
gensim.similarities.docsim
(@CLearERR & @menshikh-iv, #2030) - Remove duplication of class documentation for
IndexedCorpus
(@darindf, #2033) - Refactor documentation for
gensim.models.coherencemodel
(@CLearERR & @menshikh-iv, #1933) - Fix docstrings for
gensim.sklearn_api
(@steremma & @menshikh-iv, #1895) - Disable google-style docstring support (@menshikh-iv, #2106)
- Fix docstring of
gensim.models.KeyedVectors.similarity_matrix
(@Witiko, #1971) - Consistently use
smart_open()
instead ofopen()
in notebooks (@sharanry, #1812)
🌟
New features:
- Add
add_entity
method toKeyedVectors
to allow adding word vectors manually (@persiyanov, #1957) - Add inference for new unseen author to
AuthorTopicModel
(@Stamenov, #1766) - Add
evaluate_word_analogies
(will replaceaccuracy
) method toKeyedVectors
(@akutuzov, #1935) - Add Pivot Normalization to
TfidfModel
(@markroxor, #1780)
👍
Improvements
- Allow initialization with
max_final_vocab
in lieu ofmin_count
inWord2Vec
(@aneesh-joshi, #1915) - Add
dtype
argument forchunkize_serial
inLdaModel
(@darindf, #2027) - Increase performance in
Phrases.analyze_sentence
(@JonathanHourany, #2070) - Add
ns_exponent
parameter to control the negative sampling distribution for*2vec
models (@fernandocamargoti, #2093)
🔴
Bug fixes:
- Fix
Doc2Vec.infer_vector
+ notebook cleanup (@gojomo, #2103) - Fix linear decay for learning rate in
Doc2Vec.infer_vector
(@umangv, #2063) - Fix negative sampling floating-point error for `gensim.models.Poincare (@jayantj, #1959)
- Fix loading
word2vec
anddoc2vec
models saved using old Gensim versions (@manneshiva, #2012) - Fix
SoftCosineSimilarity.get_similarities
on corpora ssues/1955) (@Witiko, #1972) - Fix return dtype for
matutils.unitvec
according to input dtype (@o-P-o, #1992) - Fix passing empty dictionary to
gensim.corpora.WikiCorpus
(@steremma, #2042) - Fix bug in
Similarity.query_shards
in multiprocessing case (@bohea, #2044) - Fix SMART from TfidfModel for case when
df == "n"
(@PeteBleackley, #2021) - Fix OverflowError when loading a large term-document matrix in compiled MatrixMarket format (@arlenk, #2001)
- Update rules for removing table markup from Wikipedia dumps (@chaitaliSaini, #1954)
- Fix
_is_single
fromPhrases
for case when corpus is a NumPy array (@rmalouf, #1987) - Fix tests for
EuclideanKeyedVectors.similarity_matrix
(@Witiko, #1984) - Fix deprecated parameters in
D2VTransformer
andW2VTransformer
(@MritunjayMohitesh, #1945) - Fix
Doc2Vec.infer_vector
after loading oldDoc2Vec
(gensim<=3.2
)(@manneshiva, #1974) - Fix inheritance chain for
load_word2vec_format
(@DennisChen0307, #1968) - Update Keras version (avoid bug from
keras==2.1.5
) (@menshikh-iv, #1963)
⚠️
Deprecations (will be removed in the next major release)
-
Remove
gensim.models.wrappers.fasttext
(obsoleted by the new nativegensim.models.fasttext
implementation)gensim.examples
gensim.nosy
gensim.scripts.word2vec_standalone
gensim.scripts.make_wiki_lemma
gensim.scripts.make_wiki_online
gensim.scripts.make_wiki_online_lemma
gensim.scripts.make_wiki_online_nodebug
gensim.scripts.make_wiki
(all of these obsoleted by the new nativegensim.scripts.segment_wiki
implementation)- "deprecated" functions and attributes
-
Move
gensim.scripts.make_wikicorpus
➡gensim.scripts.make_wiki.py
gensim.summarization
➡gensim.models.summarization
gensim.topic_coherence
➡gensim.models._coherence
gensim.utils
➡gensim.utils.utils
(old imports will continue to work)gensim.parsing.*
➡gensim.utils.text_utils
3.4.0
menshikh-iv released this
Assets
- 21.8 MB gensim-3.4.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 21.5 MB gensim-3.4.0-cp27-cp27m-manylinux1_i686.whl
- 21.5 MB gensim-3.4.0-cp27-cp27m-manylinux1_x86_64.whl
- 21.5 MB gensim-3.4.0-cp27-cp27m-win32.whl
- 21.5 MB gensim-3.4.0-cp27-cp27m-win_amd64.whl
- 21.5 MB gensim-3.4.0-cp27-cp27mu-manylinux1_i686.whl
- 21.5 MB gensim-3.4.0-cp27-cp27mu-manylinux1_x86_64.whl
- 21.8 MB gensim-3.4.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 21.5 MB gensim-3.4.0-cp35-cp35m-manylinux1_i686.whl
- 21.5 MB gensim-3.4.0-cp35-cp35m-manylinux1_x86_64.whl
- 21.4 MB gensim-3.4.0-cp35-cp35m-win32.whl
- 21.5 MB gensim-3.4.0-cp35-cp35m-win_amd64.whl
- 21.8 MB gensim-3.4.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 21.5 MB gensim-3.4.0-cp36-cp36m-manylinux1_i686.whl
- 21.5 MB gensim-3.4.0-cp36-cp36m-manylinux1_x86_64.whl
- 21.5 MB gensim-3.4.0-cp36-cp36m-win32.whl
- 21.5 MB gensim-3.4.0-cp36-cp36m-win_amd64.whl
- 21.1 MB gensim-3.4.0.tar.gz
- 21.7 MB gensim-3.4.0.win-amd64-py2.7.exe
- 22 MB gensim-3.4.0.win-amd64-py3.5.exe
- 22 MB gensim-3.4.0.win-amd64-py3.6.exe
- 21.6 MB gensim-3.4.0.win32-py2.7.exe
- 21.9 MB gensim-3.4.0.win32-py3.5.exe
- 21.9 MB gensim-3.4.0.win32-py3.6.exe
- Source code (zip)
- Source code (tar.gz)
3.4.0, 2018-03-01
🌟
New features:
-
Massive optimizations of
gensim.models.LdaModel
: much faster training, using Cython. (@arlenk, #1767)-
Training benchmark
💥 dataset old LDA [sec] optimized LDA [sec] speed up nytimes 3473 1975 1.76x enron 774 437 1.77x -
This change affects all models that depend on
LdaModel
, such asLdaMulticore
,LdaSeqModel
,AuthorTopicModel
.
-
-
Huge speed-ups to corpus I/O with
MmCorpus
(Cython) (@arlenk, #1825)-
File reading benchmark
dataset file compressed? old MmReader [sec] optimized MmReader [sec] speed up enron no 22.3 2.6 8.7x yes 37.3 14.4 2.6x nytimes no 419.3 49.2 8.5x yes 686.2 275.1 2.5x text8 no 25.4 2.5 10.1x yes 41.9 17.0 2.5x -
Overall, a 2.5x speedup for compressed
.mm.gz
input and 8.5x🔥 🔥 🔥 for uncompressed plaintext.mm
.
-
-
Performance and memory optimization to
gensim.models.FastText
🚀 (@jbaiter, #1916)-
Benchmark (first 500,000 articles from English Wikipedia)
Metric old FastText optimized FastText improvement Training time (1 epoch) 4823.4s (80.38 minutes) 1873.6s (31.22 minutes) 2.57x Training time (full) 1h 26min 13s 36min 43s 2.35x Training words/sec 72,781 187,366 2.57x Training peak memory 5.2 GB 3.7 GB 1.4x -
Overall, a 2.5x speedup & memory usage reduced by 30%.
-
-
Implemented Soft Cosine Measure (@Witiko, #1827)
-
New method for assessing document similarity, a nice faster alternative to WMD, Word Mover's Distance
-
Benchmark
Technique MAP score Duration softcossim 45.99 1.24 sec wmd-relax 44.48 12.22 sec cossim 44.22 4.39 sec wmd-gensim 44.08 98.29 sec -
Soft Cosine notebook with detailed description, examples & benchmarks
-
Related papers:
-
👍
Improvements:
- New method to show the Gensim installation parameters:
python -m gensim.scripts.package_info --info
. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903) - Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
- Added
license
field tosetup.py
, allowing the use of tools likepip-licenses
(@nils-werner, #1909)
🔴
Bug fixes:
- Fix Python 3 compatibility for
gensim.corpora.UciCorpus.save_corpus
(@darindf, #1875) - Add
wv
property to KeyedVectors for backward compatibility. Fix #1882 (@manneshiva, #1884) - Fix deprecation warning from
inspect.getargspec
. Fix #1878 (@aneesh-joshi, #1887) - Add
LabeledSentence
togensim.models.doc2vec
for backward compatibility. Fix #1886 (@manneshiva, #1891) - Fix empty output bug in
Phrases
(when usingmodel[tokens]
twice). Fix #1401 (@sj29-innovate, #1853) - Fix type problems for
D2VTransformer.fit_transform
. Fix #1834 (@Utkarsh-Mishra-CIC, #1845) - Fix
datatype
parameter forKeyedVectors.load_word2vec_format
. Fix #1682 (@pushpankar, #1819) - Fix deprecated parameters in
doc2vec-lee
notebook (@TheFlash10, #1918) - Fix file-like closing bug in
gensim.corpora.MmCorpus
. Fix #1869 (@sj29-innovate, #1911) - Fix precision problem in
test_similarities.py
, no more FP fails. (@menshikh-iv, #1928) - Fix encoding in Lee corpus reader. (@menshikh-iv, #1931)
- Fix OOV pairs counter in
WordEmbeddingsKeyedVectors.evaluate_word_pairs
. (@akutuzov, #1934)
📚
Tutorial and doc improvements:
- Fix example block for
gensim.models.Word2Vec
(@nzw0301, #1870) - Fix
doc2vec-lee
notebook (@numericlee, #1870) - Store images from
README.md
directly in repository. Fix #1849 (@ibrahimsharaf, #1861) - Add windows venv activate command to
CONTRIBUTING.md
(@aneesh-joshi, #1880) - Add anaconda-cloud badge. Partial fix #1901 (@sharanry, #1905)
- Fix docstrings for lsi-related code (@steremma, #1892)
- Fix parameter description of
sg
parameter forgensim.models.word2vec
(@mdcclv, #1919) - Refactor documentation for
gensim.similarities.docsim
andMmCorpus-related
. (@CLearERR & @menshikh-iv, #1910) - Fix docstrings for
gensim.test.utils
(@yurkai & @menshikh-iv, #1904) - Refactor docstrings for
gensim.scripts
. Partial fix #1665 (@yurkai & @menshikh-iv, #1792) - Refactor API reference
gensim.corpora
. Partial fix #1671 (@CLearERR & @menshikh-iv, #1835) - Fix documentation for
gensim.models.wrappers
(@kakshay21 & @menshikh-iv, #1859) - Fix docstrings for
gensim.interfaces
(@yurkai & @menshikh-iv, #1913)
⚠️
Deprecations (will be removed in the next major release)
-
Remove
gensim.models.wrappers.fasttext
(obsoleted by the new nativegensim.models.fasttext
implementation)gensim.examples
gensim.nosy
gensim.scripts.word2vec_standalone
gensim.scripts.make_wiki_lemma
gensim.scripts.make_wiki_online
gensim.scripts.make_wiki_online_lemma
gensim.scripts.make_wiki_online_nodebug
gensim.scripts.make_wiki
(all of these obsoleted by the new nativegensim.scripts.segment_wiki
implementation)- "deprecated" functions and attributes
-
Move
gensim.scripts.make_wikicorpus
➡gensim.scripts.make_wiki.py
gensim.summarization
➡gensim.models.summarization
gensim.topic_coherence
➡gensim.models._coherence
gensim.utils
➡gensim.utils.utils
(old imports will continue to work)gensim.parsing.*
➡gensim.utils.text_utils
3.3.0
menshikh-iv released this
Assets
- 21.2 MB gensim-3.3.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 21.4 MB gensim-3.3.0-cp27-cp27m-manylinux1_i686.whl
- 21.4 MB gensim-3.3.0-cp27-cp27m-manylinux1_x86_64.whl
- 21.1 MB gensim-3.3.0-cp27-cp27m-win32.whl
- 21.1 MB gensim-3.3.0-cp27-cp27m-win_amd64.whl
- 21.4 MB gensim-3.3.0-cp27-cp27mu-manylinux1_i686.whl
- 21.4 MB gensim-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
- 21.2 MB gensim-3.3.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 21.4 MB gensim-3.3.0-cp35-cp35m-manylinux1_i686.whl
- 21.4 MB gensim-3.3.0-cp35-cp35m-manylinux1_x86_64.whl
- 21.1 MB gensim-3.3.0-cp35-cp35m-win32.whl
- 21.1 MB gensim-3.3.0-cp35-cp35m-win_amd64.whl
- 21.2 MB gensim-3.3.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
- 21.4 MB gensim-3.3.0-cp36-cp36m-manylinux1_i686.whl
- 21.4 MB gensim-3.3.0-cp36-cp36m-manylinux1_x86_64.whl
- 21.1 MB gensim-3.3.0-cp36-cp36m-win32.whl
- 21.1 MB gensim-3.3.0-cp36-cp36m-win_amd64.whl
- 20.9 MB gensim-3.3.0.tar.gz
- 21.3 MB gensim-3.3.0.win-amd64-py2.7.exe
- 21.7 MB gensim-3.3.0.win-amd64-py3.5.exe
- 21.7 MB gensim-3.3.0.win-amd64-py3.6.exe
- 21.3 MB gensim-3.3.0.win32-py2.7.exe
- 21.5 MB gensim-3.3.0.win32-py3.5.exe
- 21.5 MB gensim-3.3.0.win32-py3.6.exe
- Source code (zip)
- Source code (tar.gz)
3.3.0, 2018-01-02
-
Re-designed all "*2vec" implementations (@manneshiva, #1777)
- Modular organization of
Word2Vec
,Doc2Vec
,FastText
, etc ..., making it easier to add new models in the future and re-use code - Fully backward compatible (even with loading models stored by a previous Gensim version)
- Detailed documentation for the *2vec refactoring project
- Modular organization of
-
Improve
gensim.scripts.segment_wiki
by retaining interwiki links. Fix #1712
(@steremma, PR #1839)-
Optionally extract interlinks from Wikipedia pages (use the
--include-interlinks
option). This will output one additional JSON dict for each article:{ "interlinks": { "article title 1": "interlink text 1", "article title 2": "interlink text 2", ... } }
-
Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:
python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz
- Read this field from the
segment_wiki
output:
import json from smart_open import smart_open with smart_open("enwiki-latest.jsonl.gz") as infile: for doc in infile: doc = json.loads(doc) src_node = doc['title'] dst_nodes = doc['interlinks'].keys() print(u"Source node: {}".format(src_node)) print(u"Destination nodes: {}".format(u", ".join(dst_nodes))) break """ OUTPUT: Source node: Anarchism Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics
- Read this field from the
-
-
Add support for SMART notation for
TfidfModel
. Fix #1785 (@markroxor, #1791)- Natural extension of
TfidfModel
to allow different weighting and normalization schemesfrom gensim.corpora import Dictionary from gensim.models import TfidfModel import gensim.downloader as api data = api.load("text8") dct = Dictionary(data) corpus = [dct.doc2bow(line) for line in data] # Train Tfidf model using the SMART notation, smartirs="ntc" where # 'n' - natural term frequency # 't' - idf document frequency # 'c' - cosine normalization # # More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.html model = TfidfModel(corpus, id2word=dct, smartirs="ntc") vectorized_corpus = list(model[corpus])
- SMART Information Retrieval System (wiki)
- Natural extension of
-
Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)
- An easy way to preview the rendered documentation (especially, if don't use Linux)
- Go to "Details" link of CircleCI in your PR, click on the "Artifacts" tab, choose the HTML file that you want to view; a new tab will open with the rendered HTML page
- Integration with Github, to see the documentation directly from the pull request page
- Install a user-script plugin: greasemonkey (for firefox) or tampermonkey (for chrome)
- Add this user-script to the plugin
- Now you’ll see a new button "See CircleCI doc for this PR" in each PR in the Gensim repository. Click it to see the full rendered documentation.
- An easy way to preview the rendered documentation (especially, if don't use Linux)
- Fix import in
get_my_ip
. Fix #1771 (@darindf, #1772) - Fix tox.ini/setup.cfg configuration (@menshikh-iv, #1815)
- Fix formula in
gensim.summarization.bm25
. Fix #1828 (@sj29-innovate, #1833) - Fix the train method of
TranslationMatrix
(@robotcator, #1838) - Fix positional params used for
gensim.models.CoherenceModel
ingensim.models.callbacks
(@Alexjmsherman, #1823) - Fix parameter setting for
FastText.train
. Fix #1818 (@sj29-innovate, #1837) - Pin python2 explicitly for building documentation (@menshikh-iv, #1840)
- Remove dispatcher deadlock for distributed LDA (@darindf, #1817)
- Fix
score_function
fromLexicalEntailmentEvaluation
. Fix #1858 (@hachibaka, #1863) - Fix symmetrical case for hellinger distance. Fix #1854 (@caiyulun, #1860)
- Remove wrong logging at import. Fix #1706 (@menshikh-iv, #1871)
- Refactor documentation API Reference for
gensim.summarization
(@yurkai & @menshikh-iv, #1709) - Fix docstrings for
gensim.similarities.index
. Partial fix #1666 (@menshikh-iv, #1681) - Fix docstrings for
gensim.models.translation_matrix
(@KokuKUSIAKU & @menshikh-iv, #1806) - Fix docstrings for
gensim.models.rpmodel
(@jazzmuesli & @menshikh-iv, #1802) - Fix docstrings for
gensim.utils
(@kakshay21 & @menshikh-iv, #1797) - Fix docstrings for
gensim.matutils
(@Cheukting & @menshikh-iv, #1804) - Fix docstrings for
gensim.models.logentropy_model
(@minggli & @menshikh-iv, #1803) - Fix docstrings for
gensim.models.normmodel
(@AustenLamacraft & @menshikh-iv, #1805) - Refactor API reference
gensim.topic_coherence
. Fix #1669 (@CLearERR & @menshikh-iv, #1714) - Fix documentation for
gensim.corpora.dictionary
andgensim.corpora.hashdictionary
. Partial fix #1671 (@CLearERR & @menshikh-iv, #1814) - Fix documentation for
gensim.corpora
. Partial fix #1671 (@anotherbugmaster & @menshikh-iv, #1729) - Update banner in doc pages (@piskvorky, #1865)
- Fix errors in the doc2vec-lee notebook (@PeterHamilton, #1841)
- Add wordnet mammal train file for Poincare notebook (@jayantj, #1781)
- Update Poincare notebooks (#1774) (@jayantj, #1774)
- Update contributing guide. Fix #1786 (@menshikh-iv, #1793)
- Add
model_to_dict
one-liner to word2vec notebook. Fix #1269 (@kakshay21, #1776) - Add word embedding viz to word2vec notebook. Fix #1419 (@markroxor, #1800)
- Fix description of
sg
parameter forgensim.models.FastText
(@akutuzov, #1801) - Fix typo in
doc2vec-IMDB
. Fix #1788 (@apoorvaeternity, #1796) - Remove outdated bz2 examples from tutorials[2] (@menshikh-iv, #1868)
- Remove outdated
bz2
+MmCorpus
examples from tutorials (@menshikh-iv, #1867)
-
Remove
gensim.models.wrappers.fasttext
(obsoleted by the new nativegensim.models.fasttext
implementation)gensim.examples
gensim.nosy
gensim.scripts.word2vec_standalone
gensim.scripts.make_wiki_lemma
gensim.scripts.make_wiki_online
gensim.scripts.make_wiki_online_lemma
gensim.scripts.make_wiki_online_nodebug
gensim.scripts.make_wiki
(all of these obsoleted by the new nativegensim.scripts.segment_wiki
implementation)- "deprecated" functions and attributes
-
Move
gensim.scripts.make_wikicorpus
➡gensim.scripts.make_wiki.py
gensim.summarization
➡gensim.models.summarization
gensim.topic_coherence
➡gensim.models._coherence
gensim.utils
➡gensim.utils.utils
(old imports will continue to work)gensim.parsing.*
➡gensim.utils.text_utils
Watchers:471 |
Star:8663 |
Fork:3236 |
创建时间: 2011-02-10 15:43:04 |
最后Commits: 昨天 |
4f8c03c
3.7.1, 2019-01-31
FastText.load_fasttext_model
(@mpenkov, #2340)Doc2Vec.infer_vector
(@tobycheese, #2347)LdaSeqModel
(@horpto, #2360)process_result_queue
from cycle inLdaMulticore
(@horpto, #2358)LdaModel.do_mstep
(@horpto, #2344)FastTextKeyedVectors
usingKeyedVectors
(missing attributecompatible_hash
) (@menshikh-iv, #2349)WordEmbeddingsKeyedVectors.most_similar
(@Witiko, #2356)flake8==3.7.1
(@horpto, #2365)FastText
documentation (@mpenkov, #2353)Any*Vec
docstrings (@tobycheese, #2345)poincare
documentation to indicate the relation format (@AMR-KELEG, #2357)Remove
gensim.models.wrappers.fasttext
(obsoleted by the new nativegensim.models.fasttext
implementation)gensim.examples
gensim.nosy
gensim.scripts.word2vec_standalone
gensim.scripts.make_wiki_lemma
gensim.scripts.make_wiki_online
gensim.scripts.make_wiki_online_lemma
gensim.scripts.make_wiki_online_nodebug
gensim.scripts.make_wiki
(all of these obsoleted by the new nativegensim.scripts.segment_wiki
implementation)Move
gensim.scripts.make_wikicorpus
➡gensim.scripts.make_wiki.py
gensim.summarization
➡gensim.models.summarization
gensim.topic_coherence
➡gensim.models._coherence
gensim.utils
➡gensim.utils.utils
(old imports will continue to work)gensim.parsing.*
➡gensim.utils.text_utils