gensim - Python库用于主题建模,文档索引和相似性检索大全集

gensim - Python库用于主题建模,文档索引和相似性检索大全集。目标受众是自然语言处理(NLP)和信息检索(IR)社区。
详细内容 问题 256 同类相比4052 发布的版本 3.8.1
3.8.1
3.8.1
Latest release

@mpenkov mpenkov released this Sep 26, 2019 · 1 commit to develop since this release

3.8.1, 2019-09-23

🔴 Bug fixes

  • Fix usage of base_dir instead of BASE_DIR in _load_info in downloader. (movb, #2605)
  • Update the version of smart_open in the setup.py file (AMR-KELEG, #2582)
  • Properly handle unicode_errors arg parameter when loading a vocab file (wmtzk, #2570)
  • Catch loading older TfidfModels without smartirs (bnomis, #2559)
  • Fix bug where a module import set up logging, pin doctools for Py2 (piskvorky, #2552)

📚 Tutorial and doc improvements

👍 Improvements

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
Assets 2
3.8.0
3.8.0
Latest release

@mpenkov mpenkov released this Jul 8, 2019 · 5 commits to develop since this release

3.8.0, 2019-07-08

⚠️ 3.8.x will be the last gensim version to support Py2.7. Starting with 4.0.0, gensim will only support Py3.5 and above

🌟 New Features

  • Enable online training of Poincare models (koiizukag, #2505)
  • Make BM25 more scalable by adding support for generator inputs (saraswatmks, #2479)
  • Allow the Gensim dataset / pre-trained model downloader gensim.downloader to run offline, by introducing a local file cache (mpenkov, #2545)
  • Make the gensim.downloader target directory configurable (mpenkov, #2456)
  • Allow fast kNN document similarity retrieval with NMSLIB (masa3141, #2417)

🔴 Bug fixes

  • Fix smart_open deprecation warning globally (itayB, #2530)
  • Fix AppVeyor issues with Windows and Py2 (mpenkov, #2546)
  • Fix topn=0 versus topn=None bug in most_similar, accept topn of any integer type (Witiko, #2497)
  • Fix Python version check (charsyam, #2547)
  • Fix typo in FastText documentation (Guitaricet, #2518)
  • Fix "Market Matrix" to "Matrix Market" typo. (Shooter23, #2513)
  • Fix auto-generated hyperlinks in CHANGELOG.md (mpenkov, #2482)

📚 Tutorial and doc improvements

  • Generate documentation for the gensim.similarities.termsim module (Witiko, #2485)
  • Simplify the Support section in README (piskvorky, #2542)

👍 Improvements

  • Pin sklearn version for Py2, because sklearn dropped py2 support (mpenkov, #2510)

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
Assets 2
3.7.3
3.7.3
Latest release

@mpenkov mpenkov released this May 8, 2019 · 3 commits to develop since this release

3.7.3, 2019-05-06

🔴 Bug fixes

📚 Tutorial and doc improvements

👍 Improvements

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
Assets 2
3.7.2
3.7.2
Latest release

@mpenkov mpenkov released this Apr 10, 2019 · 2 commits to develop since this release

3.7.2, 2019-04-06

🌟 New Features

  • gensim.models.fasttext.load_facebook_model function: load full model (slower, more CPU/memory intensive, supports training continuation)

    >>> from gensim.test.utils import datapath
>>>
>>> cap_path = datapath("crime-and-punishment.bin")
>>> fb_model = load_facebook_model(cap_path)
>>>
>>> 'landlord' in fb_model.wv.vocab  # Word is out of vocabulary
False
>>> oov_term = fb_model.wv['landlord']
>>>
>>> 'landlady' in fb_model.wv.vocab  # Word is in the vocabulary
True
>>> iv_term = fb_model.wv['landlady']
>>>
>>> new_sent = [['lord', 'of', 'the', 'rings'], ['lord', 'of', 'the', 'flies']]
>>> fb_model.build_vocab(new_sent, update=True)
>>> fb_model.train(sentences=new_sent, total_examples=len(new_sent), epochs=5)

  • gensim.models.fasttext.load_facebook_vectors function: load embeddings only (faster, less CPU/memory usage, does not support training continuation)

    >>> fbkv = load_facebook_vectors(cap_path)
>>>
>>> 'landlord' in fbkv.vocab  # Word is out of vocabulary
False
>>> oov_vector = fbkv['landlord']
>>>
>>> 'landlady' in fbkv.vocab  # Word is in the vocabulary
True
>>> iv_vector = fbkv['landlady']

🔴 Bug fixes

  • Fix unicode error when loading FastText vocabulary (@mpenkov, #2390)
  • Avoid division by zero in fasttext_inner.pyx (@mpenkov, #2404)
  • Avoid incorrect filename inference when loading model (@mpenkov, #2408)
  • Handle invalid unicode when loading native FastText models (@mpenkov, #2411)
  • Avoid divide by zero when calculating vectors for terms with no ngrams (@mpenkov, #2411)

📚 Tutorial and doc improvements

👍 Improvements

⚠️ Changes in FastText behavior

Out-of-vocab word handling

To achieve consistency with the reference implementation from Facebook,
a FastText model will now always report any word, out-of-vocabulary or
not, as being in the model, and always return some vector for any word
looked-up. Specifically:

  1. 'any_word' in ft_model will always return True. Previously, it
    returned True only if the full word was in the vocabulary. (To test if a
    full word is in the known vocabulary, you can consult the wv.vocab
    property: 'any_word' in ft_model.wv.vocab will return False if the full
    word wasn't learned during model training.)
  2. ft_model['any_word'] will always return a vector. Previously, it
    raised KeyError for OOV words when the model had no vectors
    for any ngrams of the word.
  3. If no ngrams from the term are present in the model,
    or when no ngrams could be extracted from the term, a vector pointing
    to the origin will be returned. Previously, a vector of NaN (not a number)
    was returned as a consequence of a divide-by-zero problem.
  4. Models may use more more memory, or take longer for word-vector
    lookup, especially after training on smaller corpuses where the previous
    non-compliant behavior discarded some ngrams from consideration.

Loading models in Facebook .bin format

The gensim.models.FastText.load_fasttext_format function (deprecated) now loads the entire model contained in the .bin file, including the shallow neural network that enables training continuation.
Loading this NN requires more CPU and RAM than previously required.

Since this function is deprecated, consider using one of its alternatives (see below).

Furthermore, you must now pass the full path to the file to load, including the file extension.
Previously, if you specified a model path that ends with anything other than .bin, the code automatically appended .bin to the path before loading the model.
This behavior was confusing, so we removed it.

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use load_facebook_vectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and load_facebook_model to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
Assets 2
3.7.1
3.7.1
Latest release

@menshikh-iv menshikh-iv released this Jan 31, 2019 · 1 commit to develop since this release

Assets 33

3.7.1, 2019-01-31

👍 Improvements

🔴 Bug fixes

  • Fix infinite diff in LdaModel.do_mstep (@horpto, #2344)
  • Fix backward compatibility issue: loading FastTextKeyedVectors using KeyedVectors (missing attribute compatible_hash) (@menshikh-iv, #2349)
  • Fix logging issue (conda-forge related) (@menshikh-iv, #2339)
  • Fix WordEmbeddingsKeyedVectors.most_similar (@Witiko, #2356)
  • Fix issues of flake8==3.7.1 (@horpto, #2365)

📚 Tutorial and doc improvements

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
3.7.0
3.7.0
Latest release

@menshikh-iv menshikh-iv released this Jan 18, 2019 · 1 commit to develop since this release

Assets 2

3.7.0, 2019-01-18

🌟 New features

  • Fast Online NMF (@anotherbugmaster, #2007)

    • Benchmark wiki-english-20171001

      Model Perplexity Coherence L2 norm Train time (minutes)
      LDA 4727.07 -2.514 7.372 138
      NMF 975.74 -2.814 7.265 73
      NMF (with regularization) 985.57 -2.436 7.269 441

    • Simple to use (same interface as LdaModel)

      from gensim.models.nmf import Nmf
from gensim.corpora import Dictionary
import gensim.downloader as api

text8 = api.load('text8')

dictionary = Dictionary(text8)
dictionary.filter_extremes()

corpus = [
    dictionary.doc2bow(doc) for doc in text8
]

nmf = Nmf(
    corpus=corpus,
    num_topics=5,
    id2word=dictionary,
    chunksize=2000,
    passes=5,
    random_state=42,
)

nmf.show_topics()
"""
[(0, '0.007*"km" + 0.006*"est" + 0.006*"islands" + 0.004*"league" + 0.004*"rate" + 0.004*"female" + 0.004*"economy" + 0.003*"male" + 0.003*"team" + 0.003*"elections"'),
 (1, '0.006*"actor" + 0.006*"player" + 0.004*"bwv" + 0.004*"writer" + 0.004*"actress" + 0.004*"singer" + 0.003*"emperor" + 0.003*"jewish" + 0.003*"italian" + 0.003*"prize"'),
 (2, '0.036*"college" + 0.007*"institute" + 0.004*"jewish" + 0.004*"universidad" + 0.003*"engineering" + 0.003*"colleges" + 0.003*"connecticut" + 0.003*"technical" + 0.003*"jews" + 0.003*"universities"'),
 (3, '0.016*"import" + 0.008*"insubstantial" + 0.007*"y" + 0.006*"soviet" + 0.004*"energy" + 0.004*"info" + 0.003*"duplicate" + 0.003*"function" + 0.003*"z" + 0.003*"jargon"'),
 (4, '0.005*"software" + 0.004*"games" + 0.004*"windows" + 0.003*"microsoft" + 0.003*"films" + 0.003*"apple" + 0.003*"video" + 0.002*"album" + 0.002*"fiction" + 0.002*"characters"')]
"""

    • See also:

  • Massive improvement of FastText compatibilities (@mpenkov, #2313)

    from gensim.models import FastText

# 'cc.ru.300.bin' - Russian Facebook FT model trained on Common Crawl
# Can be downloaded from https://s3-us-west-1.amazonaws.com/fasttext-vectors/word-vectors-v2/cc.ru.300.bin.gz

model = FastText.load_fasttext_format("cc.ru.300.bin")

# Fixed hash-function allow to produce same output as FB FastText & works correctly for non-latin languages (for example, Russian)
assert "мяу" in m.wv.vocab  # 'мяу' - vocab word
model.wv.most_similar("мяу")
"""
[('Мяу', 0.6820122003555298),
 ('МЯУ', 0.6373013257980347),
 ('мяу-мяу', 0.593108594417572),
 ('кис-кис', 0.5899622440338135),
 ('гав', 0.5866007804870605),
 ('Кис-кис', 0.5798211097717285),
 ('Кис-кис-кис', 0.5742273330688477),
 ('Мяу-мяу', 0.5699705481529236),
 ('хрю-хрю', 0.5508339405059814),
 ('ав-ав', 0.5479759573936462)]
"""

assert "котогород" not in m.wv.vocab  # 'котогород' - out-of-vocab word
model.wv.most_similar("котогород", topn=3)
"""
[('автогород', 0.5463314652442932),
 ('ТагилНовокузнецкНовомосковскНовороссийскНовосибирскНовотроицкНовочеркасскНовошахтинскНовый',
  0.5423436164855957),
 ('областьНовосибирскБарабинскБердскБолотноеИскитимКарасукКаргатКуйбышевКупиноОбьТатарскТогучинЧерепаново',
  0.5377570390701294)]
"""

# Now we load full model, for this reason, we can continue an training

from gensim.test.utils import datapath
from smart_open import smart_open

with smart_open(datapath("crime-and-punishment.txt"), encoding="utf-8") as infile:  # russian text
    corpus = [line.strip().split() for line in infile]

model.train(corpus, total_examples=len(corpus), epochs=5)

  • Similarity search improvements (@Witiko, #2016)

    • Add similarity search using the Levenshtein distance in gensim.similarities.LevenshteinSimilarityIndex

    • Performance optimizations to gensim.similarities.SoftCosineSimilarity (full benchmark)

      dictionary size corpus size speed
      1000 100 1.0×
      1000 1000 53.4×
      1000 100000 156784.8×
      100000 100 3.8×
      100000 1000 405.8×
      100000 100000 66262.0×

    • See updated soft-cosine tutorial for more information and usage examples

  • Add python3.7 support (@menshikh-iv, #2211)

👍 Improvements

Optimizations
  • Reduce Phraser memory usage (drop frequencies) (@jenishah, #2208)
  • Reduce memory consumption of summarizer (@horpto, #2298)
  • Replace inline slow equivalent of mean_absolute_difference with fast (@horpto, #2284)
  • Reuse precalculated updated prior in ldamodel.update_dir_prior (@horpto, #2274)
  • Improve KeyedVector.wmdistance (@horpto, #2326)
  • Optimize remove_unreachable_nodes in gensim.summarization (@horpto, #2263)
  • Optimize mz_entropy from gensim.summarization (@horpto, #2267)
  • Improve filter_extremes methods in Dictionary and HashDictionary (@horpto, #2303)
Additions
Cleanup

🔴 Bug fixes

📚 Tutorial and doc improvements

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
3.5.0

Docs 💬

@menshikh-iv menshikh-iv released this Jul 6, 2018 · 1 commit to develop since this release

Assets

3.5.0, 2018-07-06

This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs 💬 "!

Apart from the massive overhaul of all Gensim documentation (including docstring style and examples—you asked for it), we also managed to sneak in some new functionality and a number of bug fixes. As usual, see the notes below for a complete list, with links to pull requests for more details.

Huge thanks to all contributors! Nobody loves working on documentation. 3.5.0 is a result of several months of laborious, unglamorous, and sometimes invisible work. Enjoy!

📚 Documentation improvements

🌟 New features:

  • Add add_entity method to KeyedVectors to allow adding word vectors manually (@persiyanov, #1957)
  • Add inference for new unseen author to AuthorTopicModel (@Stamenov, #1766)
  • Add evaluate_word_analogies (will replace accuracy) method to KeyedVectors (@akutuzov, #1935)
  • Add Pivot Normalization to TfidfModel (@markroxor, #1780)

👍 Improvements

🔴 Bug fixes:

  • Fix Doc2Vec.infer_vector + notebook cleanup (@gojomo, #2103)
  • Fix linear decay for learning rate in Doc2Vec.infer_vector (@umangv, #2063)
  • Fix negative sampling floating-point error for `gensim.models.Poincare (@jayantj, #1959)
  • Fix loading word2vec and doc2vec models saved using old Gensim versions (@manneshiva, #2012)
  • Fix SoftCosineSimilarity.get_similarities on corpora ssues/1955) (@Witiko, #1972)
  • Fix return dtype for matutils.unitvec according to input dtype (@o-P-o, #1992)
  • Fix passing empty dictionary to gensim.corpora.WikiCorpus (@steremma, #2042)
  • Fix bug in Similarity.query_shards in multiprocessing case (@bohea, #2044)
  • Fix SMART from TfidfModel for case when df == "n" (@PeteBleackley, #2021)
  • Fix OverflowError when loading a large term-document matrix in compiled MatrixMarket format (@arlenk, #2001)
  • Update rules for removing table markup from Wikipedia dumps (@chaitaliSaini, #1954)
  • Fix _is_single from Phrases for case when corpus is a NumPy array (@rmalouf, #1987)
  • Fix tests for EuclideanKeyedVectors.similarity_matrix (@Witiko, #1984)
  • Fix deprecated parameters in D2VTransformer and W2VTransformer(@MritunjayMohitesh, #1945)
  • Fix Doc2Vec.infer_vector after loading old Doc2Vec (gensim<=3.2)(@manneshiva, #1974)
  • Fix inheritance chain for load_word2vec_format (@DennisChen0307, #1968)
  • Update Keras version (avoid bug from keras==2.1.5) (@menshikh-iv, #1963)

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
3.4.0

3.4.0

@menshikh-iv menshikh-iv released this Mar 1, 2018 · 1 commit to develop since this release

Assets

3.4.0, 2018-03-01

🌟 New features:

👍 Improvements:

  • New method to show the Gensim installation parameters: python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903)
  • Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
  • Added license field to setup.py, allowing the use of tools like pip-licenses (@nils-werner, #1909)

🔴 Bug fixes:

📚 Tutorial and doc improvements:

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
3.3.0

3.3.0

@menshikh-iv menshikh-iv released this Feb 2, 2018 · 1 commit to develop since this release

Assets

3.3.0, 2018-01-02

🌟 New features:

  • Re-designed all "*2vec" implementations (@manneshiva, #1777)

  • Improve gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712
    (@steremma, PR #1839)

    • Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article:

      {
    "interlinks": {
        "article title 1": "interlink text 1",
        "article title 2": "interlink text 2",
        ...
    }
}

    • Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:

      python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz
      • Read this field from the segment_wiki output:
      import json
from smart_open import smart_open

with smart_open("enwiki-latest.jsonl.gz") as infile:
    for doc in infile:
        doc = json.loads(doc)

        src_node = doc['title']
        dst_nodes = doc['interlinks'].keys()

        print(u"Source node: {}".format(src_node))
        print(u"Destination nodes: {}".format(u", ".join(dst_nodes)))
        break

"""
OUTPUT:

Source node: Anarchism
Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics

  • Add support for SMART notation for TfidfModel. Fix #1785 (@markroxor, #1791)

    • Natural extension of TfidfModel to allow different weighting and normalization schemes 
      from gensim.corpora import Dictionary
from gensim.models import TfidfModel
import gensim.downloader as api

data = api.load("text8")
dct = Dictionary(data)
corpus = [dct.doc2bow(line) for line in data]

# Train Tfidf model using the SMART notation, smartirs="ntc" where
# 'n' - natural term frequency
# 't' - idf document frequency
# 'c' - cosine normalization
#
# More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.html

model = TfidfModel(corpus, id2word=dct, smartirs="ntc")
vectorized_corpus = list(model[corpus])
    • SMART Information Retrieval System (wiki)

  • Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)

    • An easy way to preview the rendered documentation (especially, if don't use Linux)
      • Go to "Details" link of CircleCI in your PR, click on the "Artifacts" tab, choose the HTML file that you want to view; a new tab will open with the rendered HTML page
    • Integration with Github, to see the documentation directly from the pull request page

🔴 Bug fixes:

📚 Tutorial and doc improvements:

👍 Improvements:

⚠️ Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils
热门度与活跃度
10.0 6.3
访问主页 访问GitHub主页
Watchers：464
Star：9990
Fork：3604
创建时间： 2011-02-10 15:43:04
最后Commits： 前天
RaRe-Technologies
RARE Technologies
GitHub作者资料
基本信息
分类：机器学习 / Python开发
收录时间：2016-09-10 16:55:26