Hotfix release

Fixes

• 08b048d@If AzureCLI section is not in msticpconfig.yaml, no longer throws exception
• Updated AzureSentinel API notebook so that it has data (obfuscated).
• 3384c87@Updated .pre-commit-config.yaml to exclude tests from pylint and flake8 checks
  7b276a3@Adding azure-mgmt-core>=1.2.1 to requirements.txt
• Added exceptions to test_pkg_imports.py to skip vt and vt_graph_api (now extras)
• d3ad345@ Fixing to mordor_driver for Mitre web site change and making web scraping from Mitre more resilient
• Adding two scripts for testing pre-release msticpy.
• Some additions to VTLookupV3.ipynb text and trapping for import of nest_asyncio without installing
• ReadTheDocs conf.py now retrieves version number from main package build
• d8cc378@Updated ReadtheDocs requirements.txt
• db6e6f9@Changes to requirements,txt to remove unneeded dependencies (#114)*
• Added exclusions to conf.py and setup.py to skip install of some dependencies for ReadTheDocs (e.g. Kqlmagic).
  These packages will be mocked in their build - previously they caused dependency conflict errors in pip install
• Bug in azure_auth_core.py - raise from should have been an exception object, not a class
• Bug in secret_settings.py - potential uninitialized variable
• Updated test_mordor_driver.py to remove files on completion, Also marked file download tests as CI-only
• Added pylint and flake8 as pre-commit hooks - should stop fewer errors getting into build
• Some tidying of code (Sourcery) in azure_auth_core and secret_settings.
• Updated version to 0.8.7
• Fixing some typos and linting warnings in the documentation. Clarifying language.
• 3c87222@Updated vt-py version in requirements.txt (#113)

This release includes several new features:

• Support for VirusTotal v3 API
• Streamlined authentication for Azure using chained authentication
• Azure Sentinel API support
• Mordor data provider and data browser

New Features

• VirusTotal V3 API support (#97 and #106)
  The VirusTotal team contributed a new module that gives access to the latest version of their API. This
  includes building of relationships between malicious items and viewing the relationship graph.
  (thanks to Andres Ramirez/@aramirezmartin and Juan Infantes at VirusTotal!).
  See the VirusTotal V3 notebook
  To use this feature you will need to install msticpy with the "vt3" extra

   pip install msticpy[vt3]

• Azure Authentication (#109)
  We previously has several mechanism to authenticate to Azure components - mainly Key Vault and the Azure Data module.
  This caused unnecessary authentications. These now use a single mechanism and can also piggy-back on existing AzureCLI
  and Managed Service Identity (MSI) credentials. For example, if you have run az login from Azure CLI before launching
  your notebook the credentials from Azure CLI will be used to fetch the required authentication token for other
  Azure modules in MSTICPy. Note: this does not yet include Kqlmagic but we hope to have that in the next release.
  To see this working, check out the AzureSentinel API notebook link below.
• Azure Sentinel API Support (#109)
  This release includes initial support for Azure Sentinel APIs to retrieve Azure Sentinel Workspaces, Alert Rules,
  Hunting Queries and Hunting bookmarks (more to follow in a later release).
  See the Azure Sentinel API notebook
• Mordor data (#108)
  Mordor is an open source data repository of logs illustrating different kinds
  of adversary behavior and referenced to the Mitre ATT&CK framework. The MSTICPy
  Mordor package includes a notebook browser to search through and display the data sets and a dataprovider
  allowing you to download datasets into pandas DataFrames to use in your notebooks.
  See the Mordor Data notebook

Experimental Features

• Split queries by time period (#110)
  This allows you to split a very large or long-running query into subsets split along by time interval (e.g. 1day, 1hour, etc.)
  The DataProvider data queries now accept a split_query_by option to split queries by time period. The time period
  follows the pandas Timedelta syntax e.g. "1D", "6H". The query is divided, each piece run sequentially and the results
  re-assembled into a single DataFrame.

Fixes

1bce3f3@Bug fix and nasty workaround for old test setup removed in pkg_config.py
0a21b59@Updating pre-commit version for black
36d2539@Updated formatting for new black version
7897cb9@Moving VT3 dependencies to extras. (#111)

Hot fix for - GetEnvironmentString widget throws exception when variable is already defined

New Features

• #90 - new gettext and option button widgets.
• Registered widgets - some widgets (GetText and QueryTimes) now have a "memory" - so if used in Jupyter notebooks and you accidentally re-run the cell they will revert to their last settings rather than the defaults.
• Registered widgets can also accept notebook parameters from papermill
• Added query_browser module
• Added method to tilookup and data_providers modules to access respective browser widgets.
• #93 Added class diagrams for DataProviders, TIProviders and Secrets sub-packages.
• #94 - Added function to extract anomalous time ranges from time series anomaly results
• Added functions to obfuscate/hash data to protect PII
• #96 kql query failure and nbinit notebook initialization now have friendly errors
• Updated data queries list

Fixes

• #90 Splunk and data_providers fixes

• Tooltip formatting fix for datetimes in timeline and timeseries Bokeh tools

• #92 - Fixes for deprecated networkx API
• #93 Error in pkg_config validate when no config sections are populated.
• #95 - Removed redundant typing library from requirements
• #96 - Miscellaneous linter and spelling fixes
• Error in pkg_config validate when no config sections are populated.
• Updated Splunk queries to use datetime type for parameters.
• Adding some extra checks for null entries in msticpyconfig in pkg_config
• Making nbinit skip but report any exceptions while validating msticpyconfig
• Change dataproviders to use custom paths outside of the package
• Fix to entity entityschema
• Fix to kql_driver to handle running if not in IPython.
• Add version param to test-pypi-test-pkg.cmd help.
• Unit test for query_browser.
• Formatting using updated black v20
• Fixing conda-reqs-pip.txt requirements
• Updating msal req to ~=1.0.0
• Fixing mypy errors, incorrect annotation in query_source
• Fixing time format bug in timeline
• 3fc07e7@timeformat re-ordering
• 9d20842@updated requirements for dateutil bug-fixes

Maintenance release to fix import errors caused by incorrect package dependency versions.
Also includes minor documentation updates.

Maintenance release to fix some package version conflicts.

New Features

• db86480:
  • LocalDataDriver for using CSV and pickled DF files as a QueryProvider (#64)
    This is primary for demonstration and test purposes where you do not have access to online data sources. It
    replicates the functionality of QueryProvider allowing drop-in replacement in existing notebooks.
  • Updated DataQueries.rst ReadtheDocs page with new queries
  • Add documentation for LocalDataDriver to DataProviders.rst and updated section on creating query files.
• 66a66d2:
  • Checked in notebook to create DataQueries.rst
  • Added "AzureSentinel" alias for LogAnalytics DataEnvironment

Breaking Changes

• db86480:
  • Removed deprecated kql.py, query_builtin_queries, query_mgr.py, query_schema.py
  • Changed location of query_defns.py and made pkg reference updates in several modules and notebooks.
  • Some fixes to support local_data_driver in query_store.py, driver_base.py and data_providers.py
  • Unit test - test_localdata_queries.yaml and supporting data and query files.
  • Fixed test in test_utils.py to work on Linux
  • Reduced warnings produced during pytest run to something more reasonable (mainly by removing
    deprecated code
• 8a32ad5:
  • Changed tilookup and kql_base/kql_driver so that handling failure to load is a bit friendlier. E.g. running
    TILookup in a non-IPython environment (with ASTI provider) will now just cause a warning, not an exception.
  • kql_driver.py also updated to check for get_ipython() returning None and output friendlier message.
  • Changed driver_base.py and derived class to take additional QuerySource parameter for query() method -
    not yet used but required so that we can implement driver-specific checks on query parameters.

This release includes:

Anomaly sequence analysis and visualization using Markov chain @karishma-dixit
Morph Chart visualization of log events @petebryan

New Features

• Anomalous sequences
  Markov Chain anomaly analysis for sequences of commands/patterns in a session
• Morph Charts visualization - 3D visualization of event data using experimental
  Morph Charts exploration
• nbinit: a neater and more robust startup/setup function for Jupyter notebooks
  handling package installs, imports and option setting
• Azure Sentinel Queries
  • Added two Logon fail queries for linux
  • Add Linux logons for host
• Added msticpy.common.pkg_config.validate_config() to validate current config
  or external config file

Fixes

• f78a29e:
  • Change return type on for bokeh graphs to return whole layout
  • Improved geoip error messages when Api key is missing
  • Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
• 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
• 827477b: make titles consistent on the widgets page (#59)
• 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
• f793d55:
  • Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
  • Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
  • Fixed an error in test-pypi-test-pkg.cmd
• 3e42e42: Doc fix and OutOfBoundsDatetime catch
• efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting
• 0ad166a: fixing headings in rst docs for timeseries
• 606fc8f: Fixing broken Readthedocs link (#53)
• 4810e1f: Fixing some documentation omissions/errors (#52)
• 43bbd3c: Updating pylintrc to change limits for some checks.
• f50eec2: Notebooklet queries and timeline hide option

This release includes:

• Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
• Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in
  Jupyter Notebooks using Bokeh Charts.
• Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers.
  Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in
  a future release.

New Features

• Azure data expansion and documentation
• Keyvault and keyring secrets management with support for multiple Azure clouds
  • config2kv.py KV secret update tool
• Timeseries - Bokeh with KQL and documentation
  • KQL generic time series decomposition queries
  • Bokeh time series visualization
• Added pandas version of get_whois_info and added as DataFrame accessor function.
• Added cmd script to test PyPi test deployment
• Added Conda package requirements files
• Updated TI providers to provide more consistent output and reduce false positives
  • Using text rather than number to express severity
• Made TISeverity class comparable and parsable from string or int
• Added mp_demo_data.py notebook helper to tools.
• SecurityAlert has more flexible recognition of entities
• Added additional dependencies for azure mgmt, keyvault and others.

Fixes

• Fixed get_ip_type ordering to return more accurate IP types
• Fix entity extraction in SecurityAlert to allow nested entities to work correctly
• Additional test cases

This is primarily a maintenance release with improvements to unit testing/code coverage. New features are early implementations of some features as pandas extensions so that you can invoke functionality directly from a DataFrame:
my_df.mp_timeline.plot()

Also, IoCExtract and Base64 decode functions have IPython magics (%%ioc and %%b64) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

New Features

• process_tree - added pandas extension and changed main function so that it returns the plot figure and layout
• timeline - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
  than number)
• base64unpack - added pandas extension, added IPython %%b64 magic
• iocextract - added pandas extension, added IPython %%ioc magic
• Added documentation and notebook examples for the pandas extensions and magics.
• wsconfig - added method to displqy available workspaces
• README.MD - added some graphics to brighten the page up a little
• Added unit test test_folium.py
• Adding FoliumMap.ipynb sample notebook
• Added additional geolocation centering functions for FoliumMap
• Updates to GeoIPLookups.ipynb
• Add parameter checks to timeline.py and process_tree.py so that invalid *kwargs produced a helpful error message.
• Added requirements-dev.txt

Fixes

• Typos in AzureData.rst
• Adding GeoIP tests.
• Removing deprecated lines from coverage reports.
• Cleaned up pytest coverage report.
• Adding suppression file for credscan false positives
• Removing SecurityAlertandEntities notebook with misleading content
• Removed failing cell from end of GeoIPLookups notebook
• Fixed a few errors in foliummap.py
• Fixed bug in GeoIP DB downloader
• Changed foliummap center functions to use median by default
• Removed largely redundant os_family param from iocextract.py functions
• Fixed sectools_magics iocextract class
• Update test_ioc_extractor for new parameters
• domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
  http request if data on class instantiation)
• Added more tests for utility.py
• Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
• Tidied up/refactored some code in base64unpack.py

New Features

The documentation now includes a user guide covering many aspects of msticpy
It includes the following sections:

• Getting started section (Installation and configuration)
• Data Acquisition (querying and data)
• Data Enrichment (GeoIP, Threat Intel)
• Data Analysis (IoC extraction, decoding, clustering
• Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on ReadTheDocs

Fixes

• Broken links and outdated docs updated
• Fixes to some unit tests