New Features

• db86480:
  • LocalDataDriver for using CSV and pickled DF files as a QueryProvider (#64)
    This is primary for demonstration and test purposes where you do not have access to online data sources. It
    replicates the functionality of QueryProvider allowing drop-in replacement in existing notebooks.
  • Updated DataQueries.rst ReadtheDocs page with new queries
  • Add documentation for LocalDataDriver to DataProviders.rst and updated section on creating query files.
• 66a66d2:
  • Checked in notebook to create DataQueries.rst
  • Added "AzureSentinel" alias for LogAnalytics DataEnvironment

Breaking Changes

• db86480:
  • Removed deprecated kql.py, query_builtin_queries, query_mgr.py, query_schema.py
  • Changed location of query_defns.py and made pkg reference updates in several modules and notebooks.
  • Some fixes to support local_data_driver in query_store.py, driver_base.py and data_providers.py
  • Unit test - test_localdata_queries.yaml and supporting data and query files.
  • Fixed test in test_utils.py to work on Linux
  • Reduced warnings produced during pytest run to something more reasonable (mainly by removing
    deprecated code
• 8a32ad5:
  • Changed tilookup and kql_base/kql_driver so that handling failure to load is a bit friendlier. E.g. running
    TILookup in a non-IPython environment (with ASTI provider) will now just cause a warning, not an exception.
  • kql_driver.py also updated to check for get_ipython() returning None and output friendlier message.
  • Changed driver_base.py and derived class to take additional QuerySource parameter for query() method -
    not yet used but required so that we can implement driver-specific checks on query parameters.

This release includes:

Anomaly sequence analysis and visualization using Markov chain @karishma-dixit
Morph Chart visualization of log events @petebryan

New Features

• Anomalous sequences
  Markov Chain anomaly analysis for sequences of commands/patterns in a session
• Morph Charts visualization - 3D visualization of event data using experimental
  Morph Charts exploration
• nbinit: a neater and more robust startup/setup function for Jupyter notebooks
  handling package installs, imports and option setting
• Azure Sentinel Queries
  • Added two Logon fail queries for linux
  • Add Linux logons for host
• Added msticpy.common.pkg_config.validate_config() to validate current config
  or external config file

Fixes

• f78a29e:
  • Change return type on for bokeh graphs to return whole layout
  • Improved geoip error messages when Api key is missing
  • Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
• 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
• 827477b: make titles consistent on the widgets page (#59)
• 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
• f793d55:
  • Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
  • Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
  • Fixed an error in test-pypi-test-pkg.cmd
• 3e42e42: Doc fix and OutOfBoundsDatetime catch
• efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting
• 0ad166a: fixing headings in rst docs for timeseries
• 606fc8f: Fixing broken Readthedocs link (#53)
• 4810e1f: Fixing some documentation omissions/errors (#52)
• 43bbd3c: Updating pylintrc to change limits for some checks.
• f50eec2: Notebooklet queries and timeline hide option

This release includes:

• Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
• Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in
  Jupyter Notebooks using Bokeh Charts.
• Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers.
  Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in
  a future release.

New Features

• Azure data expansion and documentation
• Keyvault and keyring secrets management with support for multiple Azure clouds
  • config2kv.py KV secret update tool
• Timeseries - Bokeh with KQL and documentation
  • KQL generic time series decomposition queries
  • Bokeh time series visualization
• Added pandas version of get_whois_info and added as DataFrame accessor function.
• Added cmd script to test PyPi test deployment
• Added Conda package requirements files
• Updated TI providers to provide more consistent output and reduce false positives
  • Using text rather than number to express severity
• Made TISeverity class comparable and parsable from string or int
• Added mp_demo_data.py notebook helper to tools.
• SecurityAlert has more flexible recognition of entities
• Added additional dependencies for azure mgmt, keyvault and others.

Fixes

• Fixed get_ip_type ordering to return more accurate IP types
• Fix entity extraction in SecurityAlert to allow nested entities to work correctly
• Additional test cases

This is primarily a maintenance release with improvements to unit testing/code coverage. New features are early implementations of some features as pandas extensions so that you can invoke functionality directly from a DataFrame:
my_df.mp_timeline.plot()

Also, IoCExtract and Base64 decode functions have IPython magics (%%ioc and %%b64) allowing you to paste a block of text into a notebook cell and run the function directly on that text.

New Features

• process_tree - added pandas extension and changed main function so that it returns the plot figure and layout
• timeline - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
  than number)
• base64unpack - added pandas extension, added IPython %%b64 magic
• iocextract - added pandas extension, added IPython %%ioc magic
• Added documentation and notebook examples for the pandas extensions and magics.
• wsconfig - added method to displqy available workspaces
• README.MD - added some graphics to brighten the page up a little
• Added unit test test_folium.py
• Adding FoliumMap.ipynb sample notebook
• Added additional geolocation centering functions for FoliumMap
• Updates to GeoIPLookups.ipynb
• Add parameter checks to timeline.py and process_tree.py so that invalid *kwargs produced a helpful error message.
• Added requirements-dev.txt

Fixes

• Typos in AzureData.rst
• Adding GeoIP tests.
• Removing deprecated lines from coverage reports.
• Cleaned up pytest coverage report.
• Adding suppression file for credscan false positives
• Removing SecurityAlertandEntities notebook with misleading content
• Removed failing cell from end of GeoIPLookups notebook
• Fixed a few errors in foliummap.py
• Fixed bug in GeoIP DB downloader
• Changed foliummap center functions to use median by default
• Removed largely redundant os_family param from iocextract.py functions
• Fixed sectools_magics iocextract class
• Update test_ioc_extractor for new parameters
• domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
  http request if data on class instantiation)
• Added more tests for utility.py
• Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
• Tidied up/refactored some code in base64unpack.py

New Features

The documentation now includes a user guide covering many aspects of msticpy
It includes the following sections:

• Getting started section (Installation and configuration)
• Data Acquisition (querying and data)
• Data Enrichment (GeoIP, Threat Intel)
• Data Analysis (IoC extraction, decoding, clustering
• Visualization (Event timeline, Process tree, Mapping, widgets)

Documentation is on ReadTheDocs

Fixes

• Broken links and outdated docs updated
• Fixes to some unit tests