Microsoft Threat Intelligence Python Security Tools.
Microsoft Threat Intelligence Python Security Tools.This release includes several new features:
- Support for VirusTotal v3 API
- Streamlined authentication for Azure using chained authentication
- Azure Sentinel API support
- Mordor data provider and data browser
New Features
- VirusTotal V3 API support (#97 and #106)
The VirusTotal team contributed a new module that gives access to the latest version of their API. This
includes building of relationships between malicious items and viewing the relationship graph.
(thanks to Andres Ramirez/@aramirezmartin and Juan Infantes at VirusTotal!).
See the VirusTotal V3 notebook
To use this feature you will need to install msticpy with the "vt3" extra
pip install msticpy[vt3]
- Azure Authentication (#109)
We previously has several mechanism to authenticate to Azure components - mainly Key Vault and the Azure Data module.
This caused unnecessary authentications. These now use a single mechanism and can also piggy-back on existing AzureCLI
and Managed Service Identity (MSI) credentials. For example, if you have runaz login
from Azure CLI before launching
your notebook the credentials from Azure CLI will be used to fetch the required authentication token for other
Azure modules in MSTICPy. Note: this does not yet include Kqlmagic but we hope to have that in the next release.
To see this working, check out the AzureSentinel API notebook link below. - Azure Sentinel API Support (#109)
This release includes initial support for Azure Sentinel APIs to retrieve Azure Sentinel Workspaces, Alert Rules,
Hunting Queries and Hunting bookmarks (more to follow in a later release).
See the Azure Sentinel API notebook - Mordor data (#108)
Mordor is an open source data repository of logs illustrating different kinds
of adversary behavior and referenced to the Mitre ATT&CK framework. The MSTICPy
Mordor package includes a notebook browser to search through and display the data sets and a dataprovider
allowing you to download datasets into pandas DataFrames to use in your notebooks.
See the Mordor Data notebook
Experimental Features
- Split queries by time period (#110)
This allows you to split a very large or long-running query into subsets split along by time interval (e.g. 1day, 1hour, etc.)
The DataProvider data queries now accept asplit_query_by
option to split queries by time period. The time period
follows the pandas Timedelta syntax e.g. "1D", "6H". The query is divided, each piece run sequentially and the results
re-assembled into a single DataFrame.
Fixes
1bce3f3@Bug fix and nasty workaround for old test setup removed in pkg_config.py
0a21b59@Updating pre-commit version for black
36d2539@Updated formatting for new black version
7897cb9@Moving VT3 dependencies to extras. (#111)
Assets
2
New Features
- #90 - new gettext and option button widgets.
- Registered widgets - some widgets (GetText and QueryTimes) now have a "memory" - so if used in Jupyter notebooks and you accidentally re-run the cell they will revert to their last settings rather than the defaults.
- Registered widgets can also accept notebook parameters from papermill
- Added query_browser module
- Added method to tilookup and data_providers modules to access respective browser widgets.
- #93 Added class diagrams for DataProviders, TIProviders and Secrets sub-packages.
- #94 - Added function to extract anomalous time ranges from time series anomaly results
- Added functions to obfuscate/hash data to protect PII
- #96 kql query failure and nbinit notebook initialization now have friendly errors
- Updated data queries list
Fixes
- #90 Splunk and data_providers fixes
- Tooltip formatting fix for datetimes in timeline and timeseries Bokeh tools
- #92 - Fixes for deprecated networkx API
- #93 Error in pkg_config validate when no config sections are populated.
- #95 - Removed redundant typing library from requirements
- #96 - Miscellaneous linter and spelling fixes
- Error in pkg_config validate when no config sections are populated.
- Updated Splunk queries to use datetime type for parameters.
- Adding some extra checks for null entries in msticpyconfig in pkg_config
- Making nbinit skip but report any exceptions while validating msticpyconfig
- Change dataproviders to use custom paths outside of the package
- Fix to entity entityschema
- Fix to kql_driver to handle running if not in IPython.
- Add version param to test-pypi-test-pkg.cmd help.
- Unit test for query_browser.
- Formatting using updated black v20
- Fixing conda-reqs-pip.txt requirements
- Updating msal req to ~=1.0.0
- Fixing mypy errors, incorrect annotation in query_source
- Fixing time format bug in timeline
- 3fc07e7@timeformat re-ordering
- 9d20842@updated requirements for dateutil bug-fixes
Assets
2
Maintenance release to fix import errors caused by incorrect package dependency versions.
Also includes minor documentation updates.
Assets
2
ianhelle released this
Maintenance release to fix some package version conflicts.
Assets
2
New Features
- db86480:
- LocalDataDriver for using CSV and pickled DF files as a QueryProvider (#64)
This is primary for demonstration and test purposes where you do not have access to online data sources. It
replicates the functionality of QueryProvider allowing drop-in replacement in existing notebooks. - Updated DataQueries.rst ReadtheDocs page with new queries
- Add documentation for LocalDataDriver to DataProviders.rst and updated section on creating query files.
- LocalDataDriver for using CSV and pickled DF files as a QueryProvider (#64)
- 66a66d2:
- Checked in notebook to create DataQueries.rst
- Added "AzureSentinel" alias for LogAnalytics DataEnvironment
Breaking Changes
- db86480:
- Removed deprecated kql.py, query_builtin_queries, query_mgr.py, query_schema.py
- Changed location of query_defns.py and made pkg reference updates in several modules and notebooks.
- Some fixes to support local_data_driver in query_store.py, driver_base.py and data_providers.py
- Unit test - test_localdata_queries.yaml and supporting data and query files.
- Fixed test in test_utils.py to work on Linux
- Reduced warnings produced during pytest run to something more reasonable (mainly by removing
deprecated code
- 8a32ad5:
- Changed tilookup and kql_base/kql_driver so that handling failure to load is a bit friendlier. E.g. running
TILookup in a non-IPython environment (with ASTI provider) will now just cause a warning, not an exception. - kql_driver.py also updated to check for get_ipython() returning None and output friendlier message.
- Changed driver_base.py and derived class to take additional QuerySource parameter for query() method -
not yet used but required so that we can implement driver-specific checks on query parameters.
- Changed tilookup and kql_base/kql_driver so that handling failure to load is a bit friendlier. E.g. running
Assets
2
ianhelle released this
This release includes:
Anomaly sequence analysis and visualization using Markov chain @karishma-dixit
Morph Chart visualization of log events @petebryan
New Features
- Anomalous sequences
Markov Chain anomaly analysis for sequences of commands/patterns in a session - Morph Charts visualization - 3D visualization of event data using experimental
Morph Charts exploration - nbinit: a neater and more robust startup/setup function for Jupyter notebooks
handling package installs, imports and option setting - Azure Sentinel Queries
- Added two Logon fail queries for linux
- Add Linux logons for host
- Added msticpy.common.pkg_config.validate_config() to validate current config
or external config file
Fixes
- f78a29e:
- Change return type on for bokeh graphs to return whole layout
- Improved geoip error messages when Api key is missing
- Fixing bug in pkg_config if no workspaces are defined (empty workspaces key)
- 31cb17f: Added context manager to temporarily set msticpyconfig to another path and auto-revert settings afterwards.
- 827477b: make titles consistent on the widgets page (#59)
- 7964b5f: Fix to utility.py - check_and_install_missing_packages to all package version to be specified.
- f793d55:
- Updated pkg_config to allow AzureCLI and AzureSentinel sections to use Key Vault protection of the keys and use of Env Vars, etc.
- Timeline - fixed Tooltip representation of Timestamps for different representations of numpy's types
- Fixed an error in test-pypi-test-pkg.cmd
- 3e42e42: Doc fix and OutOfBoundsDatetime catch
- efc3d69: OTX TI Provider fixes to encode URL IoC prior to submitting
- 0ad166a: fixing headings in rst docs for timeseries
- 606fc8f: Fixing broken Readthedocs link (#53)
- 4810e1f: Fixing some documentation omissions/errors (#52)
- 43bbd3c: Updating pylintrc to change limits for some checks.
- f50eec2: Notebooklet queries and timeline hide option
Assets
2
ianhelle released this
This release includes:
- Expansion of Azure Data API for retrieving additional data about subscriptions and resources from Azure APIs.
- Time Series anomaly detection for arbitrary Kusto data sets together with visualization of time series charts in
Jupyter Notebooks using Bokeh Charts. - Using KeyVault and Python Keyring to store secrets used to authenticate to web data providers.
Examples include API keys for Threat Intel and Geo IP Providers. Other provider types will be included in
a future release.
New Features
- Azure data expansion and documentation
- Keyvault and keyring secrets management with support for multiple Azure clouds
- config2kv.py KV secret update tool
- Timeseries - Bokeh with KQL and documentation
- KQL generic time series decomposition queries
- Bokeh time series visualization
- Added pandas version of get_whois_info and added as DataFrame accessor function.
- Added cmd script to test PyPi test deployment
- Added Conda package requirements files
- Updated TI providers to provide more consistent output and reduce false positives
- Using text rather than number to express severity
- Made TISeverity class comparable and parsable from string or int
- Added mp_demo_data.py notebook helper to tools.
- SecurityAlert has more flexible recognition of entities
- Added additional dependencies for azure mgmt, keyvault and others.
Fixes
- Fixed get_ip_type ordering to return more accurate IP types
- Fix entity extraction in SecurityAlert to allow nested entities to work correctly
- Additional test cases
Assets
2
This is primarily a maintenance release with improvements to unit testing/code coverage. New features are early implementations of some features as pandas extensions so that you can invoke functionality directly from a DataFrame:
my_df.mp_timeline.plot()
Also, IoCExtract and Base64 decode functions have IPython magics (%%ioc
and %%b64
) allowing you to paste a block of text into a notebook cell and run the function directly on that text.
New Features
- process_tree - added pandas extension and changed main function so that it returns the plot figure and layout
- timeline - added pandas extension. added support for DateTime column in Tooltips (display as date time rather
than number) - base64unpack - added pandas extension, added IPython
%%b64
magic - iocextract - added pandas extension, added IPython
%%ioc
magic - Added documentation and notebook examples for the pandas extensions and magics.
- wsconfig - added method to displqy available workspaces
- README.MD - added some graphics to brighten the page up a little
- Added unit test test_folium.py
- Adding FoliumMap.ipynb sample notebook
- Added additional geolocation centering functions for FoliumMap
- Updates to GeoIPLookups.ipynb
- Add parameter checks to timeline.py and process_tree.py so that invalid *kwargs produced a helpful error message.
- Added requirements-dev.txt
Fixes
- Typos in AzureData.rst
- Adding GeoIP tests.
- Removing deprecated lines from coverage reports.
- Cleaned up pytest coverage report.
- Adding suppression file for credscan false positives
- Removing SecurityAlertandEntities notebook with misleading content
- Removed failing cell from end of GeoIPLookups notebook
- Fixed a few errors in foliummap.py
- Fixed bug in GeoIP DB downloader
- Changed foliummap center functions to use median by default
- Removed largely redundant os_family param from iocextract.py functions
- Fixed sectools_magics iocextract class
- Update test_ioc_extractor for new parameters
- domain_tools - changed tld_index and ssl_bl attributes to properties that auto-load on first use (prevents remote
http request if data on class instantiation) - Added more tests for utility.py
- Add environment variable to selectively run some long-duration tests during build only (these are no skipped in local tests)
- Tidied up/refactored some code in base64unpack.py
Assets
2
New Features
The documentation now includes a user guide covering many aspects of msticpy
It includes the following sections:
- Getting started section (Installation and configuration)
- Data Acquisition (querying and data)
- Data Enrichment (GeoIP, Threat Intel)
- Data Analysis (IoC extraction, decoding, clustering
- Visualization (Event timeline, Process tree, Mapping, widgets)
Documentation is on ReadTheDocs
Fixes
- Broken links and outdated docs updated
- Fixes to some unit tests
Assets
2
Watchers:37 |
Star:434 |
Fork:74 |
创建时间: 2019-02-22 01:30:03 |
最后Commits: 12天前 |
许可协议:MIT |
efe2618
Compare
Verified
Hotfix release
Fixes
7b276a3@Adding azure-mgmt-core>=1.2.1 to requirements.txt
These packages will be mocked in their build - previously they caused dependency conflict errors in pip install