All pages
Powered by GitBook
1 of 1

Loading...

DiviK package

Python implementation of Divisive iK-means (DiviK) algorithm.

Tools within this package

  • Clustering at your command line with fit-clusters

  • Set of algorithm implementations for unsupervised analyses

    • Clustering

      • - hands-free clustering method with built-in feature selection

      • for selecting the number of clusters

      • for selecting the number of clusters

      • Modular with custom distance metrics and initializations

      • data-driven feature selection

      • - generates samples of fixed number of rows from given dataset, preserving groups proportion

      • - generates samples of random observations within boundaries of an original dataset, and preserving the rotation of the data

The recommended way to use this software is through . This is the most convenient way, if you want to use divik application.

To install latest stable version use:

Prerequisites for installation of base package:

  • Python 3.6 / 3.7 / 3.8

  • compiler capable of compiling the native C code and OpenMP support

You should have it already installed with GCC compiler, but if somehow not, try the following:

OpenMP is available as part of LLVM. You may need to install it with conda:

Having prerequisites installed, one can install latest base version of the package:

If you want to have compatibility with , you can install necessary extras with:

Note: Remember about \ before [ and ] in zsh shell.

You can install all extras with:

If you are using DiviK to run the analysis that could fail to fit RAM of your computer, consider disabling the default parallelism and switch to . It's easy to achieve through configuration:

  • set all parameters named n_jobs to 1;

  • set all parameters named allow_dask to True.

Note: Never set n_jobs>1 and allow_dask=True at the same time, the computations will freeze due to how multiprocessing and dask handle parallelism.

It can happen if the he gamred_native package (part of divik package) was compiled with different numpy ABI than scikit-learn. This could happen if you used different set of compilers than the developers of the scikit-learn package.

In such a case, a handler is defined to display the stack trace. If the trace comes from _matlab_legacy.py, the most probably this is the issue.

To resolve the issue, consider following the installation instructions once again. The exact versions get updated to avoid the issue.

Contribution guide will be developed soon.

Format the code with:

This software is part of contribution made by , rest of which is published .

  • meta-clustering

  • - allows you to select highly variant features above noise level, based on GMM-decomposition
    • - allows you to select highly variant features above noise level, based on outlier detection

  • - allows you to select highly variant features above noise level with your predefined thresholds for each

  • - generates samples of random observations within boundaries of an original dataset
    docker pull gmrukwa/divik
    sudo apt-get install libgomp1
    conda install -c conda-forge "compilers>=1.0.4,!=1.1.0" llvm-openmp
    pip install divik
    pip install divik[gin]
    pip install divik[all]
    isort -m 3 --fgw 3 --tc .
    black -t py36 .

    Installation

    Docker

    Python package

    Installation of OpenMP for Ubuntu / Debian

    Installation of OpenMP for Mac

    DiviK Installation

    High-Volume Data Considerations

    Known Issues

    Segmentation Fault

    Contributing

    References

    DiviK
    K-Means with Dunn method
    K-Means with GAP index
    K-Means implementation
    Feature extraction
    PCA with knee-based components selection
    Locally Adjusted RBF Spectral Embedding
    Feature selection
    EXIMS
    Gaussian Mixture Model based
    Sampling
    Stratified Sampler
    Uniform PCA Sampler
    Docker
    gin-config
    dask
    Data Mining Group of Silesian University of Technology
    here
    Mrukwa, G. and Polanska, J., 2020. DiviK: Divisive intelligent K-means for hands-free unsupervised clustering in biological big data. arXiv preprint arXiv:2009.10706.
    Two-step
    High Abundance And Variance Selector
    Outlier based selector
    Outlier Abundance And Variance Selector
    Percentage based selector
    Uniform Sampler