Package: BigDataStatMeth 2.0.6

BigDataStatMeth: Scalable Statistical Computing with HDF5-Backed Matrices

A framework for 'scalable' statistical computing on large on-disk matrices stored in 'HDF5' files. It provides efficient block-wise implementations of core linear-algebra operations (matrix multiplication, SVD, PCA, and QR decomposition) written in C++ and R, along with building blocks from which higher-level multivariate methods such as canonical correlation analysis can be constructed. These building blocks are designed not only for direct use, but also as foundational components for developing new statistical methods that must operate on datasets too large to fit in memory. The package supports data provided either as 'HDF5' files or standard R objects, and is intended for high-dimensional applications such as 'omics' and precision-medicine research.

Authors:Dolors Pelegri-Siso [aut, cre], Juan R. Gonzalez [aut]

# Install 'BigDataStatMeth' in R:

install.packages('BigDataStatMeth', repos = c('https://isglobal-exposomehub.r-universe.dev', 'https://cloud.r-project.org'))

Uses libs:

openblas– Optimized BLAS
curl– Easy-to-use client-side URL transfer library
openssl– Secure Sockets Layer toolkit
c++– GNU Standard C++ Library v3
openmp– GCC OpenMP (GOMP) support library

Datasets:

cancer - Cancer classification
colesterol - Dataset colesterol
miRNA - MiRNA

On CRAN:

This package does not link to any Github/Gitlab/R-forge repository. No issue tracker or development information is available.

openblas curl openssl cpp openmp

4.87 score 2 packages 23 scripts 1.3k downloads 81 exports 10 dependencies

Last updated from:09e2a42420. Checks:14 OK, 1 FAIL. Indexed: yes.

Target	Result	Time
linux-devel-arm64	OK	1091
linux-devel-x86_64	OK	1388
source / vignettes	OK	2149
linux-release-arm64	OK	1109
linux-release-x86_64	OK	1354
macos-release-arm64	OK	804
macos-release-x86_64	OK	941
macos-oldrel-arm64	OK	658
macos-oldrel-x86_64	OK	995
windows-devel-arm64	OK	1007
windows-devel-x86_64	OK	1269
windows-release-arm64	OK	1002
windows-release-x86_64	OK	1386
windows-oldrel-x86_64	OK	1348
wasm-release	FAIL	506

Exports:%*%apply_function bd_wproduct bdapply_Function_hdf5 bdblockMult bdblockSubstract bdblockSum bdCorr_matrix bdCreate_hdf5_group bdCreate_hdf5_matrix bdCrossprod bdgetDatasetsList_hdf5 bdImportData_hdf5 bdImportTextFile_hdf5 bdmove_hdf5_dataset bdpseudoinv bdpseudoinv_hdf5 bdReduce_hdf5_dataset bdScalarwproduct bdtCrossprod bdWrite_hdf5_dimnames can_allocate colMaxs colMeans colMins colnames.HDF5Matrix colnames<-.HDF5Matrix colSds colSums colVars cor crossprod diag diag_op diag_scale diag<-eigen filter_low_coverage filter_maf get_available_ram get_cpu_cores get_memory_thresholds get_recommended_threads get_total_ram hdf5_apply hdf5_close_all hdf5_close_file hdf5_create_matrix hdf5_import hdf5_import_multiple hdf5_matrix hdf5_reduce hdf5_remove hdf5matrix_options impute_snps is_open list_datasets memory_info multiply_sparse object_size pseudoinverse qr reduce rowMaxs rowMeans rowMins rownames.HDF5Matrix rownames<-.HDF5Matrix rowSds rowSums rowVars scale sd show_hdf5matrix_options split_dataset svd svd_auto_threshold sweep system_info tcrossprod var

Dependencies:biocmake bitops data.table dir.expiry filelock R6 Rcpp RcppEigen RCurl Rhdf5lib

Working with HDF5-Backed Matrices in BigDataStatMeth

Rendered fromBigDataStatMeth.Rmdusingknitr::rmarkdown

Last update: 2026-09-14
Started: 2025-11-29

Help page	Topics
Subset an HDF5Matrix	[.HDF5Matrix
Subsetting assignment for HDF5Matrix objects	[<-.HDF5Matrix
Matrix multiplication for HDF5Matrix	%*%
Apply a statistical or algebraic function to HDF5 datasets (generic)	apply_function
Convert HDF5Matrix to data.frame	as.data.frame.HDF5Matrix
Convert HDF5Matrix to in-memory matrix	as.matrix.HDF5Matrix
Weighted matrix–vector products and cross-products	bd_wproduct
Apply function to different datasets inside a group	bdapply_Function_hdf5
Block-Based Matrix Multiplication	bdblockMult
Block-Based Matrix Subtraction	bdblockSubstract
Block-Based Matrix Addition	bdblockSum
Compute correlation matrix for in-memory matrices (unified function)	bdCorr_matrix
Create Group in an HDF5 File	bdCreate_hdf5_group
Create HDF5 data file and write data to it	bdCreate_hdf5_matrix
Efficient Matrix Cross-Product Computation	bdCrossprod
List Datasets in HDF5 Group	bdgetDatasetsList_hdf5
Import data from URL or file to HDF5 format	bdImportData_hdf5
Import Text File to HDF5	bdImportTextFile_hdf5
Move HDF5 Dataset	bdmove_hdf5_dataset
Compute Matrix Pseudoinverse (In-Memory)	bdpseudoinv
Compute Matrix Pseudoinverse (HDF5-Stored)	bdpseudoinv_hdf5
Reduce Multiple HDF5 Datasets	bdReduce_hdf5_dataset
Matrix–scalar weighted product	bdScalarwproduct
Efficient Matrix Transposed Cross-Product Computation	bdtCrossprod
Write dimnames to an HDF5 dataset	bdWrite_hdf5_dimnames
BigDataStatMeth: Scalable statistical computing with R, C++, and HDF5	BigDataStatMeth
Check if memory allocation is safe	can_allocate
Cancer classification	cancer
Column-bind HDF5Matrix objects	cbind.HDF5Matrix
Cholesky decomposition of a symmetric positive-definite HDF5Matrix	chol.HDF5Matrix
Close HDF5Matrix	close.HDF5Matrix
Dataset colesterol	colesterol
Column and row maximums for HDF5Matrix	colMaxs colMaxs.HDF5Matrix rowMaxs rowMaxs.HDF5Matrix
Column and row means for HDF5Matrix	colMeans colMeans.HDF5Matrix rowMeans rowMeans.HDF5Matrix
Column and row minimums for HDF5Matrix	colMins colMins.HDF5Matrix rowMins rowMins.HDF5Matrix
Column and row standard deviations for HDF5Matrix	colSds colSds.HDF5Matrix rowSds rowSds.HDF5Matrix
Column and row sums for HDF5Matrix	colSums colSums.HDF5Matrix rowSums rowSums.HDF5Matrix
Column and row variances for HDF5Matrix	colVars colVars.HDF5Matrix rowVars rowVars.HDF5Matrix
Correlation (generic)	cor
Correlation matrix for HDF5Matrix objects	cor.HDF5Matrix
Cross product of HDF5Matrix objects	crossprod crossprod.HDF5Matrix
Extract or construct a diagonal for HDF5Matrix	diag diag.default diag.HDF5Matrix
Diagonal-vector operation on an HDF5Matrix	diag_op diag_op.HDF5Matrix
Scalar diagonal operation on an HDF5Matrix	diag_scale diag_scale.HDF5Matrix
Set diagonal of an HDF5Matrix (generic)	diag<-
Dimensions of an HDF5Matrix	dim.HDF5Matrix
Get dimension names of an HDF5Matrix	colnames.HDF5Matrix colnames<-.HDF5Matrix dimnames.HDF5Matrix rownames.HDF5Matrix rownames<-.HDF5Matrix
Set dimension names on an HDF5Matrix	dimnames<-.HDF5Matrix
Spectral decomposition	eigen eigen.default eigen.HDF5Matrix
Remove high-missingness features from an HDF5Matrix	filter_low_coverage filter_low_coverage.HDF5Matrix
Remove SNPs by Minor Allele Frequency from an HDF5Matrix	filter_maf filter_maf.HDF5Matrix
Get available (free) system RAM	get_available_ram
Get number of CPU cores	get_cpu_cores
Get dynamic memory thresholds based on system RAM	get_memory_thresholds
Get recommended number of threads for parallel operations	get_recommended_threads
Get total system RAM	get_total_ram
Apply a mathematical operation to multiple HDF5 datasets	hdf5_apply
Close all HDF5Matrix objects	hdf5_close_all
Close all HDF5 handles for a specific file	hdf5_close_file
Create an HDF5 dataset and return an HDF5Matrix object	hdf5_create_matrix
Import data from file or URL into HDF5 format	hdf5_import
Import multiple files into HDF5	hdf5_import_multiple
Open an HDF5 dataset as an HDF5Matrix object	hdf5_matrix
Reduce all datasets in an HDF5 group by a binary operation	hdf5_reduce
Remove (delete) a dataset from an HDF5 file	hdf5_remove
Set or get HDF5Matrix computation options	hdf5matrix_options
S3 methods for HDF5Matrix	HDF5Matrix-S3
Summary statistics for HDF5Matrix	HDF5Matrix-scalar-aggregations mean.HDF5Matrix Summary.HDF5Matrix
Impute missing SNP values in an HDF5Matrix	impute_snps impute_snps.HDF5Matrix
Check if HDF5Matrix is open	is_open
Length of an HDF5Matrix	length.HDF5Matrix
List datasets in an HDF5 file or group	list_datasets
Print system memory information	memory_info
miRNA	miRNA
Sparse-aware matrix multiplication (generic)	multiply_sparse
Sparse-aware matrix multiplication for HDF5Matrix	multiply_sparse.HDF5Matrix
Get memory size of HDF5Matrix without loading	object_size
Elementwise arithmetic operators for HDF5Matrix objects	Ops.HDF5Matrix
Principal Component Analysis of an HDF5Matrix	prcomp.HDF5Matrix
Print an HDF5Matrix object	print.HDF5Matrix
Print method for HDF5PCA objects	print.HDF5PCA
Moore-Penrose pseudoinverse	pseudoinverse pseudoinverse.default pseudoinverse.HDF5Matrix pseudoinverse.matrix
QR decomposition of an HDF5Matrix	qr
QR decomposition of an HDF5Matrix	qr.HDF5Matrix
Row-bind HDF5Matrix objects	rbind.HDF5Matrix
Reduce a group of HDF5 datasets by accumulation (generic)	reduce
Scale / normalize an HDF5Matrix	scale scale.HDF5Matrix
Standard deviation of all elements of an HDF5Matrix	sd sd.HDF5Matrix
Show current HDF5Matrix performance settings	show_hdf5matrix_options
Matrix inverse of a symmetric positive-definite HDF5Matrix via Cholesky	solve.HDF5Matrix
Split an HDF5Matrix into multiple block datasets	split_dataset split_dataset.HDF5Matrix
Split an HDF5Matrix into a list of blocks	split.HDF5Matrix
Structure of an HDF5Matrix object	str.HDF5Matrix
Singular Value Decomposition (generic)	svd
Element-count boundary of the automatic SVD/PCA path selection	svd_auto_threshold
Singular Value Decomposition of an HDF5Matrix	svd.HDF5Matrix
Sweep out array summaries (generic)	sweep
Broadcast a vector over an HDF5Matrix (sweep)	sweep.HDF5Matrix
Get system information summary	system_info
Transposed cross product of HDF5Matrix objects	tcrossprod tcrossprod.HDF5Matrix
Variance of all elements of an HDF5Matrix	var var.HDF5Matrix

Package: BigDataStatMeth 2.0.6

BigDataStatMeth: Scalable Statistical Computing with HDF5-Backed Matrices

Citation

Readme and manuals

Help Manual

Usage by other packages (reverse dependencies)