bdCreate_hdf5_matrix() error paths.dim() on a closed HDF5Matrix returns NULL and length() returns 0
instead of signalling an error.hdf5_close_all() is silent when there is nothing to close.impute_snps() could crash,
deadlock or silently corrupt its output on matrices with over 4000 columns.hdf5_apply() with "CrossProd" / "tCrossProd" returned an all-zero
result for inputs of 1,048,576 elements or more.svd() / prcomp() with scale = TRUE now stop on a constant column
instead of silently returning all-zero singular values. scale() is
unchanged (NaN, as in base R).threads argument of svd() / prcomp() is now honoured; it had no
effect on the internal matrix products.impute_snps() and the omics filters now propagate dimnames, and the
filters report a clear error when they remove every row or column.crossprod(), %*% and tcrossprod() between HDF5Matrix
objects stored in different files no longer fail.memory.size() (defunct since
R 4.2.0), which inflated the block-size and preload budget.checkHDF5File() uses the portable HDF5 C API instead of the deprecated
H5::H5File::isHdf5(), fixing a downstream link failure with
Rhdf5lib >= 1.32.filter_low_coverage(): pcent counts entries equal to 3, not NA.filter_maf(): the documented threshold direction was inverted (variants
with MAF at or below the threshold are removed).svd() / prcomp() now record whether the decomposition was exact or
approximate (new attributes plus a message()), and the new
svd_auto_threshold() returns the size boundary. See ?svd.HDF5Matrix.svd() / prcomp() warn when the requested threads cannot be honoured.hdf5_remove() and HDF5Matrix$remove() delete a dataset from a file.bdCorr_matrix(): fixed a C++ exception (Expecting a single value: [extent=0]) that could abort the R session under recent Rcpp versions.
The returned list evaluated an optional Nullable<bool> argument in a
boolean context, which now resolves to as<bool>(NULL); the unwrapped
logical values are used instead. This resolves the CRAN r-devel example
check failure."c++ exception" prefix from all error messages so
failures surface as ordinary R errors rather than text that reads like a
leaked C++ exception (per CRAN request). Error-handling control flow is
unchanged.cbind() / rbind(): documentation no longer states that plain R matrices
are accepted and auto-converted (they are not). Convert inputs first with
hdf5_matrix() / hdf5_create_matrix().RSpectra from the README system-requirements list (it is not a
package dependency; the bundled C++ Spectra library is used internally).Ops.HDF5Matrix definition into a single
documented method and documented the 1xn / nx1 broadcast behavior.qr(..., method = "tsqr"): fixed incorrect results (loss of orthogonality
in Q) when the last row-block had fewer rows than columns (m mod block_size < n). The last block is now merged with the previous one, guaranteeing all
blocks satisfy the rank condition required by the local Householder QR step.
Affected configurations: any (m, n, threads) where m mod max(2n, ceil(m / max(4, threads * 4))) < n. Discovered via a systematic regression test
covering 202 (m, n, threads, block_size) combinations.
pseudoinverse() now correctly zeroes singular values at or below the
tolerance (1e-9), fixing incorrect results for rank-deficient or
near-singular matrices.
Fixed silent failure on matrix conversion errors in bdCrossprod(),
bdtCrossprod(), and bd_wproduct(); these now raise informative errors.
Removed unreachable code in the correlation HDF5 backend (no functional effect).
pseudoinverse.matrix() for consistent in-memory support alongside
the existing HDF5Matrix method.pseudoinverse() and prcomp(rank. =) documentation to match
actual behavior.crossprod(), tcrossprod(),
aggregation functions, scale(), and solve(): when input fits
within 20% of available RAM, a single HDF5 read + BLAS call + single
write replaces the block loop.scale() PATH 2 (block-wise streaming) is now parallelized with OpenMP.hdf5matrix_options(paral = TRUE) now forces PATH 2 (OMP streaming)
in %*%, crossprod(), tcrossprod(), aggregations, and scale(),
giving explicit thread control regardless of matrix size.getAvailableMemoryMB() now uses platform-native detection (macOS:
host_statistics64; Linux: /proc/meminfo), replacing the hardcoded
4 GB fallback.target_blocks = max(4, nthreads * 4)).rbind() and cbind() with 3 or more HDF5Matrix arguments no
longer produce dataset names exceeding 1400 characters. Auto-generated
names are now rbind_N_<uid> with a random 8-character identifier,
also preventing "dataset already exists" collisions. Two-argument
calls are unchanged (A_rbind_B).paral = FALSE being silently ignored in multiplication()
PATH 2/3.#pragma omp critical(accessFile) in crossprod(),
tcrossprod(), and aggregation PATH 2 (race condition fix).Cholesky_decomposition_intermediate_hdf5 on
macOS ARM64 (removed throw from OpenMP parallel region).scale(): added paral, threads parameters; PATH 1/PATH 2
@details; @seealso hdf5matrix_options().split(), hdf5_apply(), hdf5_reduce(): clarified calling
conventions and two-level access pattern. Fixed legacy example.hdf5_import example.This is a major release of BigDataStatMeth. It introduces a new
HDF5Matrix user-facing interface, standard S3 methods for
HDF5-backed matrices, and an extended C++ infrastructure for block-wise
statistical computing with HDF5 files.
HDF5Matrix interface for working with matrices stored in
HDF5 files.dim(), [, [<-, %*%, crossprod(),
tcrossprod(), scale(), cor(), svd(), prcomp(), qr(),
chol(), and solve().hdf5matrix_options() for common
settings such as parallel execution, number of threads, block size,
and HDF5 compression.HDF5Matrix objects.%*%,
crossprod(), tcrossprod(), cbind(), and rbind().colSums(),
rowSums(), colMeans(), rowMeans(), colVars(), rowVars(),
colSds(), rowSds(), colMins(), rowMins(), colMaxs(),
rowMaxs(), mean(), var(), and sd().scale(),
sweep(), and cor().svd(), prcomp(), qr(), chol(),
solve(), eigen(), and pseudoinverse().diag(),
diag<-(), diag_op(), and diag_scale().bd* utilities for specialized
workflows that do not map directly to standard R generics.list_datasets() for inspecting datasets stored in HDF5 files.hdf5_close_all() for closing open HDF5 handles managed by the
package.hdf5matrix_options().HDF5Matrix interface
and standard R methods.help("BigDataStatMeth").extdata directory.minimumBlockSize promoted from int to long in
Cholesky_decomposition_intermediate_hdf5 and
Inverse_of_Cholesky_decomposition_intermediate_hdf5 to prevent
overflow before promotion to double for sqrt(). Detected by
gcc-ASAN on R-hub.matrixSvdBlock.hpp): wrong singular value was evaluated in the
rank-truncation check, causing incorrect nzeros counts.H5SL_insert
from H5I_register) when the number of hierarchical levels q >= 2:
Next_level_SvdBlock_decomposition_hdf5 is now sequential; the outer
block loop already provides sufficient parallelism.nev parameter not being applied to per-block truncation in
First_level_SvdBlock_decomposition_hdf5: the parameter was declared
in the function signature but never used, causing unnecessarily large
intermediate matrices and a final SVD over more components than
requested.bdCCA_hdf5_rcpp().