data_utils
anomaly_match.datasets.data_utils
¶
Data loading and splitting utilities for dataset preparation.
get_sampler_by_name(name)
¶
Get a PyTorch sampler class by its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Name of the sampler class (e.g., "RandomSampler") |
required |
Returns:
| Type | Description |
|---|---|
type
|
The sampler class corresponding to the name |
Raises:
| Type | Description |
|---|---|
AttributeError
|
If the sampler name is not found in torch.utils.data.sampler |
Source code in anomaly_match/datasets/data_utils.py
get_data_loader(dset, batch_size=None, shuffle=False, num_workers=4, pin_memory=True, data_sampler=None, replacement=True, num_epochs=None, num_iters=None, generator=None, drop_last=True, use_weighted_sampler=False)
¶
Create a PyTorch DataLoader with various sampling strategies.
This function provides a flexible way to create DataLoader objects with different sampling strategies, including weighted sampling for imbalanced datasets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dset
|
Dataset
|
Dataset object |
required |
batch_size
|
int | None
|
Size of each batch |
None
|
shuffle
|
bool
|
Whether to shuffle the data (only used if data_sampler is None) |
False
|
num_workers
|
int
|
Number of subprocesses to use for data loading |
4
|
pin_memory
|
bool
|
If True, copy tensors into CUDA pinned memory |
True
|
data_sampler
|
str | type | None
|
Sampler name (string) or sampler class |
None
|
replacement
|
bool
|
Whether to sample with replacement |
True
|
num_epochs
|
int | None
|
Number of epochs (used to calculate total samples) |
None
|
num_iters
|
int | None
|
Number of iterations (alternative way to calculate total samples) |
None
|
generator
|
Generator | None
|
Random number generator for reproducibility |
None
|
drop_last
|
bool
|
Whether to drop the last incomplete batch |
True
|
use_weighted_sampler
|
bool
|
If True, use WeightedRandomSampler for class imbalance |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
DataLoader |
DataLoader
|
PyTorch DataLoader configured with the specified parameters |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If an unsupported sampler is specified |
Source code in anomaly_match/datasets/data_utils.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | |