Repository navigation
Expand file tree
/
Copy pathFirstOrderOrientation.py
More file actions
1815 lines (1577 loc) · 79.2 KB
/
Copy pathFirstOrderOrientation.py
File metadata and controls
1815 lines (1577 loc) · 79.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""
Structural Subsampling Engine — Orientation Data Spatial Reduction
Overview
--------
Reduces spatial clustering in bedding / foliation point datasets while
preserving the statistical character of the orientations. Designed for
GSWA-style structural datasets where the raw shapefile may contain mixed
feature types (foliation, joints, etc.) and non-standard column names.
The main entry point is run_subsampling_file(). It runs a two-stage pipeline:
Stage 0 — Column standardisation and bedding filter
1. Filters the raw shapefile to bedding-only records (optional).
2. Renames source columns to the four standard names:
DIP – dip angle in degrees (0 = horizontal, 90 = vertical)
DIP_DIR – dip direction in degrees (0–360, clockwise from north)
EASTING – projected easting coordinate (metres)
NORTHING – projected northing coordinate (metres)
3. Converts strike to dip-direction if the source records strike.
4. Writes the processed shapefile to input_beddings_only/ so all
subsequent methods read from one clean, consistently-named file.
Stage 1 — Subsampling
Applies the chosen algorithm(s) to the standardised bedding file.
Six algorithms are available; they can be run individually or in
combination by passing a list of method names to run_subsampling_file().
Seven subsampling algorithms
-----------------------------
1. decimation – retain every nth point by digitisation order
2. stochastic – random-fraction sampling with a fixed seed
3. gridcell_average – grid-cell mean-vector averaging
4. spherical_kent – grid-cell Kent statistics (κ, β); optional kappa_min filter
5. outlier_removal – max-Δκ single-outlier removal per cell (min_for_outlier=3)
5b. outlier_carmichael – iterative Δκ-break algorithm (Carmichael 2016)
6. firstorder – proximity + angular alignment to contacts
Key parameters
--------------
bedding_file : standardised bedding point shapefile path
geology_file : geology polygon shapefile (used by Method 6 for contacts)
output_dir : directory where subsampled shapefiles and CSVs are written
methods : list of algorithm names to run (or 'all' to run every method)
grid_n : grid cell size in metres for Methods 3–5b (default 1000 m)
decimation_n : step size for Method 1 (default 5)
stoch_frac : sampling fraction for Method 2 (default 0.5)
dist_buffer : contact proximity buffer for Method 6 (default 500 m)
angle_tol : strike alignment tolerance for Method 6 (default 15°)
References
----------
Carmichael, T. & Ailleres, L. (2016). Method and analysis for the upscaling
of structural data. Journal of Structural Geology, 83, 121–133.
Leong, L.S. & Carlile, J.C. (1998). A method for estimating the Kent
distribution parameters for orientation data. Mathematical Geology.
"""
import os
import math
from math import sin, cos, asin, atan, atan2, degrees, radians, sqrt, acos, pi
import geopandas as gpd
import pandas as pd
import numpy as np
import shapely
from pandas.errors import EmptyDataError
from shapely.geometry import Point, LineString
from shapely.ops import unary_union
# =============================================================================
# SUBSAMPLING ENGINE
# Encapsulates all seven subsampling algorithms together with their shared
# statistical machinery. Instantiate one engine per run; the engine is
# stateless between method calls so the same instance can be reused for
# multiple datasets or parameter sweeps.
# =============================================================================
class SubsamplingEngine:
"""
Main engine for structural orientation data subsampling.
Wraps all seven subsampling algorithms and their shared statistical helpers
(direction-cosine conversion, mean-orientation, Kent distribution) in a
single class. Instantiate once and call subsample() with the chosen
method name, or call individual method functions directly.
Parameters
----------
dip : Standard column name for dip angle. Default ``'DIP'``.
dipdir : Standard column name for dip direction. Default ``'DIP_DIR'``.
easting : Standard column name for easting. Default ``'EASTING'``.
northing : Standard column name for northing. Default ``'NORTHING'``.
"""
def __init__(self,
dip = 'DIP',
dipdir = 'DIP_DIR',
easting = 'EASTING',
northing = 'NORTHING'):
self.dip = dip
self.dipdir = dipdir
self.easting = easting
self.northing = northing
# =========================================================================
# UTILITY
# =========================================================================
@staticmethod
def _bounds(gdf):
"""Return integer bounding-box tuple (min_x, max_x, min_y, max_y)."""
b = gdf.total_bounds # [min_x, min_y, max_x, max_y]
return int(b[0]), int(b[2]) + 1, int(b[1]), int(b[3]) + 1
# =========================================================================
# STATISTICAL HELPERS
# Private methods used internally by the grid-cell and outlier algorithms.
# =========================================================================
def _add_direction_cosines(self, gpdataframe):
"""
Append a ``vector`` column of direction-cosine tuples to a DataFrame.
Each row receives a ``(l, m, n)`` tuple derived from its dip and
dip-direction values. The computation is fully vectorised using NumPy.
Parameters
----------
gpdataframe : Input GeoDataFrame or DataFrame.
Returns
-------
DataFrame with an additional column ``vector`` containing
``(l, m, n)`` tuples.
"""
df = pd.DataFrame(gpdataframe)
dip_r = np.radians(df[self.dip].astype(float))
dipdir_r = np.radians(df[self.dipdir].astype(float))
# Vectorised: pole pointing downward into the plane
l_arr = np.sin(dip_r) * np.sin(dipdir_r) # east
m_arr = np.sin(dip_r) * np.cos(dipdir_r) # north
n_arr = np.cos(dip_r) # vertical (down)
df['vector'] = list(zip(l_arr, m_arr, n_arr))
return df
def _calc_mean_orientation(self, df):
"""
Compute the mean orientation of a group of structural measurements.
The mean is the normalised sum of the individual unit vectors
(direction cosines), converted back to dip / dip-direction.
Parameters
----------
df : DataFrame that **must** already contain a ``vector`` column of
``(l, m, n)`` tuples (added by :meth:`_add_direction_cosines`).
Returns
-------
Tuple ``(dip, dipdir)`` in degrees, or ``(nan, nan)`` if the resultant
vector length is near zero (antipodal vectors cancel).
"""
df = pd.DataFrame(df)
df[['l', 'm', 'n']] = pd.DataFrame(df['vector'].tolist(), index=df.index)
l_sum, m_sum, n_sum = df['l'].sum(), df['m'].sum(), df['n'].sum()
count = len(df.index)
r_bar = sqrt(float((l_sum / count)**2 + (m_sum / count)**2 +
(n_sum / count)**2))
if r_bar < 1e-12:
return (float('nan'), float('nan'))
l_unit = (l_sum / count) / r_bar
m_unit = (m_sum / count) / r_bar
n_unit = (n_sum / count) / r_bar
return dircos2ddd(l_unit, m_unit, n_unit)
def _calc_kent(self, gpdataframe):
"""
Estimate Kent distribution parameters for a cell of measurements.
The Kent (5-parameter Fisher-Bingham) distribution on the sphere is
characterised by:
- **kappa** : concentration parameter (higher = tighter cluster)
- **beta** : ovalness / anisotropy parameter (0 = rotationally symmetric)
Moment estimates of Kent (1982), as given by Leong & Carlile (1998)::
kappa = 1 / (2 - 2*r1 - r2) + 1 / (2 - 2*r1 + r2)
beta = 0.5 * [1 / (2 - 2*r1 - r2) - 1 / (2 - 2*r1 + r2)]
where ``r1`` is the **mean** resultant length (resultant length / N)
and ``r2`` the difference between the two largest eigenvalues of the
scatter matrix in the plane normal to the mean direction, found by
rotating the scatter matrix with H (mean direction to the z axis) and
then K (the sword angle psi).
Parameters
----------
gpdataframe : DataFrame with a ``vector`` column of ``(l, m, n)`` tuples.
Returns
-------
Tuple ``(count, kappa, beta)``. kappa is ``inf`` and beta ``nan`` when
the estimate has no finite value: a single measurement, identical
measurements, or a cell whose scatter makes a denominator <= 0.
Notes
-----
Corrections (v1.1.0) to the v1.0.0 implementation, which produced
invalid kappa values (0, -1 or negative):
* ``r1`` is the mean resultant length, not the resultant length itself
(the resultant length grows with N, so 2 - 2*r1 was negative).
* H has the rotated axes as **columns** (H^T S H puts the mean
direction on z); v1.0.0 used the transpose.
* The mean-direction azimuth uses ``atan2`` so the quadrant is kept.
* The sword angle uses the diagonal difference B[0,0] - B[1,1]
(unchanged from v1.0.0).
"""
df = pd.DataFrame(gpdataframe)
x = np.array(df['vector'].tolist(), dtype=float)
count = len(x)
if count < 2:
return (count, float('inf'), float('nan'))
xbar = x.mean(axis=0)
r1 = float(np.sqrt((xbar ** 2).sum())) # mean resultant length
if r1 >= 1 - 1e-12: # identical measurements
return (count, float('inf'), float('nan'))
if r1 < 1e-12: # antipodal vectors cancel
return (count, float('nan'), float('nan'))
g = xbar / r1
theta = np.arccos(np.clip(g[2], -1.0, 1.0))
phi = np.arctan2(g[1], g[0])
H = np.array([
[np.cos(theta) * np.cos(phi), -np.sin(phi), np.sin(theta) * np.cos(phi)],
[np.cos(theta) * np.sin(phi), np.cos(phi), np.sin(theta) * np.sin(phi)],
[-np.sin(theta), 0.0, np.cos(theta)],
])
S = x.T @ x / count # scatter (orientation) matrix / N
B = H.T @ S @ H
psi = 0.5 * np.arctan2(2 * B[0, 1], B[0, 0] - B[1, 1]) # sword angle
K = np.array([
[np.cos(psi), -np.sin(psi), 0.0],
[np.sin(psi), np.cos(psi), 0.0],
[0.0, 0.0, 1.0],
])
G = H @ K
V = G.T @ S @ G
r2 = V[0, 0] - V[1, 1]
d1 = 2 - 2 * r1 - r2
d2 = 2 - 2 * r1 + r2
if d1 <= 0 or d2 <= 0:
return (count, float('inf'), float('nan'))
kappa = 1 / d1 + 1 / d2
beta = 0.5 * (1 / d1 - 1 / d2)
return (count, float(kappa), float(beta))
@staticmethod
def _fmt_dip(value):
"""Dip for a CSV field, rounded to 0.1 degree."""
return f"{round(float(value), 1):.1f}"
@staticmethod
def _fmt_dipdir(value):
"""Dip direction for a CSV field, rounded to 0.1 degree and kept in 0-360 (360.0 -> 0.0)."""
return f"{round(float(value), 1) % 360.0:.1f}"
@staticmethod
def _fmt_kent(value):
"""Kent parameter for a CSV field: 2 decimals, or -998 when undefined."""
return f"{value:.2f}" if np.isfinite(value) else "-998"
# =========================================================================
# METHOD 1: DECIMATION
# Retain every nth point by digitisation order.
# =========================================================================
def decimation(self, gpdataframe, n, path_out='outputs/'):
"""
Retain every nth point by digitisation order.
The simplest possible subsample: slice the dataset with a fixed step
so that evenly-spaced records are kept.
Parameters
----------
gpdataframe : Input GeoDataFrame of structural measurements.
n : Step size — keep every nth record (e.g. ``n=5`` keeps
records 0, 5, 10, …).
path_out : Output directory path.
Returns
-------
GeoDataFrame of retained measurements with a point geometry column.
"""
df = pd.DataFrame(gpdataframe)
df_sub = df.iloc[::n].copy()
df_sub['geometry'] = df_sub.apply(
lambda row: Point(float(row[self.easting]),
float(row[self.northing])), axis=1)
df_sub = gpd.GeoDataFrame(df_sub, geometry='geometry')
if df_sub.empty:
print("DataFrame Empty")
else:
df_sub.to_file(
os.path.join(path_out, f"structure_file_decimation_{n}.shp"),
driver='ESRI Shapefile')
df_sub.to_csv(
os.path.join(path_out, f"structure_file_decimation_{n}.csv"))
return df_sub
# =========================================================================
# METHOD 2: STOCHASTIC SUBSAMPLING
# Randomly sample a fraction of the input dataset.
# =========================================================================
def stochastic(self, gpdataframe, frac=0.5, replace=False,
random_state=42, path_out='outputs/'):
"""
Randomly subsample a fraction of the input dataset.
Applies ``DataFrame.sample`` with the specified fraction and random
seed, giving a reproducible random subset.
Parameters
----------
gpdataframe : Input GeoDataFrame of structural measurements.
frac : Fraction of rows to retain, in the range (0, 1].
Default: 0.5 (50 %).
replace : Whether to sample with replacement. Default: ``False``.
random_state : Integer seed for reproducibility. Default: 42.
path_out : Output directory path.
Returns
-------
GeoDataFrame of sampled measurements with a point geometry column.
"""
df = pd.DataFrame(gpdataframe)
df_sub = df.sample(frac=frac, replace=replace, random_state=random_state)
df_sub['geometry'] = df_sub.apply(
lambda row: Point(float(row[self.easting]),
float(row[self.northing])), axis=1)
df_sub = gpd.GeoDataFrame(df_sub, geometry='geometry')
if df_sub.empty:
print("DataFrame Empty")
else:
df_sub.to_file(
os.path.join(path_out, f"structure_file_stochastic_{frac}.shp"),
driver='ESRI Shapefile')
df_sub.to_csv(
os.path.join(path_out, f"structure_file_stochastic_{frac}.csv"))
return df_sub
# =========================================================================
# METHOD 3: GRID-CELL AVERAGING
# Compute the mean orientation vector for measurements within each cell.
# =========================================================================
# =========================================================================
# ARRAY HELPERS FOR THE GRID-CELL METHODS
# The grid-cell methods work on plain numpy arrays sorted by cell: no pandas work
# is done per cell (that was most of their run time).
# =========================================================================
def _grid_groups(self, gdf, minx, maxx, miny, maxy, n):
"""
Bin the measurements into grid cells. Returns a dict of arrays sorted by cell
(cells in the same order as ``groupby(['_cx', '_cy'])``, measurements within a cell
in their original order): ``X`` direction cosines (n, 3), ``e`` / ``n`` coordinates,
``labels`` original index labels, and ``starts`` / ``ends`` of each cell.
Measurements outside the bounds or with a null dip / dip direction are left out.
"""
e = np.asarray(gdf[self.easting], dtype=float)
nn = np.asarray(gdf[self.northing], dtype=float)
dip = np.asarray(gdf[self.dip], dtype=float)
dd = np.asarray(gdf[self.dipdir], dtype=float)
ok = ((e >= minx) & (e < maxx) & (nn >= miny) & (nn < maxy)
& np.isfinite(dip) & np.isfinite(dd))
labels = np.asarray(gdf.index)[ok]
e, nn = e[ok], nn[ok]
dip_r, dd_r = np.radians(dip[ok]), np.radians(dd[ok])
X = np.stack([np.sin(dip_r) * np.sin(dd_r), # east
np.sin(dip_r) * np.cos(dd_r), # north
np.cos(dip_r)], axis=1) # down
cx = ((e - minx) / n).astype(np.int64)
cy = ((nn - miny) / n).astype(np.int64)
order = np.lexsort((cy, cx)) # stable: original order inside a cell
cx, cy = cx[order], cy[order]
if len(cx):
starts = np.flatnonzero(np.r_[True, (cx[1:] != cx[:-1]) | (cy[1:] != cy[:-1])])
ends = np.r_[starts[1:], len(cx)]
else:
starts = ends = np.zeros(0, dtype=np.int64)
return {'X': X[order], 'e': e[order], 'n': nn[order], 'labels': labels[order],
'starts': starts, 'ends': ends}
@staticmethod
def _orientation_from_sums(l_sum, m_sum, n_sum, count):
"""(dip, dipdir) of the vector mean given the component sums - as :meth:`_calc_mean_orientation`."""
r_bar = sqrt(float((l_sum / count)**2 + (m_sum / count)**2 + (n_sum / count)**2))
if r_bar < 1e-12:
return (float('nan'), float('nan'))
return dircos2ddd((l_sum / count) / r_bar, (m_sum / count) / r_bar, (n_sum / count) / r_bar)
@staticmethod
def _cell_sums(X, starts):
"""Per-cell sum of the direction cosines (K, 3) and of their outer products (K, 3, 3)."""
sums = np.add.reduceat(X, starts, axis=0) + 0.0
x, y, z = X[:, 0], X[:, 1], X[:, 2]
comp = [np.add.reduceat(v, starts) for v in (x * x, x * y, x * z, y * y, y * z, z * z)]
T = np.empty((len(starts), 3, 3))
T[:, 0, 0], T[:, 0, 1], T[:, 0, 2] = comp[0], comp[1], comp[2]
T[:, 1, 0], T[:, 1, 1], T[:, 1, 2] = comp[1], comp[3], comp[4]
T[:, 2, 0], T[:, 2, 1], T[:, 2, 2] = comp[2], comp[4], comp[5]
return sums, T
def _kappa_beta_of(self, X):
"""Kent (kappa, beta) of one set of direction cosines X (n, 3), as :meth:`_calc_kent`."""
n = len(X)
k, b = self._kent_kappa_batch(n, X.mean(axis=0)[None, :], (X.T @ X / n)[None, :, :],
return_beta=True)
return float(k[0]), float(b[0])
def gridcell_average(self, gpdataframe, minx, maxx, miny, maxy,
n=1000, path_out='outputs/'):
"""
Compute the mean orientation vector for measurements within each grid cell.
The study area is divided into a regular grid of n × n metre cells.
All measurements that fall within a cell are combined into a single
representative orientation using the normalised vector mean. Cells
with no measurements are omitted from the output.
Parameters
----------
gpdataframe : Input GeoDataFrame of structural measurements.
minx, maxx : Western and eastern study-area boundaries (metres).
miny, maxy : Southern and northern study-area boundaries (metres).
n : Grid cell size in metres. Default: 1000 m.
path_out : Output directory path.
Returns
-------
Filename stem (str) of the CSV written to ``path_out``. Columns:
EASTING, NORTHING, DIP, DIP_DIR and ``count`` (the number of
measurements averaged in the cell). Pass this to
:func:`save_grid_to_shapefile` to produce a point shapefile.
"""
gr = self._grid_groups(gpdataframe, minx, maxx, miny, maxy, n)
e, nn, starts, ends = gr['e'], gr['n'], gr['starts'], gr['ends']
file = f"structure_file_gridcell_{n}"
fieldnames = ['EASTING', 'NORTHING', 'DIP', 'DIP_DIR', 'count']
with open(os.path.join(path_out, file + ".csv"), "w",
encoding="utf-8") as out:
out.write(','.join(fieldnames) + '\n')
if len(starts):
counts = ends - starts
sums = np.add.reduceat(gr['X'], starts, axis=0) + 0.0
for k in range(len(starts)):
s, t_ = starts[k], ends[k]
# Output at the mean position of the input points, not the cell
# centre, so a single point in a cell stays where it was.
centx = float(e[s:t_].mean())
centy = float(nn[s:t_].mean())
dip_val, dipdir_val = self._orientation_from_sums(
sums[k, 0], sums[k, 1], sums[k, 2], int(counts[k]))
dip_str = ('NaN' if math.isnan(dip_val)
else self._fmt_dip(dip_val))
dipdir_str = ('NaN' if math.isnan(dipdir_val)
else self._fmt_dipdir(dipdir_val))
out.write(f"{centx:.8f},{centy:.8f},"
f"{dip_str},{dipdir_str},{int(counts[k])}\n")
return file
def count_cells(self, gpdataframe, minx, maxx, miny, maxy, n, min_points=1):
"""
Number of grid cells (same grid as the grid-cell methods) holding at least
``min_points`` measurements - a fast stand-in for running a grid method
just to count its output (used by the auto-fit to N points search).
``min_points=1`` gives the number of output points of the Kent and Kent
Outlier methods (without a kappa filter); ``min_points=4`` that of
:meth:`outlier_carmichael`, which skips cells of 3 or fewer.
"""
e = np.asarray(gpdataframe[self.easting], dtype=float)
nn = np.asarray(gpdataframe[self.northing], dtype=float)
keep = (e >= minx) & (e < maxx) & (nn >= miny) & (nn < maxy)
cx = ((e[keep] - minx) / n).astype(np.int64)
cy = ((nn[keep] - miny) / n).astype(np.int64)
_, sizes = np.unique(np.stack([cx, cy], axis=1), axis=0, return_counts=True)
return int((sizes >= min_points).sum())
# =========================================================================
# METHOD 4: SPHERICAL STATISTICS (KENT DISTRIBUTION)
# Compute Kent distribution parameters (mean, κ, β) per grid cell.
# =========================================================================
def spherical_kent(self, gpdataframe, minx, maxx, miny, maxy,
n=1000, kappa_min=None, path_out='outputs/'):
"""
Spherical statistics per grid cell: Kent mean orientation, kappa and beta,
with an optional kappa filter.
The study area is divided into n x n metre cells (the same grid as
:meth:`gridcell_average`). For each non-empty cell the representative
orientation is the Kent mean direction (the normalised vector mean, so
the orientations equal those of :meth:`gridcell_average`), written at
the mean position of the cell's measurements together with the number
of measurements and the Kent moment estimates of kappa (concentration)
and beta (ovalness), see :meth:`_calc_kent`.
Two variants:
* ``kappa_min=None`` (spherical statistics **without** kappa filter):
every non-empty cell is retained; kappa and beta are reported but do
not change the output orientations.
* ``kappa_min=k`` (spherical statistics **with** kappa filter): only
cells whose kappa is finite and >= k are retained, i.e. cells with a
statistically concentrated mean. Cells with a single measurement
(kappa undefined) and dispersed or girdle-shaped cells are dropped.
Undefined kappa or beta (one measurement, identical measurements) are
written as -998.
Parameters
----------
gpdataframe : Input GeoDataFrame of structural measurements.
minx, maxx : Western and eastern study-area boundaries (metres).
miny, maxy : Southern and northern study-area boundaries (metres).
n : Grid cell size in metres. Default: 1000 m.
kappa_min : None (no filter, default) or the minimum Kent kappa a
cell must reach to be retained.
path_out : Output directory path.
Returns
-------
Filename stem (str) of the CSV written to ``path_out``:
``structure_file_spherical_{n}`` without filter,
``structure_file_spherical_kappa{kappa_min}_{n}`` with filter.
"""
gr = self._grid_groups(gpdataframe, minx, maxx, miny, maxy, n)
e, nn, starts, ends = gr['e'], gr['n'], gr['starts'], gr['ends']
file = (f"structure_file_spherical_{n}" if kappa_min is None
else f"structure_file_spherical_kappa{kappa_min:g}_{n}")
fieldnames = ['EASTING', 'NORTHING', 'DIP', 'DIP_DIR',
'count', 'kappa', 'beta']
with open(os.path.join(path_out, file + ".csv"), "w",
encoding="utf-8") as out:
out.write(','.join(fieldnames) + '\n')
if len(starts):
counts = ends - starts
sums, T = self._cell_sums(gr['X'], starts) # all cells at once
kappa, beta = self._kent_kappa_batch(
counts, sums / counts[:, None], T / counts[:, None, None], return_beta=True)
for k in range(len(starts)):
if kappa_min is not None and not (
np.isfinite(kappa[k]) and kappa[k] >= kappa_min):
continue
dip2, dipdir2 = self._orientation_from_sums(
sums[k, 0], sums[k, 1], sums[k, 2], int(counts[k]))
if math.isnan(dip2) or math.isnan(dipdir2):
continue
s, t_ = starts[k], ends[k]
# Mean position of the input points (see gridcell_average)
centx = float(e[s:t_].mean())
centy = float(nn[s:t_].mean())
out.write(f"{centx:.8f},{centy:.8f},"
f"{self._fmt_dip(dip2)},{self._fmt_dipdir(dipdir2)},{int(counts[k])},"
f"{self._fmt_kent(kappa[k])},{self._fmt_kent(beta[k])}\n")
return file
# =========================================================================
# METHOD 5: OUTLIER REMOVAL (SINGLE-REMOVAL PER CELL)
# Remove the one measurement per cell whose removal most increases kappa.
# =========================================================================
@staticmethod
def _kent_kappa_batch(count, xbar, S, return_beta=False):
"""
Kent kappa (and optionally beta) for a batch of sets, from their sufficient statistics.
The same estimator as :meth:`_calc_kent`, vectorised so that the kappa of many
sets can be computed in one go (cells, leave-one-out sets, ...).
Parameters
----------
count : number of measurements in each set (int, or array of shape (m,)).
xbar : mean direction cosines of each set, shape (m, 3).
S : scatter matrix (sum of x x^T) / count of each set, shape (m, 3, 3).
Returns
-------
kappa (m,), or (kappa, beta) with ``return_beta``. ``inf`` where undefined for a
finite reason (fewer than 2 measurements, identical measurements, or a
non-positive denominator; beta is then ``nan``) and ``nan`` where the resultant
vector cancels - exactly the conventions of :meth:`_calc_kent`.
"""
m = xbar.shape[0]
count = np.broadcast_to(np.asarray(count), (m,))
r1 = np.sqrt((xbar ** 2).sum(axis=1))
kappa = np.full(m, np.inf)
beta = np.full(m, np.nan)
kappa[(r1 < 1e-12) & (count >= 2)] = np.nan
ok = (count >= 2) & (r1 < 1 - 1e-12) & (r1 >= 1e-12)
if not ok.any():
return (kappa, beta) if return_beta else kappa
xb, S_, r = xbar[ok], S[ok], r1[ok]
g = xb / r[:, None]
theta = np.arccos(np.clip(g[:, 2], -1.0, 1.0))
phi = np.arctan2(g[:, 1], g[:, 0])
ct, st, cp, sp = np.cos(theta), np.sin(theta), np.cos(phi), np.sin(phi)
# H maps the mean direction to z; only its first two columns h1, h2 are needed
h1 = np.stack([ct * cp, ct * sp, -st], axis=1)
h2 = np.stack([-sp, cp, np.zeros_like(cp)], axis=1)
S1, S2 = np.einsum('mij,mj->mi', S_, h1), np.einsum('mij,mj->mi', S_, h2)
b00 = (h1 * S1).sum(axis=1)
b01 = (h1 * S2).sum(axis=1)
b11 = (h2 * S2).sum(axis=1)
psi = 0.5 * np.arctan2(2 * b01, b00 - b11) # sword angle
c, s_ = np.cos(psi), np.sin(psi)
v00 = b00 * c * c + 2 * b01 * c * s_ + b11 * s_ * s_
v11 = b00 * s_ * s_ - 2 * b01 * c * s_ + b11 * c * c
r2 = v00 - v11
d1 = 2 - 2 * r - r2
d2 = 2 - 2 * r + r2
good = (d1 > 0) & (d2 > 0)
k = np.full(len(r), np.inf)
b = np.full(len(r), np.nan)
k[good] = 1 / d1[good] + 1 / d2[good]
b[good] = 0.5 * (1 / d1[good] - 1 / d2[good])
kappa[ok] = k
beta[ok] = b
return (kappa, beta) if return_beta else kappa
def _loo_kappas(self, X):
"""
Kent kappa of the cell with each measurement left out in turn (X: (n, 3)).
The same estimator as :meth:`_kent_kappa_batch`, but evaluated without building a
scatter matrix per left-out set: the quadratic forms of the reduced scatter matrix
are the quadratic forms of the full one minus the left-out point's own contribution.
"""
n = len(X)
m = n - 1
kappa = np.full(n, np.inf)
if m < 2:
return kappa
total = X.sum(axis=0)
T = X.T @ X
xbar = (total - X) / m
r1 = np.sqrt((xbar ** 2).sum(axis=1))
kappa[r1 < 1e-12] = np.nan
ok = (r1 < 1 - 1e-12) & (r1 >= 1e-12)
if not ok.any():
return kappa
x, xb, r = X[ok], xbar[ok], r1[ok]
g = xb / r[:, None]
theta = np.arccos(np.clip(g[:, 2], -1.0, 1.0))
phi = np.arctan2(g[:, 1], g[:, 0])
ct, st, cp, sp = np.cos(theta), np.sin(theta), np.cos(phi), np.sin(phi)
h1 = np.stack([ct * cp, ct * sp, -st], axis=1)
h2 = np.stack([-sp, cp, np.zeros_like(cp)], axis=1)
Th1, Th2 = h1 @ T, h2 @ T
p1, p2 = (x * h1).sum(axis=1), (x * h2).sum(axis=1)
b00 = ((h1 * Th1).sum(axis=1) - p1 * p1) / m
b01 = ((h2 * Th1).sum(axis=1) - p1 * p2) / m
b11 = ((h2 * Th2).sum(axis=1) - p2 * p2) / m
psi = 0.5 * np.arctan2(2 * b01, b00 - b11) # sword angle
c, s_ = np.cos(psi), np.sin(psi)
r2 = (b00 * c * c + 2 * b01 * c * s_ + b11 * s_ * s_) - (b00 * s_ * s_ - 2 * b01 * c * s_ + b11 * c * c)
d1 = 2 - 2 * r - r2
d2 = 2 - 2 * r + r2
good = (d1 > 0) & (d2 > 0)
k = np.full(len(r), np.inf)
k[good] = 1 / d1[good] + 1 / d2[good]
kappa[ok] = k
return kappa
def _iterative_outliers(self, X, min_keep, break_ratio):
"""
Outliers of one cell by repeated leave-one-out removal with a break rule.
Points are removed one at a time, each time the one whose removal gives
the highest Kent kappa, up to half the cell and never below ``min_keep``
points after the first removal. The relative kappa gain of each removal
is recorded. Just stopping once a removal no longer raises kappa does
not work, because removing any point from a small sample usually raises
kappa a little (30-70% for 8-15 points): in a clean cell that would trim
half of it. Real outliers instead show up as one large jump, so the
sequence is cut at its largest gain (as in :meth:`outlier_carmichael`),
and only if that gain is at least ``break_ratio`` times the median gain
of the other steps; otherwise nothing is removed.
``X`` is the (n, 3) array of direction cosines of the cell. Returns the list
of row positions (into X) to remove, in removal order (possibly empty).
"""
n = len(X)
active = np.arange(n)
k_prev = float(self._kent_kappa_batch(
n, X.mean(axis=0)[None, :], (X.T @ X / n)[None, :, :])[0])
seq = [] # (position removed, relative kappa gain)
max_remove = n // 2
while len(seq) < max_remove and np.isfinite(k_prev) and k_prev > 0:
if seq and len(active) <= min_keep:
break
kap = self._loo_kappas(X[active])
kap = np.where(np.isfinite(kap), kap, -np.inf)
j = int(np.argmax(kap))
if not np.isfinite(kap[j]):
break # kappa undefined for everything left
seq.append((int(active[j]), (kap[j] - k_prev) / k_prev))
k_prev = float(kap[j])
active = np.delete(active, j)
if not seq:
return []
gains = [g for _, g in seq]
best_i = int(np.argmax(gains))
if gains[best_i] <= 0:
return []
others = gains[:best_i] + gains[best_i + 1:]
if others and gains[best_i] < break_ratio * max(float(np.median(others)), 0.05):
return [] # no jump stands out from the noise
return [pos for pos, _ in seq[:best_i + 1]]
def outlier_removal(self, gdf, minx, maxx, miny, maxy,
n=1000, min_for_outlier=3, path_out='outputs/',
iterative=False, break_ratio=5.0):
"""
Remove outliers per grid cell using the max-delta-kappa criterion: at
most one by default, or several with ``iterative=True``.
Same grid as :meth:`gridcell_average`. For each cell with at least
``min_for_outlier`` measurements, the Kent kappa of the cell is
computed, then each measurement is left out in turn and
``delta_kappa = kappa(without it) - kappa(all)`` recorded. The
measurement with the largest delta_kappa is the outlier and is removed,
provided its removal increases kappa (delta_kappa > 0); otherwise
nothing is removed. The representative orientation is the vector mean
of the remaining measurements, at their mean position.
Cells with one measurement keep it; cells with two are averaged, since
removing either leaves a single measurement whose kappa is undefined,
so delta_kappa cannot tell the two apart.
The single-removal rule is a deliberate simplification of the iterative
method of Carmichael & Ailleres (2016), which is available as
:meth:`outlier_carmichael`. With ``iterative=True`` the leave-one-out
step is repeated to remove several outliers: points are taken out in
order of their effect on kappa and the sequence is cut at its largest
kappa jump, provided that jump stands out (see
:meth:`_iterative_outliers`). This differs from
:meth:`outlier_carmichael`, which ranks points by angle from the cell
mean rather than by their effect on kappa, and has no check that the
jump stands out.
Parameters
----------
gdf : Input GeoDataFrame of structural measurements.
minx, maxx : Western and eastern study-area boundaries (metres).
miny, maxy : Southern and northern study-area boundaries (metres).
n : Grid cell size in metres. Default: 1000 m.
min_for_outlier : Minimum measurements in a cell for outlier removal.
Default: 3.
path_out : Output directory path.
iterative : Remove several outliers (see above). Default: False.
break_ratio : With ``iterative``, the largest kappa jump must be at
least this many times the median of the other steps
to count as an outlier boundary. Default: 5.
Returns
-------
Filename stem (str) of the CSV written to ``path_out``. Columns:
EASTING, NORTHING, DIP, DIP_DIR, count (measurements kept), kappa and
beta of the kept measurements (-998 if undefined), REMOVED_INDEX
(index of the first removed measurement, -1 if none) and n_removed.
"""
gr = self._grid_groups(gdf, minx, maxx, miny, maxy, n)
X, e, nn, labels = gr['X'], gr['e'], gr['n'], gr['labels']
starts, ends = gr['starts'], gr['ends']
file = (f"structure_file_outlier_iterative_{n}" if iterative
else f"structure_file_outlier_{n}")
csv_path = os.path.join(path_out, f"{file}.csv")
with open(csv_path, "w", encoding="utf-8") as out:
out.write("EASTING,NORTHING,DIP,DIP_DIR,"
"count,kappa,beta,REMOVED_INDEX,n_removed\n")
for s, t_ in zip(starts, ends):
Xc = X[s:t_]
removed = []
if len(Xc) >= min_for_outlier and not iterative:
k0, _ = self._kappa_beta_of(Xc)
if np.isfinite(k0):
deltas = self._loo_kappas(Xc) - k0 # all leave-one-out at once
deltas = np.where(np.isnan(deltas), -np.inf, deltas)
j = int(np.argmax(deltas))
if deltas[j] > 0:
removed = [j]
elif len(Xc) >= min_for_outlier:
removed = self._iterative_outliers(Xc, min_for_outlier, break_ratio)
if removed:
keep = np.ones(len(Xc), dtype=bool)
keep[removed] = False
else:
keep = slice(None)
Xf = Xc[keep]
sums = Xf.sum(axis=0) + 0.0
dip_f, dipdir_f = self._orientation_from_sums(
sums[0], sums[1], sums[2], len(Xf))
if math.isnan(dip_f) or math.isnan(dipdir_f):
continue
kappa, beta = self._kappa_beta_of(Xf)
# Mean position of the retained points, not the cell centre
centx = float(e[s:t_][keep].mean())
centy = float(nn[s:t_][keep].mean())
first = labels[s + removed[0]] if removed else -1
out.write(f"{centx:.8f},{centy:.8f},{self._fmt_dip(dip_f)},{self._fmt_dipdir(dipdir_f)},"
f"{len(Xf)},{self._fmt_kent(kappa)},"
f"{self._fmt_kent(beta)},"
f"{first},"
f"{len(removed)}\n")
return file
# =========================================================================
# METHOD 5b: CARMICHAEL ITERATIVE OUTLIER REMOVAL
# Remove multiple outliers per cell using the Δκ-break algorithm.
# =========================================================================
def outlier_carmichael(self, gdf, minx, maxx, miny, maxy,
n=1000, path_out='outputs/'):
"""
Remove multiple outliers per grid cell using the Carmichael (2016)
iterative delta-kappa-break algorithm.
For each grid cell, points are ranked by angular distance from the cell
mean direction (farthest first). They are then removed iteratively and
kappa is recomputed after each removal. The step with the largest
increase in kappa defines the outlier boundary: every point removed up
to and including that step is treated as an outlier. At most N // 2
points may be removed from any cell of N measurements.
If no removal step produces a positive delta-kappa, the cell is left
unchanged. Cells with 3 or fewer measurements are excluded. Each
output point is at the mean position of the retained measurements.
Parameters
----------
gdf : Input GeoDataFrame of structural measurements.
minx, maxx : Western and eastern study-area boundaries (metres).
miny, maxy : Southern and northern study-area boundaries (metres).
n : Grid cell size in metres. Default: 1000 m.
path_out : Output directory path.
Returns
-------
Filename stem (str) of the CSV written to ``path_out``. Columns:
EASTING, NORTHING, DIP, DIP_DIR, count (measurements kept), kappa and
beta of the kept measurements (-998 if undefined), n_removed.
References
----------
Carmichael, T. & Ailleres, L. (2016). Method and analysis for the
upscaling of structural data. Journal of Structural Geology, 83, 121-133.
"""
gr = self._grid_groups(gdf, minx, maxx, miny, maxy, n)
X, e, nn = gr['X'], gr['e'], gr['n']
starts, ends = gr['starts'], gr['ends']
file = f"structure_file_outlier_carmichael_{n}"
csv_path = os.path.join(path_out, f"{file}.csv")
with open(csv_path, "w", encoding="utf-8") as out:
out.write("EASTING,NORTHING,DIP,DIP_DIR,"
"count,kappa,beta,n_removed\n")
for s, t_ in zip(starts, ends):
N = t_ - s
if N <= 3:
continue # too few measurements to define an outlier
P = X[s:t_]
# Cell mean direction as a unit vector
l_s, m_s, n_s = (float(v) for v in P.sum(axis=0) + 0.0)
g_len = sqrt(l_s**2 + m_s**2 + n_s**2)
if g_len < 1e-12:
continue # antipodal vectors cancel: no mean direction
mean_vec = np.array([l_s / g_len, m_s / g_len, n_s / g_len])
# Rank points by angular distance from the mean (farthest first)
dots = np.clip(P @ mean_vec, -1.0, 1.0)
ang_dists = np.degrees(np.arccos(dots))
rank = np.argsort(ang_dists)[::-1]
# Kappa after removing the 0, 1, ..., N//2 farthest points - all sets at once
# from suffix sums (sfx[i] = sum of the points ranked i and beyond)
m_rem = N // 2
Pr = P[rank]
sfx = np.cumsum(Pr[::-1], axis=0)[::-1]
outer = Pr[:, :, None] * Pr[:, None, :]
sfx_T = np.cumsum(outer[::-1], axis=0)[::-1]
cnt = N - np.arange(m_rem + 1)
kap = self._kent_kappa_batch(
cnt, sfx[:m_rem + 1] / cnt[:, None], sfx_T[:m_rem + 1] / cnt[:, None, None])
# kappa is undefined (inf/nan) for one or identical measurements:
# such steps cannot define the break
k_prev, k_t = kap[:-1], kap[1:]
delta_seq = np.where(np.isfinite(k_t) & np.isfinite(k_prev), k_t - k_prev, np.nan)
# Find the delta-kappa break (finite steps only)
if np.all(np.isnan(delta_seq)):
best_i, best_delta = 0, 0.0
else:
best_i = int(np.nanargmax(delta_seq))
best_delta = delta_seq[best_i]
if best_delta <= 0:
# No removal improves concentration - retain all points
n_removed = 0
keep = slice(None)
else:
n_removed = best_i + 1
keep = np.ones(N, dtype=bool)
keep[rank[:n_removed]] = False
Xf = P[keep]
sums = Xf.sum(axis=0) + 0.0
dip_f, dipdir_f = self._orientation_from_sums(
sums[0], sums[1], sums[2], len(Xf))
if math.isnan(dip_f) or math.isnan(dipdir_f):
continue
kappa_f, beta_f = self._kappa_beta_of(Xf)
# Mean position of the retained points, not the cell centre
centx = float(e[s:t_][keep].mean())
centy = float(nn[s:t_][keep].mean())
out.write(f"{centx:.8f},{centy:.8f},{self._fmt_dip(dip_f)},{self._fmt_dipdir(dipdir_f)},"
f"{len(Xf)},{self._fmt_kent(kappa_f)},"
f"{self._fmt_kent(beta_f)},{n_removed}\n")
return file
# =========================================================================
# METHOD 6: FIRST-ORDER SUBSAMPLING
# Filter by proximity to contacts and angular alignment with them.
# =========================================================================
def firstorder(self, gpdataframe, contact_gdf,
dist_buffer=500, angle_tol=15, path_out='outputs/',
max_points=None):
"""
Filter measurements by proximity to stratigraphic contacts and angular
alignment with them (first-order subsampling).
A measurement is retained only if **both** criteria are satisfied:
(a) **Proximity** — its distance to the nearest contact line is within
``dist_buffer`` metres.
(b) **Angular alignment** — the bedding strike differs from the contact
azimuth by no more than ``angle_tol`` degrees.
Parameters
----------
gpdataframe : Input GeoDataFrame with point geometries.
contact_gdf : GeoDataFrame of stratigraphic contact lines
(LineString or MultiLineString geometries).
dist_buffer : Maximum distance from any contact (metres).
Default: 500 m.
angle_tol : Maximum allowable strike angular difference (degrees).
Default: 15°.
path_out : Output directory path.
max_points : Optional cap on the number of points returned. If more
points pass both filters, the ``max_points`` best are
kept, ranked by
``distance/dist_buffer + angle_diff/angle_tol``
(lowest score = closest to a contact and best aligned
with it). If fewer pass, all are returned.
Returns
-------
GeoDataFrame of retained measurements passing both filters.
"""
gdf = gpd.GeoDataFrame(gpdataframe.copy())
if gdf.crs is not None and contact_gdf.crs is not None:
contact_gdf = contact_gdf.to_crs(gdf.crs)
# Contact segments and their azimuths, as arrays; nearest segment of every point from
# a spatial index (the distance to one big unioned contact geometry has no index and