💻 Pertemuan 11 | Week 11 | Hands-On Session

Hands-On: Unsupervised Learning

Praktik langsung implementasi K-Means clustering pada data well log South Barrow 18 dengan validasi geologis menggunakan formation tops data

⏱️ 3 × 50 menit
💻 Live Coding Session
🗿 Geological Validation
🔬 Praktikum 03: K-Means Clustering pada Well Log

Tutorial lengkap step-by-step dengan 40 code cells, 11 visualizations, geological interpretation, dan BONUS section tentang validasi dengan formation tops USGS!

🚀 Buka Praktikum 03
📊 South Barrow 18 Dataset
🗿 USGS Formation Tops
📈 4 Clusters + Geological Validation

🎯 Tujuan Hands-On Session

Pada akhir session ini, mahasiswa diharapkan mampu:

  1. Implement end-to-end clustering workflow: Load LAS file → preprocessing → clustering → validation → geological interpretation
  2. Integrate stratigraphic data: Load formation tops, plot boundaries, analyze cluster distribution per formation
  3. Validate clustering results: Apply Silhouette, Davies-Bouldin, Calinski-Harabasz, plus external metrics (ARI, NMI)
  4. Interpret geologis: Understand relationship antara lithofacies (clusters) dan chronostratigraphic units (formations)
  5. Troubleshoot common issues: Missing values, scaling, parameter tuning, convergence problems
  6. Compare algorithms: K-Means vs Hierarchical vs DBSCAN pada same dataset
  7. Visualize professionally: Well log plots with formation tops, cross-plots, stratigraphic columns

📋 Struktur Session (150 menit)

0-20 min
📖 Review & Setup

• Quick review: Unsupervised learning & validation metrics (Pertemuan 09-10)
• Environment setup: Jupyter Notebook, libraries installation
• Download datasets: SB18.LAS + SB18TOPS.txt dari Praktikum 03
• Introduction to South Barrow 18 stratigraphy (USGS data)

20-50 min
💻 Live Coding Part 1: Data Loading & EDA

• Load LAS file dengan lasio
• Load formation tops dari SB18TOPS.txt
• Exploratory Data Analysis: distributions, cross-plots, well log display
• Handle missing values strategy
• Feature selection & scaling dengan StandardScaler
Follow: Praktikum 03 Section 2-5

50-90 min
🎯 Live Coding Part 2: K-Means + Validation

• Determine optimal K: Elbow Method + Silhouette Analysis
• Implement K-Means clustering (K=4)
• Compute internal metrics (Silhouette, DB, CH)
• Visualize clustered well logs & cross-plots
• Plot formation tops as red dashed lines
Follow: Praktikum 03 Section 6-8

90-120 min
🗿 Geological Interpretation & Validation

• Interpret cluster centroids → lithofacies identification
• Analyze depth distribution by cluster
• Assign formations to data points
• Cross-tabulation: Cluster vs Formation distribution
• Compute external metrics (ARI, NMI) dengan proper interpretation
• Discuss: Why clusters cross-cut formation boundaries (NORMAL!)
Follow: Praktikum 03 Section 9-10 + BONUS

120-150 min
🔄 Challenges & Q&A

• Work on coding challenges (formation-specific clustering, heatmaps, etc.)
• Try alternative algorithms (Hierarchical, DBSCAN)
• Open discussion: troubleshooting, best practices
• Wrap-up & assignment briefing (must include formation tops validation)

🛠️ Persiapan Sebelum Session

1. Software & Environment

💻 Required Setup

Python 3.8+ dengan libraries berikut: 

pip install pandas numpy matplotlib seaborn lasio scikit-learn scipy

Recommended IDE: Jupyter Notebook, VS Code, atau Google Colab

2. Datasets (Download BOTH!)

Download dari Praktikum 03:

  • SB18.LAS - South Barrow 18 Well Log (Alaska, USA)
    • Size: ~786 KB
    • Records: 4,053 depth points (98-2,124 ft)
    • Curves: 13 log curves (GR, RHOB, NPHI, DT, SP, ILD, dll.)
  • SB18TOPS.txt - Formation Tops (USGS Open File Report 00-200)
    • Size: <1 KB
    • Source: Gryc, George, Ed., 1988
    • Content: 4 formation boundaries dengan depths

3. Study Materials

Review materi berikut sebelum session:

🗿 Geological Context: Formation Tops

South Barrow 18 memiliki data formation tops dari USGS (Gryc, 1988) yang memberikan ground truth stratigraphic boundaries. Data ini sangat penting untuk memvalidasi dan menginterpretasi clustering results dalam konteks geologis.

📊 Formation Tops Data (USGS)
Formation Top Depth Lithology Age
Gubik Formation 18 ft Unconsolidated sediments Quaternary
Torok Formation 75 ft Shale (deep marine) Cretaceous
Pebble Shale Unit 1,375 ft Shale with dropstones Cretaceous
Kingak Shale 1,760 ft Marine shale Jurassic

Source: USGS Open File Report 00-200 (Gryc, George, Ed., 1988) | Download SB18TOPS.txt

Expected Clustering Behavior

⚠️ Critical Understanding: Formation boundaries ≠ Cluster boundaries!

  • Formations: Chronostratigraphic units (defined by age + depositional environment)
  • Clusters: Petrophysical facies (defined by well log response patterns)
  • Relationship: NOT 1:1 mapping! One formation can contain multiple lithofacies
💡 What to Expect in Session
  • Torok Formation (75-1,375 ft): Lithologically heterogeneous (interbedded shale/silt/sand) → expect MIXED clusters within this formation
  • Pebble Shale Unit (1,375-1,760 ft): Dominant shale matrix + dropstones/pebbles → cluster variability reflecting this internal heterogeneity
  • Kingak Shale (1,760-2,124 ft): Marine shale sequence → relatively uniform but still shows internal facies changes
  • Formation tops: Will be plotted as red dashed lines on well logs for geological reference, NOT as validation targets!

Why Clusters Cross-Cut Formations (This is NORMAL!)

Geologically Expected Behavior:

  • Formations capture time + depositional environment (chronostratigraphy)
  • Clusters capture petrophysical properties at 0.5 ft resolution
  • Single formation dapat mengandung multiple lithofacies (e.g., Torok = shale + silt + sandstone interbeds)
  • Clustering reveals lithological heterogeneity WITHIN formations → valuable for reservoir modeling!
⚠️ External Validation Caveat

Ketika compute ARI & NMI comparing clusters to formations:

  • Low scores (~0.04-0.13) are EXPECTED dan geologically reasonable
  • These metrics designed untuk compare clusterings, NOT lithofacies vs stratigraphy
  • Low scores CONFIRM clustering captures heterogeneity within formations
  • High scores would actually be suspicious (oversimplified geology)

Takeaway: Use formation tops as geological CONTEXT, not validation TARGETS!

🏆 Coding Challenges

Setelah menyelesaikan workflow dasar dari Praktikum 03, coba challenges berikut. Formation tops data tersedia untuk validasi geologis!

EASY

Challenge 1: Plot Formation Tops

Add red dashed horizontal lines pada well log plots untuk mark formation boundaries. Use plt.axhline(y=depth, color='red', linestyle='--') untuk overlay tops (Torok: 75 ft, Pebble Shale: 1,375 ft, Kingak: 1,760 ft).

EASY

Challenge 2: Formation Assignment

Write function assign_formation(depth) untuk assign formation name ke setiap depth point. Create new column df_clean['Formation']. Print cluster distribution per formation using pd.crosstab() dengan normalize='index'.

MEDIUM

Challenge 3: Formation-Specific Clustering

Subset data untuk Torok Formation only (75-1,375 ft range). Run K-Means clustering again pada this interval dengan K=2 to 6. Compare optimal K dan cluster characteristics dengan full-dataset clustering. Interpret differences!

MEDIUM

Challenge 4: Cross-Tab Heatmap

Create heatmap showing cluster distribution (%) across formations. Use seaborn.heatmap() dengan pd.crosstab() result. Annotate percentages di each cell dengan annot=True, fmt='.1f'. Which formation shows highest cluster diversity?

HARD

Challenge 5: External Validation

Compute Adjusted Rand Index (ARI) dan Normalized Mutual Information (NMI) comparing cluster labels dengan formation labels. Use sklearn.metrics. CRITICAL: Explain mengapa low scores (~0.04) are EXPECTED dan geologically reasonable!

HARD

Challenge 6: Stratigraphic Column

Create publication-quality stratigraphic column plot showing: (1) Formation names & boundaries dengan different colors, (2) Cluster proportions per formation as stacked horizontal bars, (3) Mean GR values per formation. Make it professional & visually clear!

💡 Challenge Tips & Hints
  • Parsing tops file: Parse SB18TOPS.txt programmatically atau use dictionary hardcoded
  • Depth filtering: Use boolean indexing: df[(df.index >= 75) & (df.index < 1375)]
  • Formation assignment: Use df.index.map(assign_formation) untuk vectorized operation
  • External metrics: Import adjusted_rand_score, normalized_mutual_info_score from sklearn.metrics
  • Interpretation key: Low ARI/NMI ≠ bad clustering! Means formations contain multiple lithofacies!
  • Visualization: Use axhline() untuk formation tops, different colors untuk different formations

🔧 Common Issues & Solutions

1. Missing Values Error

⚠️ Problem

ValueError: Input contains NaN ketika fit K-Means

✅ Solution

Always check dan handle missing values sebelum clustering:

df_clean = df[features].dropna()  # Remove rows with any NaN
# OR
df_filled = df[features].fillna(df[features].mean())  # Impute with mean

2. Scaling Issues

⚠️ Problem

Clusters dominated by features dengan large ranges (e.g., GR: 0-200 dominates RHOB: 2.4-2.6)

✅ Solution

ALWAYS scale features sebelum distance-based clustering:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df_clean[features])

3. Formation Tops Not Showing on Plot

⚠️ Problem

Red dashed lines untuk formation tops tidak muncul di plot

✅ Solution

Check depth range dan ensure formation tops within plot limits:

# Check if formation top is within data range
for fm_name, fm_depth in formation_tops.items():
    if fm_depth > depth.min() and fm_depth < depth.max():
        ax.axhline(y=fm_depth, color='red', linestyle='--', 
                   linewidth=2, alpha=0.7, label=fm_name)

4. Low External Validation Scores

⚠️ "Problem"

ARI = 0.043, NMI = 0.127 → "apakah clustering gagal?"

✅ Solution (Understanding)

Low scores are EXPECTED and GEOLOGICALLY CORRECT!

  • Formations = chronostratigraphic units (age + environment)
  • Clusters = petrophysical facies (lithology patterns)
  • These are COMPLEMENTARY classifications, not identical
  • Low ARI/NMI confirms clustering reveals lithological heterogeneity WITHIN formations
  • Use formation tops as geological CONTEXT, not validation TARGETS

📚 Additional Resources

Documentation & Tutorials

Geological References

  • USGS Open File Report 00-200: Selected Data from Fourteen Wildcat Wells in NPR-Alaska
  • Gryc (1988): Geology and exploration of the National Petroleum Reserve in Alaska
  • Alaska North Slope Stratigraphy: Bird & Magoon (1987)

Practice Datasets

  • Kaggle: FORCE 2020 Well Log & Lithology Prediction Competition
  • Kansas Geological Survey: Open well log data dengan core descriptions
  • NLOG (Netherlands): Dutch offshore wells dengan formation tops

📝 Assignment Post-Session

Deadline: 1 minggu setelah hands-on session

Deliverables:

  1. Jupyter Notebook (.ipynb) dengan complete workflow:
    • Load SB18.LAS + SB18TOPS.txt datasets
    • EDA dengan visualizations
    • K-Means clustering implementation
    • Validation dengan internal metrics (Silhouette, DB, CH)
    • Formation tops integration: plot boundaries, assign formations, cross-tabulation
    • External validation: ARI, NMI dengan proper geological interpretation
    • At least 2 challenges completed dari coding challenges list
  2. Report (PDF, max 5 pages) mencakup:
    • Methodology summary
    • Results: optimal K determination, cluster characteristics
    • Geological interpretation: cluster facies analysis per formation
    • Formation tops validation: cross-tab analysis, external metrics interpretation
    • Discussion: Why clusters cross-cut formations, geological significance
    • Challenges completed dengan screenshots/results
    • Reflection: what worked, what didn't, lessons learned

Grading Criteria:

  • Code quality & completeness (30%) - including formation tops integration
  • Proper use of validation metrics (25%) - both internal & external
  • Geological interpretation (25%) - emphasis on formations vs clusters understanding
  • Challenges & creativity (20%) - quality over quantity
💡 Tips untuk Success
  • Start early! Don't wait until last minute - formation tops integration adds complexity
  • Comment your code clearly: Especially formation assignment logic
  • Include visualizations: Well logs with formation tops, cross-tab heatmap
  • Reference Praktikum 03: Use BONUS section as template untuk formation validation
  • Interpret geologically: Don't just report numbers - explain what they mean!
  • Ask questions: Use forum/office hours jika stuck with formation tops parsing
🎯 Key Requirements Checklist
  • ✅ Load BOTH datasets (SB18.LAS + SB18TOPS.txt)
  • ✅ Plot formation tops as red dashed lines on well logs
  • ✅ Assign formation names to each depth point
  • ✅ Create cross-tabulation: Cluster vs Formation
  • ✅ Compute ARI & NMI with PROPER geological interpretation
  • ✅ Explain why low external scores are EXPECTED
  • ✅ Complete at least 2 coding challenges
  • ✅ Include reflection on formations vs lithofacies concepts