Junior B.
3417f85eb1
Post-hoc improvement after the pre-registered v1 recovery test failed. Two changes, diagnosing v1's failure: - score on the full 12,328-gene LINCS space (week2_lincs_extract.py), lifting signature overlap from 12% to 85% (brings erythroid markers in) - src/scoring.py: KS connectivity + per-drug specificity z-score (spec_z = SDs below a 1,000 random-query null). Primary ranking is now spec_z. (Textbook tau saturated at +/-100 for a coherent query — documented; needs a reference-signature library, a v2 item.) - week3_scoring.py: spec_z primary + WTCS reference + prior-blended - tests: tau/spec_z calibration test; 19 passing - scripts/exp_genespace.py: the BING vs all-12,328 comparison Result: hydroxyurea recovers (rank 40 -> 18, top 6%, passes top-10%), confirming the v1 failure was the landmark bottleneck not the algorithm. Overall STILL FAILS: L-glutamine does not reverse (rank 213, metabolite), and negative controls (norethindrone, ciprofloxacin) rank top-3 — connectivity != therapeutic relatedness. v1.1 is post-hoc/exploratory, not a confirmatory test; reported as such in recovery_test_report.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
82 lines
3.4 KiB
Python
82 lines
3.4 KiB
Python
"""Week 4: formal recovery test against the pre-registered criteria (PLAN §6).
|
|
|
|
Pre-registered criteria (committed in docs/recovery_test_report.md before this run):
|
|
- hydroxyurea in top 10% (top 30 of 300), AND
|
|
- L-glutamine in top 25% (top 75) OR documented unscorable due to missing LINCS signature, AND
|
|
- >=4 of 5 pre-specified negative controls in the bottom half.
|
|
|
|
The 5 negative controls are pre-specified here by a category rule (one per category, alphabetically
|
|
first available) so the choice does not peek at ranks. Primary ranking = raw connectivity.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import pandas as pd
|
|
|
|
RANKED = Path("data/results/ranked_candidates_v1.csv")
|
|
|
|
# One per unrelated category, alphabetical-first — chosen without looking at ranks.
|
|
NEG_CONTROL_CATEGORIES = {
|
|
"antifungal": ["clotrimazole", "fluconazole", "itraconazole", "ketoconazole", "miconazole", "terbinafine"],
|
|
"antihistamine": ["astemizole", "cetirizine", "diphenhydramine", "fexofenadine", "loratadine"],
|
|
"antibiotic": ["azithromycin", "ciprofloxacin", "doxycycline", "tetracycline", "trimethoprim"],
|
|
"hormone": ["ethinyl-estradiol", "levonorgestrel", "medroxyprogesterone-acetate", "norethindrone"],
|
|
"misc": ["caffeine", "lidocaine", "loperamide", "omeprazole", "ranitidine"],
|
|
}
|
|
|
|
|
|
def main() -> None:
|
|
df = pd.read_csv(RANKED).set_index("drug_name")
|
|
n = len(df)
|
|
top10_cut, top25_cut, half = int(n * 0.10), int(n * 0.25), n // 2
|
|
|
|
def rk(name):
|
|
return int(df.loc[name, "rank"]) if name in df.index else None
|
|
|
|
hu, glut = rk("hydroxyurea"), rk("glutamine")
|
|
glut_z = df.loc["glutamine", "spec_z"]
|
|
|
|
# pick negative controls present in the ranking
|
|
negs = {}
|
|
for cat, options in NEG_CONTROL_CATEGORIES.items():
|
|
pick = next((d for d in options if d in df.index), None)
|
|
if pick:
|
|
negs[pick] = (cat, rk(pick))
|
|
|
|
print("=" * 60)
|
|
print(f"N = {n}; top10 cut = {top10_cut}, top25 cut = {top25_cut}, bottom-half > {half}")
|
|
print(f"\nhydroxyurea: rank {hu} (top {100*hu/n:.1f}%) -> top-10%? {hu <= top10_cut}")
|
|
print(f"L-glutamine: rank {glut} (top {100*glut/n:.1f}%), spec_z={glut_z:+.2f} "
|
|
f"-> top-25%? {glut <= top25_cut} (positive z => does not reverse; has a signature)")
|
|
print("\nnegative controls (pre-specified, 1 per category):")
|
|
n_bottom = 0
|
|
for d, (cat, r) in negs.items():
|
|
in_bottom = r > half
|
|
n_bottom += in_bottom
|
|
print(f" {d:18s} [{cat:13s}] rank {r:3d} bottom-half? {in_bottom}")
|
|
print(f" -> {n_bottom}/5 in bottom half (need >=4)")
|
|
|
|
crit_hu = hu <= top10_cut
|
|
crit_glut = glut <= top25_cut
|
|
crit_neg = n_bottom >= 4
|
|
overall = crit_hu and crit_glut and crit_neg
|
|
print(f"\nCRITERIA: hydroxyurea={crit_hu}, L-glutamine={crit_glut}, neg-controls={crit_neg}")
|
|
print(f"OVERALL (raw ranking): {'PASS' if overall else 'FAIL'}")
|
|
|
|
# secondary prior-weighted view (reported, not the primary criterion)
|
|
hu_b = int(df.loc["hydroxyurea", "blended_rank"])
|
|
print(f"\nsecondary (mechanistic-prior) ranking: hydroxyurea blended_rank {hu_b} "
|
|
f"(top {100*hu_b/n:.1f}%)")
|
|
|
|
print("\n--- TOP 10 (primary spec_z ranking) ---")
|
|
top10 = df.sort_values("rank").head(10)
|
|
for name, r in top10.iterrows():
|
|
print(f" {int(r['rank']):2d} {name:18s} z={r['spec_z']:+.2f} "
|
|
f"[{r['inclusion_reason']}] {str(r['known_targets'])[:45]}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|