Import college counseling data pipelines from futures-v2 - #23
Closed
amogh-gawaskar wants to merge 1 commit into
Closed
Import college counseling data pipelines from futures-v2#23amogh-gawaskar wants to merge 1 commit into
amogh-gawaskar wants to merge 1 commit into
Conversation
Adds four dataset pipelines: josaa, mcc, state_cet, and state_medical, along with the COLLEGE_SOURCES_HANDOFF.md overview document. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sakshi1755
reviewed
Jul 27, 2026
Comment on lines
+151
to
+152
| r"|MIO?|MIH?" | ||
| r")$" |
Contributor
There was a problem hiding this comment.
Why changed: the trailing [HOS] was mandatory, but long codes like
# DEFRSEBC(S) / PWDROBC(S) / PWDRSEBC(S) wrap their trailing letter onto
# the next header line when the column is too narrow. Confirmed via
# direct PDF text dump: DEFRSEBC (137x), PWDROBC (62x), PWDRSEBC (11x)
# appear on header lines WITHOUT the trailing letter across the
# engineering CAP1-4 PDFs. Since the match failed, these columns were
# never registered, and every rank value under them was silently
# dropped. Made [HOS] optional (matches normalise_category()/_decode_body(),
# which already ignore the suffix for DEF/PWD-prefixed categories, so
# this doesn't change how they get bucketed — it just stops discarding
# them at the column-recognition step).
r"|DEF(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]?"
r"|PWD(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]?" # Why changed: the trailing [HOS] was mandatory, but long codes like
# DEFRSEBC(S) / PWDROBC(S) / PWDRSEBC(S) wrap their trailing letter onto
# the next header line when the column is too narrow. Confirmed via
# direct PDF text dump: DEFRSEBC (137x), PWDROBC (62x), PWDRSEBC (11x)
# appear on header lines WITHOUT the trailing letter across the
# engineering CAP1-4 PDFs. Since the match failed, these columns were
# never registered, and every rank value under them was silently
# dropped. Made [HOS] optional (matches normalise_category()/_decode_body(),
# which already ignore the suffix for DEF/PWD-prefixed categories, so
# this doesn't change how they get bucketed — it just stops discarding
# them at the column-recognition step).
Comment on lines
+148
to
+149
| r"|DEF(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]" | ||
| r"|PWD(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]" |
Contributor
There was a problem hiding this comment.
Suggested change
| r"|DEF(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]" | |
| r"|PWD(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]" | |
| r"|DEF(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]?" | |
| r"|PWD(?:R)?(?:OPEN|OBC|SC|ST|VJ|NT[1-3]|SEBC|SBC)[HOS]?" |
Comment on lines
+268
to
+269
| if "(" in ln and ")" in ln: | ||
| continue |
Contributor
There was a problem hiding this comment.
Why changed: pdftotext -layout sometimes splits one real Stage row
across two output lines when a value wraps, leaving a lone
percentage stranded on the same line as other real rank numbers.
The blanket rule discarded the WHOLE row (real numbers included) —
confirmed 122 rows lost in engineering CAP1-4, 4 in pharmacy.
Fix: mask out only the parenthesised percentage spans (keeping
column offsets intact for _match_column), and only skip the line
if nothing but whitespace/parens is left after masking.
Comment on lines
+268
to
+269
| if "(" in ln and ")" in ln: | ||
| continue |
Contributor
There was a problem hiding this comment.
Suggested change
| if "(" in ln and ")" in ln: | |
| continue | |
| ln = re.sub(r"\([\d.]+\)", lambda mm: " " * len(mm.group()), ln) | |
| if not re.search(r"\d{2,7}", ln): | |
| continue |
Comment on lines
+328
to
+336
| ALLOT_PATTERN_FLAT = re.compile( | ||
| r"^\s*(\d{1,5})\s+" | ||
| r"(\d{1,7})\s*\(([\d.]+)\)\s+" | ||
| r"(\d{10}[A-Z]?)\s+" | ||
| r"(\d{5})\s*-\s*(.+?)\s{2,}" | ||
| r"(.+?)\s{2,}" | ||
| r"(MH to AI|AI to AI|MI to AI)\s+" | ||
| r"([A-Z]{1,12})\s*$" | ||
| ) |
Contributor
There was a problem hiding this comment.
#Why changed: the allotment-type + seat-type tail was mandatory, but many
real rows end right after the exam name (e.g. "...Mechanical Engineering
JEE") with no "MH to AI"/"AI to AI"/etc. text anywhere on the line at all
— confirmed directly against the source PDF, not a wrapping issue. Making
the tail optional recovered rows that were previously dropped whole
(measured: strict match rate went from ~33% to a much higher share of
loose matches across CAP1-4). allotment_type/seat_type are None on rows
where the PDF genuinely doesn't carry that tail — course_name may swallow
the exam name in that case, matching this parser's existing behaviour.
Comment on lines
+333
to
+335
| r"(.+?)\s{2,}" | ||
| r"(MH to AI|AI to AI|MI to AI)\s+" | ||
| r"([A-Z]{1,12})\s*$" |
Contributor
There was a problem hiding this comment.
Suggested change
| r"(.+?)\s{2,}" | |
| r"(MH to AI|AI to AI|MI to AI)\s+" | |
| r"([A-Z]{1,12})\s*$" | |
| r"(.+?)" | |
| r"(?:\s{2,}(MH to AI|AI to AI|MI to AI)\s+([A-Z]{1,12}))?" | |
| r"\s*$" |
Comment on lines
+341
to
+348
| ALLOT_PATTERN_RANKROW = re.compile( | ||
| r"^\s*(\d{1,5})\s+" | ||
| r"(\d{1,7})\s*\(([\d.]+)\)\s+" | ||
| r"(\d{10}[A-Z]?)\s+.*?" | ||
| r"(JEE|NEET|MHT-?CET|CET)\s+" | ||
| r"(MH to AI|AI to AI|MI to AI)\s+" | ||
| r"([A-Z]{1,12})\s*$" | ||
| ) |
Contributor
There was a problem hiding this comment.
Suggested change
| ALLOT_PATTERN_RANKROW = re.compile( | |
| r"^\s*(\d{1,5})\s+" | |
| r"(\d{1,7})\s*\(([\d.]+)\)\s+" | |
| r"(\d{10}[A-Z]?)\s+.*?" | |
| r"(JEE|NEET|MHT-?CET|CET)\s+" | |
| r"(MH to AI|AI to AI|MI to AI)\s+" | |
| r"([A-Z]{1,12})\s*$" | |
| ) | |
| # Why changed: same issue as ALLOT_PATTERN_FLAT — the allotment-type + | |
| # seat-type tail was mandatory, but some rows genuinely end right after the | |
| # exam name with no tail text at all. Made it optional so those rows are | |
| # kept (with allotment_type/seat_type as None) instead of dropped whole. | |
| ALLOT_PATTERN_RANKROW = re.compile( | |
| r"^\s*(\d{1,5})\s+" | |
| r"(\d{1,7})\s*\(([\d.]+)\)\s+" | |
| r"(\d{10}[A-Z]?)\s+.*?" | |
| r"(JEE|NEET|MHT-?CET|CET)" | |
| r"(?:\s+(MH to AI|AI to AI|MI to AI)\s+([A-Z]{1,12}))?" | |
| r"\s*$" | |
| ) |
Comment on lines
+527
to
+541
| agg = (df.groupby( | ||
| ["college_code", "college_name", "branch_code", "branch_name", | ||
| "status", "quota_section", "category_raw"], dropna=False) | ||
| .agg(closing_rank=("rank", "max"), | ||
| opening_rank=("rank", "min"), | ||
| num_rank_observations=("rank", "count")) | ||
| .reset_index()) | ||
| last = (df.sort_values("rank", ascending=False) | ||
| .drop_duplicates(["college_code", "branch_code", "quota_section", "category_raw"]) | ||
| [["college_code", "branch_code", "quota_section", "category_raw", | ||
| "round", "stage"]] | ||
| .rename(columns={"round": "last_round_with_max", | ||
| "stage": "last_stage_with_max"})) | ||
| agg = agg.merge(last, on=["college_code", "branch_code", "quota_section", "category_raw"]) | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| agg = (df.groupby( | |
| ["college_code", "college_name", "branch_code", "branch_name", | |
| "status", "quota_section", "category_raw"], dropna=False) | |
| .agg(closing_rank=("rank", "max"), | |
| opening_rank=("rank", "min"), | |
| num_rank_observations=("rank", "count")) | |
| .reset_index()) | |
| last = (df.sort_values("rank", ascending=False) | |
| .drop_duplicates(["college_code", "branch_code", "quota_section", "category_raw"]) | |
| [["college_code", "branch_code", "quota_section", "category_raw", | |
| "round", "stage"]] | |
| .rename(columns={"round": "last_round_with_max", | |
| "stage": "last_stage_with_max"})) | |
| agg = agg.merge(last, on=["college_code", "branch_code", "quota_section", "category_raw"]) | |
| # Why changed: pdftotext -layout sometimes merges a stray boilerplate | |
| # line (e.g. a recurring page-header line) onto the SAME output line as | |
| # a college-code header, so college_name/status pick up trailing | |
| # garbage text on some pages but not others for the SAME real | |
| # college_code. Grouping by college_name/branch_name/status (literal | |
| # strings) as part of the key then silently split one real institute | |
| # into multiple rows, each aggregated over only part of its true data — | |
| # producing duplicate rows with DIFFERENT, both-wrong (undercounted) | |
| # closing ranks. Confirmed: 143 conflicting duplicate rows in | |
| # engineering's final output alone. | |
| # | |
| # First attempt just dropped college_name/branch_name/status from the | |
| # groupby key entirely and canonicalized status to one value per | |
| # (college_code, branch_code) — but status is NOT always constant even | |
| # within one branch: some institutes (e.g. Bombay College of Pharmacy, | |
| # 03016) genuinely run BOTH Government-Aided AND Un-Aided seats under | |
| # the same branch code, confirmed directly in the source PDF (both | |
| # statuses appear within the same round). Collapsing status there | |
| # silently merged two real, distinctly-funded seat pools into one and | |
| # dropped 3 genuinely-govt colleges from the govt-scope count. | |
| # | |
| # Fix: classify college_type from the RAW per-row status BEFORE | |
| # aggregating, and include college_type (not raw status text) in the | |
| # groupby key. This restores the original correct behaviour for genuine | |
| # dual-status branches (they still get 2 separate rows, one per type — | |
| # confirmed Bombay College of Pharmacy's two status strings both already | |
| # map to their real types), while still fixing the actual corruption bug: | |
| # corrupted college_name/status text (e.g. Sardar Patel College's | |
| # boilerplate-contaminated row) maps to the SAME college_type as its | |
| # clean counterpart, so those still correctly merge into one row. | |
| df["college_type"] = df["status"].apply(classify_college_type) | |
| agg = (df.groupby( | |
| ["college_code", "branch_code", "quota_section", "category_raw", | |
| "college_type"], dropna=False) | |
| .agg(closing_rank=("rank", "max"), | |
| opening_rank=("rank", "min"), | |
| num_rank_observations=("rank", "count")) | |
| .reset_index()) | |
| def _canonical(frame, keys, value_col): | |
| return (frame.assign(_len=frame[value_col].fillna("").str.len()) | |
| .sort_values("_len") | |
| .drop_duplicates(keys)[keys + [value_col]]) | |
| agg = agg.merge(_canonical(df, ["college_code"], "college_name"), | |
| on="college_code", how="left") | |
| agg = agg.merge(_canonical(df, ["college_code", "branch_code"], "branch_name"), | |
| on=["college_code", "branch_code"], how="left") | |
| # status text is canonicalized WITHIN each (college_code, branch_code, | |
| # college_type) group — safe because every row in that group already | |
| # shares the same real type; this only cleans up incidental text noise | |
| # (e.g. a trailing "Home University : ..." clause present on some pages | |
| # and not others for the same status). | |
| agg = agg.merge(_canonical(df, ["college_code", "branch_code", "college_type"], "status"), | |
| on=["college_code", "branch_code", "college_type"], how="left") | |
| last = (df.sort_values("rank", ascending=False) | |
| .drop_duplicates(["college_code", "branch_code", "quota_section", | |
| "category_raw", "college_type"]) | |
| [["college_code", "branch_code", "quota_section", "category_raw", | |
| "college_type", "round", "stage"]] | |
| .rename(columns={"round": "last_round_with_max", | |
| "stage": "last_stage_with_max"})) | |
| agg = agg.merge(last, on=["college_code", "branch_code", "quota_section", | |
| "category_raw", "college_type"]) | |
| agg["quota"] = agg["quota_section"] | ||
| agg["rank_basis"] = f"{cfg['cet_name']} State Merit Rank" | ||
| agg["source_url"] = cfg["source_url"] | ||
| agg["college_type"] = agg["status"].apply(classify_college_type) |
Contributor
There was a problem hiding this comment.
Suggested change
| agg["college_type"] = agg["status"].apply(classify_college_type) | |
| # college_type is already a column (computed pre-aggregation and used as | |
| # a groupby key above) — no need to recompute it from the canonicalized | |
| # status text. |
Comment on lines
+67
to
+68
| r"[*@~^&]\s*" # color marker (^/~/*/@ etc.) | ||
| r"([A-Z]{4,12})\s*$" # SeatType (e.g., LOPENH, GOBCH) |
Contributor
There was a problem hiding this comment.
Suggested change
| r"[*@~^&]\s*" # color marker (^/~/*/@ etc.) | |
| r"([A-Z]{4,12})\s*$" # SeatType (e.g., LOPENH, GOBCH) | |
| # Why changed: the marker was mandatory, but Round 1 (and a small % of | |
| # every other round) legitimately has no marker at all — it denotes | |
| # "carried forward from a previous round," which doesn't exist yet in | |
| # Round 1. That alone silently dropped ~1,811 rows (~99% of Round 1). | |
| # Separately, {4,12} rejected real short SeatType codes like "AI" | |
| # (All India, 2 letters) and "EWS"/"SC"/"ST" (2-3 letters), dropping | |
| # another ~1,060 rows even when the marker WAS present. | |
| r"[*@~^&]?\s*" # color marker (^/~/*/@ etc.) — now optional | |
| r"([A-Z]{2,12})\s*$" ``` |
This was referenced Aug 17, 2026
suryabulusu
added a commit
that referenced
this pull request
Aug 18, 2026
* tnea: import the TN scraper with the govt-college classification fix Seeds the tnea/ source folder with the TNEA scraper from futures-v2, including the console-script portal extractor and the code-based government-college classification. Schemas, loaders and the tnea_fact_cutoffs table follow when this source ships end to end, per the one-source-at-a-time migration (PR #23). Co-authored-by: sakshi jadhave <sakshi.s.jadhav1755@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@claude.com> * tnea: add the two constituent UCEs the 2021 DOTE PDF predates Review of the imported classifier against the real 2025 portal data (423 colleges): all 36 listed codes present and correctly classified, zero false positives -- but a sweep for unclassified govt-looking names found two: 5017 University College of Engineering, Ramanathapuram 5022 University College of Engineering, Dindigul Both are Anna University constituent colleges, the same family as the 14 already in CONSTITUENT_COLLEGE_CODES; they are simply newer than the Dec-2021 "List of Colleges (Code Number wise)" PDF the sets were transcribed from. Without them the TN govt scope silently drops 2 of its ~35 colleges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@claude.com> --------- Co-authored-by: sakshi jadhave <sakshi.s.jadhav1755@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@claude.com>
Contributor
|
WBJEE: done (the first source to run the full ONBOARDING.md checklist end to end).
Remaining in the state_cet queue: AP (needs a data hunt - scraper used a 2022 proxy), KEAM, Bihar, Kerala, Odisha. |
Contributor
|
KEAM: done (second full ONBOARDING.md run; the checklist held, including its two newest steps).
Remaining in the state_cet queue: AP (data hunt needed), Bihar, Odisha. |
Contributor
|
AP EAPCET: done (fourth checklist run; the data hunt is resolved).
Remaining in the state_cet queue: Bihar, Odisha. |
Contributor
|
OJEE (Odisha): done - and with it, every scraper in the state_cet queue is migrated. Five full ONBOARDING.md runs: WBJEE, KEAM, AP EAPCET, OJEE (plus TNEA/KCET/GUJCET before the checklist existed).
state_cet queue: EMPTY. Remaining on this PR: the josaa/ waystation (already mirrored here) and the standing refresh drills (WBJEE mop-up, KEAM P3/medical, AP 2026, OJEE 2026). |
Contributor
|
Closing this PR: the migration it tracked is complete. Every source in the queue now lives in external_data_sources with the full chain - raw archive in GCS, clean parquet, BigQuery fact, data-assistant schema, college-predictor exam, and open-data publication:
futures-v2 is thinned to pointer notes; the process that did this is codified in ONBOARDING.md. Standing refresh drills (documented in each source's sources.py): WBJEE mop-up round, KEAM engineering P3 + Kerala 2026 medical, AP EAPCET 2026 consolidated, OJEE 2026 OR-CR, and the NIC-portal lead for OJEE's non-B.Tech courses. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
avantifellows/futures-v2: josaa, mcc, state_cet, state_medicalCOLLEGE_SOURCES_HANDOFF.md— high-level overview of all four sourcesschemas/,scrape/scripts/,scripts/(build, load_bq, sources, upload_to_gcs)Datasets included
josaa/mcc/state_cet/state_medical/Test plan
nirf/,plfs/,udise/)🤖 Generated with Claude Code