-
Notifications
You must be signed in to change notification settings - Fork 0
Researcher encrypted results #764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 28 commits
Commits
Show all changes
61 commits
Select commit
Hold shift + click to select a range
6e76da2
Save design notes
chrisbendel b39ec87
Allow for researcher to get keys now too
chrisbendel a5f6a60
Merge branch 'main' into researcher-encrypted-results
chrisbendel 863078e
Fix tests + ci lint
chrisbendel ff8fd26
Fix test
chrisbendel 014af9f
Add keys for test researchers
chrisbendel 1b112c8
Update and compress notes
chrisbendel f4ded75
Test CI
chrisbendel 89296d7
Merge branch 'main' into researcher-encrypted-results
chrisbendel 1df75c0
Lint
chrisbendel a15d60a
Local dev working again
chrisbendel 918bfb6
Lint
chrisbendel 38b7fa2
Temp testing results file
chrisbendel 1088d74
Merge branch 'main' into researcher-encrypted-results
chrisbendel 5f7c5be
Lint
chrisbendel c0a8969
Fix up UI, full e2e testing flow
chrisbendel d038ed0
Tests
chrisbendel 396652f
Tests
chrisbendel f1a11a6
Merge branch 'main' into researcher-encrypted-results
chrisbendel 65a0466
Self review, add more comments/followup notes
chrisbendel 15c9aee
Merge branch 'main' into researcher-encrypted-results
chrisbendel 1c8d07d
Merge branch 'main' into researcher-encrypted-results
chrisbendel 783695f
Build?
chrisbendel 7b650b3
Fix test
chrisbendel 076fe6d
Remove design file
chrisbendel d8feec2
Update comments/verbiage to be more consistent
chrisbendel f6ac10e
Merge branch 'main' into researcher-encrypted-results
chrisbendel b8c0092
Update verbiage to reflect results key instead of reviewer key
chrisbendel 6f408af
Fix key
chrisbendel 7dd8aff
Fix keys
chrisbendel 3eddc0c
Merge branch 'main' into researcher-encrypted-results
chrisbendel 2c46b59
Update to pnpm 11.7
chrisbendel 8c3ffd0
Merge branch 'main' into researcher-encrypted-results
chrisbendel 7f906bd
Update approach for all/nothing
chrisbendel 7a1889a
Merge branch 'main' into researcher-encrypted-results
chrisbendel 6cd77e2
Review comment
chrisbendel 4ee4a74
Small fixes
chrisbendel 48bc93d
Tests and cleanup
chrisbendel b5b5915
Allow researcher to see logs
chrisbendel 9c4e090
Package fileg
chrisbendel 99a6349
Vulnerability
chrisbendel 8430816
Update tests
chrisbendel 475d768
Merge branch 'main' into researcher-encrypted-results
chrisbendel cd1387d
Prune comments, commit to all or nothing log + results
chrisbendel 96eb694
More clear naming for researcher keys
chrisbendel f93ec3d
Revalidate
chrisbendel 025834b
Use consistent UI
chrisbendel dfc56c4
Fix test
chrisbendel 336b1c0
Rename table to be more clear, address PR comments
chrisbendel 530882b
Add followup comment for future work
chrisbendel 64ce4be
Merge branch 'main' into researcher-encrypted-results
chrisbendel e321501
Fix CI
chrisbendel 76a8d9d
Fix?
chrisbendel f26659a
Merge branch 'main' into researcher-encrypted-results
chrisbendel ac41221
Fix local docker dev
chrisbendel ee1ac99
Merge branch 'main' into researcher-encrypted-results
chrisbendel 6f10717
Merge branch 'main' into researcher-encrypted-results
chrisbendel b1db329
Test
chrisbendel eb3bff9
Merge branch 'main' into researcher-encrypted-results
chrisbendel c7d3587
Fix tests
chrisbendel e75cb76
Pin to new encryption version
chrisbendel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
| package-lock=false | ||
| confirm-modules-purge=false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| /* eslint-disable no-console */ | ||
| /** | ||
| * DEV-ONLY: inject encrypted results into a job, mimicking what the Trusted Output App | ||
| * POSTs to /api/job/[jobId]/results — without the org-API auth dance. Lets you exercise | ||
| * the reviewer-decrypt → approve(re-wrap) → researcher-decrypt flow locally. | ||
| * | ||
| * Prereq: the job's reviewer(s) (enclave org users) must already have a key — generate it | ||
| * in the UI at /account/keys first, so getOrgPublicKeys returns a recipient to encrypt for. | ||
| * | ||
| * Run inside the container: | ||
| * docker exec mgmnt-app pnpm exec tsx bin/inject-encrypted-results.ts <jobId> | ||
| */ | ||
| import { db } from '@/database' | ||
| import { storeStudyEncryptedLogFile, storeStudyEncryptedResultsFile } from '@/server/storage' | ||
| import { ResultsWriter } from 'si-encryption/job-results/writer' | ||
|
|
||
| // Inline (not @/server/db/queries) — that module pulls in the Action framework → Clerk, | ||
| // which tsx can't import standalone. Mirrors getOrgPublicKeys. | ||
| async function orgPublicKeys(orgId: string): Promise<{ publicKey: ArrayBuffer; fingerprint: string }[]> { | ||
| const rows = await db | ||
| .selectFrom('orgUser') | ||
| .innerJoin('userPublicKey', 'userPublicKey.userId', 'orgUser.userId') | ||
| .select(['userPublicKey.publicKey', 'userPublicKey.fingerprint']) | ||
| .where('orgUser.orgId', '=', orgId) | ||
| .execute() | ||
| return rows.map(({ publicKey, fingerprint }) => { | ||
| const ab = new ArrayBuffer(publicKey.byteLength) | ||
| new Uint8Array(ab).set(publicKey) | ||
| return { publicKey: ab, fingerprint } | ||
| }) | ||
| } | ||
|
|
||
| const toArrayBuffer = (s: string): ArrayBuffer => { | ||
| const buf = Buffer.from(s, 'utf-8') | ||
| return buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength) | ||
| } | ||
|
|
||
| async function buildZip( | ||
| files: Array<{ name: string; content: string }>, | ||
| keys: { publicKey: ArrayBuffer; fingerprint: string }[], | ||
| ): Promise<File> { | ||
| const writer = new ResultsWriter(keys) | ||
| for (const f of files) await writer.addFile(f.name, toArrayBuffer(f.content)) | ||
| const zip = await writer.generate() | ||
| return new File([zip], 'encrypted.zip', { type: 'application/zip' }) | ||
| } | ||
|
|
||
| async function main() { | ||
| const jobId = process.argv[2] | ||
| if (!jobId) throw new Error('usage: tsx bin/inject-encrypted-results.ts <jobId>') | ||
|
|
||
| const info = await db | ||
| .selectFrom('studyJob') | ||
| .innerJoin('study', 'study.id', 'studyJob.studyId') | ||
| .innerJoin('org', 'org.id', 'study.orgId') | ||
| .select(['studyJob.id as studyJobId', 'study.id as studyId', 'study.orgId as orgId', 'org.slug as orgSlug']) | ||
| .where('studyJob.id', '=', jobId) | ||
| .executeTakeFirstOrThrow(() => new Error(`job ${jobId} not found`)) | ||
|
|
||
| // Encrypt for the enclave org's reviewer keys — exactly what TOA does. The reviewer | ||
| // decrypts with their private key, then re-wraps for researchers at approve. | ||
| const reviewerKeys = await orgPublicKeys(info.orgId) | ||
| if (!reviewerKeys.length) { | ||
| throw new Error( | ||
| `org ${info.orgSlug} has no public keys — log in as the reviewer and generate one at /account/keys first`, | ||
| ) | ||
| } | ||
| console.info(`Encrypting for ${reviewerKeys.length} reviewer key(s) in org ${info.orgSlug}`) | ||
|
|
||
| // Idempotent: don't stack duplicate result rows if re-run against the same job. | ||
| const alreadyHasResults = await db | ||
| .selectFrom('studyJobFile') | ||
| .select('id') | ||
| .where('studyJobId', '=', jobId) | ||
| .where('fileType', '=', 'ENCRYPTED-RESULT') | ||
| .executeTakeFirst() | ||
|
|
||
| if (alreadyHasResults) { | ||
| console.info(`Job already has encrypted results — skipping store (delete study_job_file rows to re-inject).`) | ||
| } else { | ||
| const resultsZip = await buildZip( | ||
| [ | ||
| { name: 'results.csv', content: 'group,count\nA,42\nB,17\n' }, | ||
| { name: 'summary.txt', content: 'analysis complete: 2 groups\n' }, | ||
| ], | ||
| reviewerKeys, | ||
| ) | ||
| const logZip = await buildZip([{ name: 'run.log', content: 'job started\njob finished ok\n' }], reviewerKeys) | ||
| await storeStudyEncryptedResultsFile(info, resultsZip) | ||
| await storeStudyEncryptedLogFile(info, logZip, 'ENCRYPTED-CODE-RUN-LOG') | ||
| } | ||
|
|
||
| // Make the status chain realistic. A results-stage job must have had its code | ||
| // approved first — otherwise the proposal-level review buttons stay visible (they | ||
| // only hide once the study is APPROVED *and* the code was reviewed). Inject | ||
| // CODE-APPROVED before RUN-COMPLETE so the latest status stays RUN-COMPLETE and the | ||
| // page shows only the results Approve/Reject. | ||
| const statuses = await db | ||
| .selectFrom('jobStatusChange') | ||
| .select(['status', 'createdAt']) | ||
| .where('studyJobId', '=', jobId) | ||
| .execute() | ||
| const has = (s: string) => statuses.some((r) => r.status === s) | ||
| const runComplete = statuses.find((r) => r.status === 'RUN-COMPLETE') | ||
|
|
||
| if (!has('CODE-APPROVED')) { | ||
| // 1s before an existing RUN-COMPLETE, or 2s in the past on a fresh job so it | ||
| // reliably precedes the RUN-COMPLETE inserted just below. | ||
| const ts = runComplete | ||
| ? new Date(new Date(runComplete.createdAt).getTime() - 1000) | ||
| : new Date(Date.now() - 2000) | ||
| await db | ||
| .insertInto('jobStatusChange') | ||
| .values({ status: 'CODE-APPROVED', studyJobId: jobId, createdAt: ts }) | ||
| .execute() | ||
| } | ||
| if (!runComplete) { | ||
| await db.insertInto('jobStatusChange').values({ status: 'RUN-COMPLETE', studyJobId: jobId }).execute() | ||
| } | ||
| // Code approved ⇒ proposal-level review is done; study is back to APPROVED. | ||
| await db.updateTable('study').set({ status: 'APPROVED' }).where('id', '=', info.studyId).execute() | ||
|
|
||
| console.info(`✓ injected encrypted results + logs for job ${jobId} (CODE-APPROVED → RUN-COMPLETE, study APPROVED)`) | ||
| console.info(` Now: review as the reviewer, decrypt with the reviewer PEM, Approve in the Study Status card.`) | ||
| await db.destroy() | ||
| } | ||
|
|
||
| main().catch((err) => { | ||
| console.error(err) | ||
| process.exit(1) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.