Why This Approach src/api/proxy.py
The fundamental challenge: how do you observe someone's real-time interaction with eCitizen, Afya Yangu, KRA iTax, Niko, or another public-service route without modifying those services, collecting unnecessary personal data, or forcing the participant into an artificial lab task? We evaluated several approaches before settling on a mobile-first participant experience with proxy-plus-extension capture for real web services.
Approaches considered
| Approach | What we tested or researched | Why it was not enough |
|---|---|---|
| Screen recording | Reviewed as a direct observation method for usability sessions. | Intrusive, requires video consent, exposes more personal information than AGS needs, and does not produce structured timing/error/help signals without manual coding. It also changes participant behavior because the task becomes visibly recorded. |
| Think-aloud observation | Considered for qualitative pilots and facilitator-led sessions. | Useful for interviews, but not valid for real-time adaptive support. Asking an older adult to narrate while completing an unfamiliar form adds a second task and contaminates hesitation, timing, and confidence signals. |
| Manual facilitator assistance | Considered as the simplest support model. | Helpful but not reproducible. Different facilitators intervene at different times and with different phrasing, which makes condition comparison weak. AGS needs consistent, logged, replayable decisions. |
| Mock websites only | Used where controlled practice tasks are needed. | Good for calibration and low-risk practice, but insufficient for the main research question. Real public-service pages include unpredictable friction: login handoffs, validation copy, CAPTCHA, layout changes, and slow network behavior. |
| Native mobile app per service | Assessed as a way to fully control the user interface. | Would require building and maintaining a separate app or wrapper per service, which does not scale and would stop being the real website. It also excludes desktop/browser workflows that participants may actually use. |
| Phone-first PWA / mobile browser | Selected as the primary participant access path because older users are most likely to complete real-service journeys on a phone. | Strong for reach and ecological validity, but mobile browsers make deep capture harder: extension support is limited, iframe behavior varies, and virtual-keyboard/touch signals must be treated differently from desktop cursor signals. |
| Niko route | Tested as a real route/service path to check whether AGS can observe a practical participant journey rather than only controlled mock tasks. | Useful for validating routing, capture behavior, and participant flow assumptions, but it should be documented as one tested route, not treated as the whole architecture. AGS still needs a general capture strategy that works across multiple services. |
| Browser automation / remote control | Considered for repeatability and scripted flows. | Too artificial for participant research. Automation observes the script more than the person, and remote-control tooling can create privacy, latency, and accessibility issues. |
| Extension-only capture | Built/researched for pages that reject iframe and proxy embedding. | Technically strong on real tabs, but the installation step is a serious burden. Making every participant install an unpacked extension before onboarding would increase attrition and bias the sample toward more technically confident users. |
| Proxy-only capture | Tested as the lowest-friction browser path. | Best default, but not universal. Iframe restrictions, CAPTCHA, origin checks, and some cross-origin scripts cannot be made reliable with header rewriting alone. |
| Tauri desktop app | Prototyped as a desktop shell that could package AGS with a more controlled browser container. | Promising as a future option for lab sessions, researcher machines, or desktop-heavy studies. It is not the default because it adds installation and platform-support burden, and it does not match the mostly mobile interaction pattern expected for older users. |
Why proxy plus extension won
Architecture trade-off
The trade-off is complexity. Supporting phone-first PWA sessions, proxy capture, extension handoff, tested service routes such as Niko, and possible future desktop shells means AGS must track signal_capture_mode, capture_channel, device type, extension handoff, and fallback state. We accept that complexity because it protects the study from two larger risks: excluding participants who cannot install tooling, and losing data whenever a real service refuses to be embedded.
Current and future packaging
| Packaging path | Status | When to use it |
|---|---|---|
| Phone / PWA | Primary | Default participant path for older users. Best match for real mobile service use, touch signals, font-size preferences, and low-install onboarding. |
| Web dashboard | Current | Researcher monitoring, configuration, content management, exports, and audit review. |
| Browser extension | Conditional | Use when a real service route needs real-tab capture, CAPTCHA compatibility, or anti-frame recovery. |
| Niko route | Tested route | Use as evidence that AGS was exercised against a practical route/service path, and as a regression route when checking capture behavior. |
| Tauri desktop app | Future option | Useful for controlled desktop studies, lab deployments, or researcher-operated sessions. Keep it separate from the default participant path unless the study population shifts toward desktop use. |
The three capture modes
AGS uses whichever capture mode the task template specifies — or falls back down the chain if the preferred mode isn't available.
What it does: The AGS server fetches the target page on behalf of the participant's browser, strips the security headers that would block iframe embedding (X-Frame-Options, Content-Security-Policy), injects the AGS signal-collection script (ags-runtime.js) into the HTML, and returns the modified page. The participant sees the real website content. AGS has a JavaScript agent running inside it.
Why it works for most sites: The government sites we target don't use bot-detection systems sophisticated enough to flag server-side fetches. Most of the heavy lifting (form rendering, navigation) happens client-side in JavaScript, which runs normally inside the iframe.
Limitations: CAPTCHAs detect iframe context and break. Some pages use JavaScript that checks window.location.origin and refuse to load if it doesn't match. These cases fall through to the extension.
What it does: Instead of routing through the server, a registered Service Worker in the browser intercepts fetch requests made by the iframe and rewrites them to go through /ags/sw-proxy?url=…. The server processes them the same way as proxy mode.
Why it exists: On Android Chrome, iframe rendering inside a PWA can be choppy — especially for sites with heavy JavaScript. The SW approach lets the page render more natively while still routing requests through the AGS server for header stripping and script injection.
Limitation: The Service Worker must already be registered and active in the participant's browser. If it's not ready at session start, the session falls back to server-side proxy automatically.
What it does: The AGS Chrome/Brave extension runs directly in the context of the real tab — not an iframe, not a proxy. It injects the AGS overlay and signal collection into whatever page the participant is on, including external tabs they navigate to during a session.
Why it's necessary: reCAPTCHA and similar systems actively detect iframe embedding. They check window.top !== window.self, inspect frame ancestors in CSP headers, and sometimes check the Referer chain. A proxy can strip headers but can't defeat in-page JavaScript checks. The extension sidesteps all of this by operating in the real tab.
CAPTCHA handoff: If the proxy detects a CAPTCHA (via ags-runtime.js sending an ags_captcha_detected message), and the extension is installed, the session automatically hands off — the real URL opens in a new tab, the extension picks up the session state, and the participant continues without disruption.
Limitation: Requires installation. We provide a ZIP download and step-by-step install instructions within the app. Participants who don't have the extension stay in proxy mode with a fallback banner for CAPTCHA steps.
Known limitations and failure modes
These are real constraints that affect session quality in the field, not theoretical edge cases. Researchers should understand them when interpreting signal completeness and attrition data.
| Mode | Limitation | What happens | Mitigation |
|---|---|---|---|
| Proxy | CAPTCHA and iframe detection | reCAPTCHA and similar widgets check window.top !== window.self at runtime. The proxy can strip CSP and X-Frame-Options headers but cannot defeat in-page JavaScript checks. The widget fails to render or submits an invisible challenge the participant cannot complete. |
Extension handoff. ags-runtime.js sends ags_captcha_detected when it detects a reCAPTCHA iframe; if the extension is installed the session hands off to a real tab automatically. |
| Proxy | Origin-checking JavaScript | Some pages read window.location.origin, document.referrer, or inspect the request Origin header and refuse to proceed if the value is not the real service domain. This breaks login redirects and post-form navigation on certain eCitizen flows. |
ags-runtime.js patches Location.prototype for navigation results. Static URL rewrites in the proxy handle most cases; complex dynamically constructed URLs may still break and result in a blank iframe. |
| Proxy | Cookie jar is server-side only | Cookies are stored in Redis per session and sent by the AGS server — the participant's browser never receives the service's cookies directly. This means any pre-existing authenticated session the participant has in their own browser is invisible to the proxy. They always start fresh. | By design for privacy. Participants log in through the proxy and the login session persists for the duration of their AGS session. Researchers should instruct participants not to expect their existing logins to carry over. |
| Proxy | Nested iframes not injected | Only the top-level HTML response gets ags-runtime.js injected. Sub-iframes that the proxied page loads (payment widgets, embedded maps, third-party login frames) do not get the AGS script. Signals from those frames are not captured. |
Practice templates avoid multi-frame flows. For real-service sessions, signal gaps in nested frames appear as signal_capture_mode = "limited" in the session record. |
| Proxy | Shared server IP | All proxy requests originate from the AGS Railway server IP. Government portals with aggressive rate limiting or bot detection may eventually block or throttle this IP, particularly if many sessions run concurrently. | Keep concurrent sessions within expected study limits. Monitor GET /dashboard/alerts for data quality alerts flagging elevated error rates on specific services. |
| Extension | Desktop Chrome and Brave only | The extension requires a Chromium-based desktop browser. Android Chrome does not support extensions. iOS Safari does not support Chrome extensions. Participants on mobile-only devices cannot use the extension path — they stay in proxy mode with a fallback banner on CAPTCHA steps. | Design real-service tasks to either work in proxy mode or explicitly require a desktop session. The capture_channel field in session records shows which participants used each path. |
| Extension | Unpacked installation (developer mode) | The extension is not in the Chrome Web Store. Participants must enable developer mode in Chrome settings and load an unpacked ZIP — a multi-step process that is difficult for older adults unfamiliar with browser settings. Installation failure is the leading cause of extension path attrition. | The AGS onboarding flow includes step-by-step install instructions with screenshots. Researchers using the extension path should allow extra setup time and have a support script ready. See Support Script. |
| Service Worker | Must be registered before session start | The SW is registered when the participant first loads the AGS PWA. If they start a session before the registration completes — or in a fresh browser with no prior visit — the SW is not active. The session automatically falls back to server-side proxy. | The ServiceSelectView runs detectServiceWorker() before session creation and resolves the effective browser mode. The fallback is transparent to the participant. |
| Service Worker | Blocked in private/incognito mode | Chrome and Brave block Service Worker registration in private browsing. Participants who use incognito mode will always fall back to server-side proxy regardless of template configuration. | Instruct participants to use a normal (non-incognito) browser session. The participant onboarding consent screen includes this note. |
How the script (ags-runtime.js) works
In proxy and SW mode, every HTML response gets <script src="/ags-runtime.js?sid=…&step=…" defer></script> injected before </head>. This script:
- Listens for DOM events (clicks, keypresses, scroll, form input) and batches them for the periodic flush to
/v1/sessions/{id}/events - Monitors navigation and sends
ags_navmessages to the parent frame so the URL bar in the AGS UI stays accurate - Detects reCAPTCHA iframes and sends
ags_captcha_detectedso the extension handoff can happen - Patches
window.WebSocketto route connections through/proxy-ws/…so real-time features on proxied sites don't break - Applies font-size preference when the parent sends a
ags-practice-font-sizemessage
sid, step, and token values. By serving it as /ags-runtime.js?sid=…, the browser can cache the script itself (it doesn't change between deploys) while the query params carry the per-session context.